The Science of Learning and the Virtual Anesthesia

The Science of Learning and the Virtual Anesthesia Machine: Benefits of "schematic" simulations in learning about complex systems Ira Fischler Simulation Faculty Learning Community May 2008

Colaborators: n n n Sem Lampotang (Anesthesiology) Cynthia Kaschub (Psychology) David Lizdas (Anesthesiology) And for jump-starting this effort: n Sue Legg (Director, Partnership for Global Learning)

Plan for the talk: n n n n “Learning with understanding: ” The idea of mental models in psychology and education How multimedia presentations can boost learning Potential advantages of simulation Transparent simulations and understanding The Virtual Anesthesia Machine (VAM) Learning with Transparent versus Opaque VAM Bridging abstract and concrete models: Mixed Reality and the Augmented Anesthesia Machine A little bit about individual differences

The mini-science of learning n What makes a difference? ¨ Amount of practice (and the Power Law) ¨ Distribution of practice (and the Spacing Effect) ¨ Quality of practice (and Depth of Processing) n Making the information distinctive n Building appropriate “mental models” “I’m doing great in all my other classes. I read the book, came to class, outlined the material, and made flash cards, and still got a C. ” “Well, did you understand the material? ” “I thought I did…”

Mental models and schemas in comprehension “If the balloons popped, the sound wouldn’t be able to carry since everything would be too far away from the correct floor. A closed window would also prevent the sound from carrying, since most buildings tend to be well insulated. Since the whole operation depends on the steady flow of electricity, a break in the middle of the wire would also cause problems. Of course, the fellow could shout, but the human voice is not loud enough for the sound to carry that far. An additional problem is that the string could break on the instrument. Then there would be no accompaniment to the message. It is clear that the best situation would involve less distance….

Mental models in cognitive science n Term first used by Kenneth Craik (’ 43) “If the organism carries a “small-scale model” of external reality and of its own possible actions within its head, it is able to try out various alternatives, conclude which is the best of them, react to future situations before they arise, utilise the knowledge of past events in dealing with the present and future, and in every way to react in a much fuller, safer, and more competent manner to emergencies which face it. ” (Craik, The Nature of Explanation, 1943) Quality of the model depends on how well it captures the features of the domain that are critical for the task at hand n

Understanding problems (Greeno, 1977) n Our internal representation (or model) of the problem should have CORRESPONDENCE between relevant elements in the world and model ¨ good COHERENCE between elements in the model ¨ appropriate links to PRIOR KNOWLEDGE that can aid problem solving correspondence links to prior knowledge coherence ¨ accurate mental model of problem environment

Little things (can) mean a lot (aka the devil’s in the details) n Subtle changes in problem “framing” can have drastic effects on performance ¨ Effects of analogy on solving the “X-ray” problem n Preceded by bulb filament problem “fragile glass” framing: 33% then solve X-ray ¨ “laser intensity” framing: 69% then solve X-ray ¨ ¨ Effects of lives lost/saved on risky decisions: n Disease control programs, one more risky “Lives saved” framing: 22% choose risky action ¨ “Lives lost” framing: 75% choose risky action ¨

Pictorial Representations n n Came before text, historically Illustrations and drawings ¨ To n Animations and videos ¨ To n illuminate structure, function and relations make system dynamics visible Interactive simulations ¨ To actively explore cause-effect dynamics, test hypotheses, etc. n Advantages of “multimedia” can be dramatic:

Mayer’s work on Multimedia (e. g. , How lightning forms) n Compares. . ¨ text-only to text-with-illustration (often schematic) ¨ Narration-only with narration-plus-animation n Tests. . ¨ Retention by free recall of presented facts ¨ Transfer (understanding? ) by generating solutions to: n n Redesign (“how could you decrease lightning intensity? ”) Troubleshooting (“how could there be clouds, but no lightning? ”) Prediction (“what would happen with lower air temperature? ”) Abstraction of Principles (“What causes lightning? ”)

Retention and transfer with MM n Retention: Modest MM gains ¨ Across n 6 studies, 23% gain, 0. 67 effect size Transfer: Dramatic MM gains ¨ Across 6 studies, 89% gain, 1. 50 effect size

Potential advantages of Computer-based simulation n n Cost: cheap systems, easy to replace, low risk Track performance and provide “just-in-time” feedback on performance “Virtually Real” when needed But Reality can be played with: ¨ Increase likelihood of rare but important events ¨ Increase salience of important features ¨ Present “hyper-real” depictions of space and time ¨ Make the abstract concrete, and the invisible

Instructional Choice-Points n What do we want them to learn? ¨ Declarative knowledge, procedural skills ¨ Immediate or long-term retention ¨ Reproductive or creative, flexible learning n How do we structure the learning? ¨ Amount of “grounding” in the domain ¨ Balance of guided (reception) and free (discovery) learning ¨ Amount of online assessment and intelligent tutoring ¨ Student-tailored, or one-size-fits-all

Opaque versus Transparent Reality n n Opaque representation: Simulation may be closely analogous to the physical system (iconic, concrete, high-fidelity, virtual reality) but hides underlying structure, functions and relations Transparent representation: Simulation sacrifices physical fidelity but makes underlying aspects of system overt (abstract, idealized, schematic, symbolic)

Transparency in simulations n n n Hollan’s STEAMER (1981) Goldstone’s Concreteness Fading (2004) Butcher’s simplified diagrams (2006) Debate focusses on “extent of fidelity” and whether detail helps or hurts Little direct comparison of simulation formats

The Opaque-Reality VAM

The Transparent-Reality VAM

The Virtual Anesthesia Machine wide use, little data n n n 10 man-years of development time Available for free to individuals on the web Over 10, 000 registered users Many positive reviews, both formal and informal Our goal: assess the effectiveness of VAM’s Transparent Reality approach to simulation

Training Session n 30 -page instructional guide developed ¨ Provides foundation of knowledge n About anesthesia n About the anesthesia machine and its subsystems n Guided tour of several subsystems ¨ Breathing circuit ¨ Mechanical ventilation ¨ Manual ventilation n Stresses visualization of dynamics using VAM

Workbook: Sample text n Question 1: elimination of CO 2. Are the gases exhaled by a patient “scrubbed” of CO 2 before entering the bellows during mechanical ventilation? n Demonstration using VAM Simulation: ____ Click “Reset” to start simulation afresh ____ Point to the O 2 flowmeter control knob to enlarge it, then click-and-hold, and drag it counterclockwise until the O 2 bobbin inside is about halfway up the tube.

Workbook: sample text (cont’d) n What does this do? What happens to the flow of O 2 from the supply line? ¨ Opening the valve increases the flow of O 2 from the supply line into the breathing circuit. n Trace along its route through the plumbing. Where does it wind up? ¨ It depends. For example, If mechanical ventilation is selected, but not on, the O 2 flows “backward” through the CO 2 absorber, past the bellows and into the scavenger system

Performance on Day 2 Tests: undergraduate health majors Undergraduates 4. 0 Transparent VAM Opaque VAM Quality Score 3. 0 2. 37 2. 03 2. 0 1. 90 1. 67 1. 64 1. 51 1. 0 0. 0 Component Identity Component Function System Dynamics

Performance on Day 2 Tests: 2 nd-year medical students Medical Students 4. 0 3. 12 Quality Score 3. 0 3. 27 2. 93 Transparent VAM 3. 05 Opaque VAM 2. 59 2. 29 2. 0 1. 0 0. 0 Component Identity Component Function System Dynamics

Performance on AAA Board Exam Review questions (4 AFC) Percent Correct Board Exam Review Questions 1. 0 Transparent VAM 0. 9 Opaque VAM 0. 8 0. 62000000 01 0. 7 0. 6 0. 49 0. 46 0. 5 0. 37 0. 4 0. 3 Undergraduates Medical Students

Judgments about VAM n Confidence Judgments ¨ On n Significantly higher for Transparent VAM (p <. 01) ¨ On n n Component function: System dynamics: Marginally higher for Transparent VAM (p <. 15) Preferences for additional study 17 of 20 in Transparent group (UG) prefer TR VAM ¨ 11 of 20 in Opaque group prefer (UG) TR VAM ¨ 2 in TR, 7 in OR, think both would be preferable to either ¨ Similar trends among medical students; more want both ¨

Where to next? n Combination and order effects ¨ Goldstone’s n “concreteness fading” method? More precise tests of transfer ¨ Transfer to procedural skill: does TR improve error detection and response? n Hybrid simulations: John Quarles’ project

The Augmented Anesthesia Machine (AAM) n Integrating transparent and realistic representations with “mixed-reality” simulation John Quarles and his Magic Lens

Declarative and Procedural Knowledge with VAM and AAM n n Two groups of undergrads Training: ¨ Introduction to AM with VAM ¨ “positioning” components within the actual AM ¨ Five step-through exercises with VAM or AAM n Day 2 Testing ¨ Declarative: Board Exam Questions ¨ Procedural: Find a machine fault in the AM

Performance with VAM vs. AAM 0. 7 Virtual Machine 0. 6 Augmented Machine Percent Correct 0. 5 0. 4 0. 3 0. 2 0. 1 0 Board Exam Questions Detection of Fault

Abstract and Concrete Knowledge n “Although the VAM may offer improved abstract knowledge, participants found it difficult to transfer this knowledge to the concrete anesthesia machine. This is precisely the concern that anesthesia educators have had with the VAM. n For example, many VAM participants understood the abstract concept of the inhalation valve and they correctly answered the written questions regarding the gas flow in the valve. However, during the fault test, they could not perform the mental mapping between the abstract representation of the VAM inhalation valve and the concrete representation of the real anesthesia machine inhalation valve. Thus, it was difficult for VAM participants to apply their abstract knowledge to a concrete problem, such as the problem presented in the fault test. ”

Role of Spatial Abilities? n Three tests of spatial cognition ¨ Arrow-pointing working memory (small-scale) ¨ Perspective-taking (mid-scale) ¨ Navigating virtual environment (large-scale) n Correlations of spatial abilities and performance tend to be larger with VAM than AAM ¨ Suggests those with strong visualization skills can compensate for impoverished materials

What we’ve learned n n Dynamic simulations can improve comprehension of, and memory for, complex systems, BUT Different kinds of simulation are optimum for different kinds of learning ¨ So we need to know the goal of training Experience with both abstract (schematic) and concrete (hi-fidelity) simulations may be optimum ¨ So we may need an integrated approach Individual differences in domain-specific skills and abilities will impact effectiveness of representations ¨ But we need to know how much

It takes a Village (or at least a Learning Community) n n n Cognitive/human factors psychologists Usability analysts Instructional psychologists and educators Simulation designers and engineers Domain experts and professionals

Thanks to all those RA’s n n n Emily Mc. Alister Jonathan Greenwood Julianna Peters Shannon Bowie Sheila Holland Trudy Salmon