Diagnostic errors in medical education Where wrongs can

Diagnostic errors in medical education: Where wrongs can make rights Kevin W. Eva Centre for Health Education Scholarship University of British Columbia

No one likes to make mistakes

But they aren’t all avoidable

Conclusions • Mistakes are necessary when trying to enable learning • Diagnostic strategies that avoid one type of error may increase the rate of another type

Caveats • Promoting errors is not about reducing patient safety • Mistakes are not simply the result of individuals’ decision-making • Nor are they indicative of lack of dedication/ability of health professionals

er r e ch ia a e T l 3 O ut line M at 2 1 Learner

Considerations 1. Errors that arise while learners are learning new material 2. Errors in understanding how well said material was learned 3. Errors educators can use/induce to facilitate learning

Overconfidence • Premature closure • Lack of insight • God complex • Lake Woebegone ? y W h Learner See Berner and Graber (2008); Crosskerry (2002) Ma te ria l

Self-regulated learning y l • Requires deciding: g n • What to study i g s i n • How long to study r i p g r • When to stop n u S alle h c See Nelson and Narens (1994) Learner M at er ia l

Jowett, et al. (2007) Learner M at er ia l

Jowett, et al. (2007) • Proficiency is inferred from the rate of learning rather than the amount learned See Kornell and Metcalfe (2006) Learner M at er ia l

The Which fluency is better? heuristic Sessions Massed Training Spaced Training Few, Intense Many, Spread out Speed Faster Confidence Higher Satisfaction Greater Retention Longer Performance Better

Desirable difficulties • Teaching strategies that elicit more errors (and can make learning seem more difficult) are often more beneficial See Bjork (1999); Simon and Bjork (2001) Learner M at er ia l

Test-enhanced learning Percent Recall Study, Test 0, 9 0, 8 0, 7 0, 6 0, 5 0, 4 0, 3 0, 2 0, 1 0 5 Minutes 2 Days 1 Week Retention Interval Roediger and Karpicke (2006)

Summary • Errors are more likely to be made over the long term if they are not induced during learning • There is a clear need for external prompts and guidance Learner M at er ia l

er r e ch ia a e T l 3 O ut line M at 2 1 Learner

Three maxims • Feedback is good • The more immediate, often, and accurate, the better • Errors in medicine proliferate because feedback is often Learner unavailable r a Te e ch

Boehler, et al. (2006) Compliment group Feedback group Performance Rating 25 20 15 10 5 0 Pre-test Post-instruction Post-feedback

What predicts generation of learning goals? 0, 35 Standardized beta 0, 3 Overall R 2=0. 45; Eva et al. , 2010 0, 25 0, 2 0, 15 0, 1 0, 05 0 Station Observer rating Student Feedback quality NOTE: Absolute values illustrated for ease of comparison Station order Self rating

The ironic nature of selfassessment • We can’t trust it … but we should encourage it • We need to influence it … without changing it

The need for difficulties Promoting “self-assessment” should be about … (a) creating situations that will enable learners to discover the limits of their knowledge/ability (b) role modeling the habit of seeking and using data to guide continuous quality improvement (see Simon and Bjork, 2001; Eva, 2009)

Credibility may be the key “With your clinical partner … they know your strengths and weaknesses and they’re probably going to give you quite useful feedback. Maybe a consultant watched you one time. They might not have the whole picture of what you’re capable of and what you normally do” (Sargeant, et al. , 2010)

Credibility Of relationships: Engagement and awareness Of the process: Validity and authenticity Of intent: Beneficence and non-maleficence (Sargeant, et al. , 2010)

Summary • Learners rarely induce in themselves the sorts of errors that maximize learning • It’s the educator’s role to facilitate growth from errors and empower learners to make errors in a safe place Learner r a Te e ch

er r e ch ia a e T l 3 O ut line M at 2 1 Learner

The source of all evil • The use of pattern recognition by novices is not advocated for fear that “[i]nadequate experience might lead to potentially grim consequences” • See Coderre, et al. (2003) M at er ia l a Te er h c

What else could it be? Is there anything that doesn’t fit?

Overcoming the evil • “Cognitive forcing strategies … are designed to prevent clinicians from pursuing a pattern recognition path that will typically lead to error” • See Crosskerry (2002) M at er ia l a Te er h c

Norman, et al. (1989) Correct Incorrect DK Response Time 30 25 20 15 10 5 0 Student Clerk Resident GP Dermatologist

Non-exclusivity Non-analytic processes Combined Analytic processes See Eva (2005)

Ark, Brooks, and Eva (2006) Feature Oriented First Impression Combined Diagnostic Accuracy 70 60 50 40 30 20 10 0 Old ECGs New ECGs

Ark, Brooks, and Eva (2006) Feature calls Feature Oriented First Impression Combined 2 1, 8 1, 6 1, 4 1, 2 1 0, 8 0, 6 0, 4 0, 2 0 Hits Indicative Hits Not Indicative False Alarms

Summary • We need to structure experiences for students that help them recognize that exclusive reliance on nonanalytic processes does create error … • … but so does absolute aversion to those same M at er ia l a Te er h c

Strategies for doing so • Manipulating order of examples • Inducing learners to compare and contrast cases • Actively engaging learners in problem solving • See Eva, et al. (1998); Eva (2009) M at er ia l a Te er h c

The grand finale • Dominant discourse around diagnostic error: Heuristics bad • Problems arising from that discourse: • This view overlooks the fact that any strategy can lead to errors though they are of different types • Reinforces the tendency for students to want to be right rather than to learn how

“An expert is a person who has made all the mistakes that can be made in a very narrow field. ” (Niels Bohr)

“Education is learning what you didn’t even know you didn’t know” (Boorstin, 1914)

Thanks kevin. eva@ubc. ca See Eva, 2009 (AHSE) for more