Script Concordance Testing to Assess Clinical Reasoning Mary

Script Concordance Testing to Assess Clinical Reasoning Mary Blumer, MD; Meenu Jindal, MD; Ihsan Elkhider, LL. B, MS. Ed Tom Pace, MD; Peggy J Wagner, Ph. D Greenville Health System, School of Medicine Greenville Background and Significance • • Teaching clinical reasoning is an educational challenge Application of Script theory developed by cognitive psychologists is new in medical education • Professionals use “scripts” to process information and to confirm or eliminate diagnostic hypotheses or management options • Qualitative judgments can be measured and compared to expert practitioners allowing us to examine decision making in contexts of uncertainty. Script Concordance Tests (SCTs) offer innovation and practicality as they can be scored automatically by the process of comparing the examinees score to a panel of experts using probabilities of response options. Previous research has demonstrated internal reliability and the ability to differentiate medical learners at various levels 1 Iron deficiency anemia History of uterine bleeding -2 -1 0 +1 +2 2 B 12 deficiency anemia History of diabetes on glucophage for treatment -2 -1 0 +1 +2 No koilonychia on physical exam -2 -1 0 +1 +2 3 Students performance data in each SCT is collected and compared to the experts panel as well as to their summative exam score Global SCT statistics are calculated for each test (Test reliability, mean , and SD). For example , ID module SCT: High reliability. 65; Mean score 72, SD 11. 9 Example Data from ID module 2015 A 30 year-old male presents with fever and shortness of breath. He has been feeling sick for the last 2 -3 weeks. Choose the response that best reflects your opinion using the following options: -2 - Not indicated -1 – Not useful at this point 0 – Neither more nor less useful +1 – Indicated/useful +2 – Will be very useful 5 If you were. . . and you find the considering asking following evidence. . . for. . . Chest x-ray Crackles on lung auscultation Sputum stain High Aa gradient 6 Bronchoscopy 4 . . . the investigation becomes. . . Students’ Perception • N= 71 -2 -1 0 +1 +2 Sputum positive for pneumocystis jirovecii -2 -1 0 +1 +2 Scoring is based on the frequency of expert selection of each response item. The modal item checked receives full credit (1 point). Partial credit is awarded based on the # of expert responses choosing an option divided by the total number of modal expert responses. Outliers may be excluded. Q 1 Example (1) Diagnostic Questions A 45 year-old female presents to the office complaining of fatigue. No chest pain but some shortness of breath with exertion. Choose the response that best reflects your opinion using the following options: -2 - Highly Unlikely -1 – Less likely than before 0 – Neither more nor less likely +1 – More likely than before Iron deficiency anemia • • Example (2) Investigational Questions Intervention and Methods We conducted one pilot of the SCT in a second year module in Hematology and Oncology in 2014. Our intention with this pilot was to learn about the feasibility of such an approach for teaching preclinical medical students clinical reasoning skills. We expanded on this effort by providing script concordance tests with each module in the year 2015. • 6 Tests; 21 Cases; 150 questions • 3 Approaches: Online at home; Large group setting using clicker questions; Online at OSCE station • Students’ perception survey • Students performance data Results Students Performance If you were thinking of. . . and you find the following. . . the hypothesis the following diagnosis. . . evidence. . . becomes. . . -2 -1 0 +1 +2 Q 3 Q 4 # CA # exp CA exp 1 0 0 0 0 1. 33 0 0 3 1 6 1 2. 67 1. 33 0 0 3 1 1 . 33 0 0 Q 5 # CA exp 0 0 0 3. 5 3 . 5 Q 6 # CA exp 0 0 3 1 0 0 2. 67 3 1 1. 33 2. 67 0 0 1 . 33 0 0 Educational Implications: • • Potential to measure decision-making ability early in medical education and competency evaluation Provide an efficient format that approximates the clinical uncertainty seen in medical practice Next Steps • • Continue to evolve in this process to support clinical reasoning education of our preclinical medical students Consider comparison with other modalities of student assessment