Script Concordance Test Jean Paul Fournier Dpartement de

Script Concordance Test Jean Paul Fournier Département de Pédagogie Médicale Faculté de Médecine de Nice Sophia Antipolis, France

Clinical reasoning Semantic networks Instances Prototypes Patient’s complaint Problem representation Early problem identification (non analytic process) Compiled knowledge Early generation of relevant hypothesis Script activation Hypothesis check Clinical data interpretation Micro judgements (slots) Analytic process Quest for new clinical / paraclinical data Tempus Program March 12, 2012 Nendaz M. Pédagogie Médicale 2005 Pellacia Th. Annales Françaises de Médecine d’Urgence 2011

Clinical reasoning (2) No semantic network High complexity case No hypothesis Epidemiology Prevalence Pathophysiology … Systematic approach Forward reasoning Diagnosis Tempus Program March 12, 2012 Pellacia Th. Annales Françaises de Médecine d’Urgence 2011

Clinical reasoning (3) Non analytic process Description Pattern recognition Intuitive identification of a typical signs configuration (= pattern), correspounding to diagosi(e)s Instances Intuitive identification of a déjà vue situation and immediate diagnosi(e)s generation Tempus Program March 12, 2012 Pellacia Th. Annales Françaises de Médecine d’Urgence 2011

Clinical reasoning (4) Analytic process Description Hypothetico deductive process The physician consciously tries to confirm / infirm his early hypotheses, with an oriented quest of clinical / paraclinical signs Forward reasoning The physician goes from clinical / paraclinical data to the problem solving, through application of causal or conditionnal rules Tempus Program March 12, 2012 Pellacia Th. Annales Françaises de Médecine d’Urgence 2011

A woman, aged 82, is referred to the ED from her retirement house for a right lateral acute chest pain with shortness of breath. She suffers an ovarian carcinoma, in palliatie care. She does not move from her bed, due to intractable pain, for which she recently began morphine sulfate. Temperature is 38, 2°C, BP is 13580 mm Hg, heart rate is 130 bpm, saturation is 90% with 6 l/min of oxygen via a face mask. Last creatinine level was 150 μmol/l. She has venous disease sequellae and bilateral pitting edema. You think of a possible pulmonary embolism. Physician 1 Physician 2 Physician 3 D dimers assay unesuseful in such a context D dimers assay D dimers > 5000 g/ml Venous ultrasound Femoral vein thrombosis CT pulmonary angiogram Bilateral proximal pulmonatry embolism Heparin Tempus Program March 12, 2012 With an high pre test probability and D dimers > 4000 : pulmonary embolism probability : 88 p. cent CT pulmonary angiogram benefit/risk ?

What happened ? v 3 different pertinent approaches for the resolution of the same problem Script activation v 3 different pertinent approaches but with the same result : heparin therapy ; Clinical reasoning variability v Uncertainty : D dimer results interpretation in an old wman with ovarian cancer, CT pulmonary angiogram risk/benefit in an old woman with chronic kidney impairment ; Uncertainty v Impact of new data in initial hypotheses (micro judgements). Ponderation Tempus Program March 12, 2012

Script theory v Script : Compiled knowledge network (clinical, biological, imaging, …) used for a clinical problem solving v Scripts are elaborated from early patients encounters ; v The more frequent and diverse are encounters, the richest are scripts ; v Scripts cannot be transfered ; v They vary from a physician to anaother. Charlin B. Acad Med 2000 Tempus Program March 12, 2012

Script Concordance Test and functionnality of scripts in simulated clinical situations in an uncertainty context v Standardized assessment tool ; v Clinical situations ; v Ill-defined situation and context : no consensus, no evidence-based medicine ; v Assessment of compiled clinical reasoning ; v Reference to an expert panel (aggregate scoring). Tempus Program March 12, 2012

SCT : Principles v Comparing the attributed impact of a new information on initial hypothesis between experts and students ; v Experts take the same test as students (same questions, same timing) and serve as the reference panel ; v v All experts answers are accepted (with the restriction of discordant answer or expert) ; v The closest to the experts’ choice is the students’ choice, the most performing are students.

The SCT Short clinical vignette A woman, aged 82, is referred to the ED from her retirement house for a right lateral acute chest pain with shortness of breath. She does not move from her bed, due to intractable pain, for which she recently began morphine sulfate. Temperature is 38, 2°C, BP is 135 -80 mm Hg, heart rate is 130 bpm, saturation is 90% with 6 l/min of oxygen via a face mask. You think of a possible pulmonary embolism. Question Anchor If you were planing to prescribe… And then you find… The impact in your hypothesis becomes… A D-dimer assay An ongoing ovarian carcinoma in palliative care -2 -1 0 +1 +2 A venous ultrasound A swollen painful left calf -2 -1 0 +1 +2 A CT pulmonary angiogram Créatinine level of 150 µmol/l -2 -1 0 +1 +2 -2 : Very unuseful -1 : Unuseful Pertinent hypothesis 0 : Neither more or less useful New relevant information Tempus Program March 12, 2012 +1 : Useful +2 : Very useful Likert scale : assessment of the new information on the initial hypotheses (Micro judgement) Item

SCT vs rich-context MCQ

Is SCT an efficient assessment tool ? v A valid tool ? ü Face validity : depends on test construction (blueprint) ; ü Construct validity : Ø identification of juniors vs seniors ; Ø prediction of scores for tests exploring the same domain ; v A reliable test ? Ø Cronbach alpha from 0, 79 to 0, 82. MCQ : 0, 76 à 0, 93 (2 to 8 hour testing) Patient Management Problem : 0, 69 à 0, 82 (4 to 8 hour testing) Key Feature Case : 0, 66 à 0, 79 (4 à 8 hour testing). Tempus Program March 12, 2012

SCT among other assessment tests ? v MCQ ; v Clinical cases for ECN. Tempus Program March 12, 2012

SCT vs rich-context MCQ 9523 Clinical ED situations Recommandations NBME Recommandations B. Charlin 2002 60 MCQ 60 SCT Each situation appeararing in the 2 formats Test taken by 16 EM residents Tempus Program March 12, 2012

SCT vs rich-context MCQ Tempus Program March 12, 2012

SCT vs true/false MCQ v 106 medical students from 3 rd to 6 th year ; v Endocrinology tutorial ; v SCT vs true/false MCQ : v Significant correlation in 3 rd and 4 th year ; v Correlation for the 4 years with the knowledge level estimate (true vs false answers). Tempus Program March 12, 2012 Collard A. Med Educ 2009

SCT vs ECN clinical case v 18 medical students from 4 th to 6 th year ; v ECN clinical case ; v 26 questions SCT. Gibot S. Pédagogie Médicale 2008 Tempus Program March 12, 2012

SCT vs ECN clinical case R = 0, 55 Gibot S. Pédagogie Médicale 2008 Tempus Program March 12, 2012

SCT vs other assessment tools v Significant correlation, but low clinical significance : : some common points (assessment domain), but the SCT assesses some other aspects ; v Combining several assessment formats is a key-point in assessment. Tempus Program March 12, 2012

Usual assessment tools Performance Direct observation, peers, video, audits Action Competences Standardized oral examination, OSCE, mini CEX simulation, SCT Knowledge utilization Rich context MCQ, SCT, PMP, key-features Knowledge True/false MCQ, essay-questions Compétence Miller Acad. Med. 1990 Tempus Program March 12, 2012

How is it accepted ? Tempus Program March 12, 2012 Bounouffe C. Am J Pharm Educ 2010

Technical aspects v Blueprint ; v Preparation ; v Anchor ; v Experts panel ; v Scoring ; v Quality control. Tempus Program March 12, 2012

Blueprint 4 points : v What do we do ? Certification ? Formative aspect ? Difficulty level ? Prevalence ? v Situations selection ? Severity ? Diagnosis v Questions formats Treatment Management Reliability v Items and questions number Feasibility Tempus Program March 12, 2012 Validity

Blueprint Cardio Pneumo Neuro Gastro Dg X X Trt X X Management X X X Metabol. Rheumato Geriatr. Gyneco Psy Traumato X X X Test developers + experts Tempus Program March 12, 2012 X

Blueprint Cardio Pneumo Neuro Gastro Dg Chest pain Acute shortness of breath Headache Abdomin al pain Trt CHF Pulmonary embolism Seizure Gastric bleeding Management ACS COPD exacerbation Stroke Metabol. Rheumato Geriatr. Gyneco Psy Trauma Mono arthritis Altered general condition Ectopic pregnancy Confusion Neck trauma Dehydrat. Test developers + experts Tempus Program March 12, 2012 Agitation

How many cases, how many questions ? v Case specificity ; v Many cases with one question ? Format homogeneity v Few cases with many questions ? v 1 case with n questions ? v 20 cases and 60 questions : Cronbach alpha ≥ 0, 70 Tempus Program March 12, 2012

How many cases, how many questions ? Tempus Program March 12, 2012 Gagnon R. Adv Health Sci Educ 2008

How many cases, how many questions ? Tempus Program March 12, 2012 Gagnon R. Adv Health Sci Educ 2008

Preparation : situations, questions Principles : v Clinical situations selection, fitted to students level ; v Relevant hypothesis selection in that context ; Test developers + experts v New relevant informations selection, in that context Uncertainty Tempus Program March 12, 2012 Page G. Academic Medicine 1995

Clinical vignette and questions preparation Vignette : relevant informations : v Age, gender ; v Complaint, adlission, hospitalization reason ; v Context : ED, office, ward, … v Clinical data, … Tempus Program March 12, 2012 1 to 2 writers

Clinical vignette and questions preparation Relevant new informations : v Clinical data ; v Biological test ; 1 to 2 writers v Imaging ; v Video, … Tempus Program March 12, 2012

Clinical vignette and questions preparation Vignette, hypothesis and new informations : v Sweeping the five points of the Likert scale ; v Well-balanced variability level ; v All cases are independant ; v All questions are independant ; v One format by case (diagnosis, treatment, …). Tempus Program March 12, 2012 Charlin B. Medical Education 2006

Question discrimination power and variability Tempus Program March 12, 2012 Charlin B. Medical Education 2006

Lickert scale - Anchor Recommandations : v Explicit the significance of each point of the Lickert scale, particularly the neutral point ( « 0 » ) ; v Keep the same presentation. -1 : Unlikely -2 : Very unlikely 0 : Neither more or less likely Tempus Program March 12, 2012 +1 : More likely +2 : Very likely

Lickert scale - Anchor Lickert scales : v Diagnosis anchor : If you were thinking of the following diagnosis… And then you find… This new information makes your diagnosis… Diagnosis option New information -2 -1 0 +1 +2 -1 : Unlikely -2 : Very unlikely 0 : Neither more or less likely Tempus Program March 12, 2012 +1 : More likely +2 : Very likely

Lickert scale - Anchor Lickert scales : v Anchor for test(utility) : If you were considering the usefulness of the following test… And you find… This new information makes the following test… Investigative option New information -2 -1 0 +1 +2 -1 : Useless -2 : Less useful v Anchor for test(benefit/risk) : 0 : Neither morer or less useful +1 : Useful +2 : Very useful If you were considering the benefit/risk ratio of the following test… And you find… This new information makes the test… Investigative option New information -2 -1 0 +1 +2 -1 Contraindicated -2 : Strongly contraindicated 0 : Neither more or less indicated Tempus Program March 12, 2012 +1 : Indicated +2 : Strongly indicated

Lickert scale - Anchor Non appropriate option Lickert scales : v Anchor for test(utility) : If you were considering the usefulness of the following test… And you find… This new information makes the following test… Investigative option New information -2 -1 0 +1 +2 -1 : Useless -2 : Less useful Inadequate prescription v Anchor for test(benefit/risk) : Adequate prescription 0 : Neither morer or less useful +1 : Useful +2 : Very useful If you were considering the benefit/risk ratio of the following test… And you find… This new information makes the test… Investigative option New information -2 -1 0 +1 +2 -1 Contraindicated -2 : Strongly contraindicated 0 : Neither more or less indicated Tempus Program March 12, 2012 +1 : Indicated +2 : Strongly indicated

Lickert scale - Anchor Likert scales : v Treatment anchor (utility) : If you were considering the usefulness of the following treatment… And you find… This new information makes the following treatment… Treatment option New information -2 -1 0 +1 +2 -1 : Useless -2 : Less useful v Treatment anchor (benefit/risk) : 0 : Neither morer or less useful +1 : Useful +2 : Very useful If you were considering the benefit/risk ratio of the following treatment… And you find… This new information makes the following treatment… Treatment option New information -2 -1 0 +1 +2 -1 Contraindicated -2 : Strongly contraindicated 0 : Neither more or less indicated Tempus Program March 12, 2012 +1 : Indicated +2 : Strongly indicated

Lickert scale - Anchor Non appropriate option Échelles : v Treatment anchor (utility) : Si vous considériez l’utilité du traitement suivant Et qu’alors vous trouvez Cette nouvelle information rend le traitement Treatment option New information -2 -1 0 +1 +2 -1 : Useless -2 : Less useful Inadequate prescription v Treatment anchor (benefit/risk) : Adequate prescription 0 : Neither morer or less useful +1 : Useful +2 : Very useful If you were considering the benefit/risk ratio of the following treatment… And you find… This new information makes the following treatment… Treatment option New information -2 -1 0 +1 +2 -1 Contraindicated -2 : Strongly contraindicated 0 : Neither more or less indicated Tempus Program March 12, 2012 +1 : Indicated +2 : Strongly indicated

Lickert scale - Anchor Lickert scales : v Prognosis anchor : Prognosis item New information The prognosis becomes Prognosis item New information -2 -1 0 +1 +2 -1 : Worsened -2 : Very worsened 0 : Never more or less modified Tempus Program March 12, 2012 +1 : Improved +2 : Very improved

Experts panel Roles : v Test preparation ; v Clinical vignette and questions validation ; v Scoring. Tempus Program March 12, 2012 2 different groups

Experts panel v Quality : experts in the considered domain ; v Number : 10 to 20 minimum. Tempus Program March 12, 2012 Gagnon R. Medical Education 2005

Mean reliability following the experts number Tempus Program March 12, 2012 Gagnon R. Medical Education 2005

Expert panel composition Teaching physicians vs physicians ? v SCT in community medicine ; v Comparison of scores ; v Panel : teaching physicians vs physicians in a CME program ; v Perfect concordance of the scores (R = 0. 98) ; v Scores attributed by physicians more elevated (p = NS). Tempus Program March 12, 2012 Charlin B. Medical Teacher 2007

Scoring : v. Panel choice : v Modal transformation of the panel choice : v Residents’ choice : v Residents’ scores : Tempus Program March 12, 2012

Scoring Excel® scoring calculator available : www. cme. umontreal. ca/tcs Tempus Program March 12, 2012

Summary Test preparation v Blueprint v Clinical vignettes v Hypotheses v New informations v Scoring v Pass/fail decision Experts 1 v Clinical vignettes v Hypotheses v New informations Experts 2 Validation v Scoring (reference panel scoring) Tempus Program March 12, 2012

Quality control A priori quality control : Tempus Program March 12, 2012 Caire F. Neurochirurgie 2004

Quality control A posteriori quality control : item analysis : v Difficulty index ; v Discrimination index ; v Question impact on the test reliability (Cronbach alpha) ; v Etc… Tempus Program March 12, 2012

Thank you for your attention !