How to write a Multiple Choice Question Matt

How to write a Multiple Choice Question Matt Kelleher

Brad Mathis paraphrase…. . “Doing multiple choice questions is nothing like taking care of patients but writing a multiple choice question is very similar”

Table 7. 1 Constructed-Response Selected-Response Strengths • Non-cued • Easy to create • Logic, reasoning, steps in problem solving • Ease of partial credit scoring • In-depth assessment Weakness • • • Broad representative content • Accurate, objective and reproducible scores • Defensiblity • Accurate, timely feedback • Reusable • Efficient (time, cost, info) • Difficult to write well • Bad public relations (guessing, memorable) Subjective human scoring Limited breath of content Reproducibility issues Inefficient (scoring, testing, information) • Limited psychometrics • Limited quality control Downing and Yudkowsky. Assessment in Health Professionals Education page 150

Selected Response Examples Traditional MCQ’s True-False Alternate-Choice Multiple True-False Traditional Matching Extended Matching Complex Multiple-Choice Testlets or Context-Dependent Item Sets

Multiple Choice Questions (MCQ’s) There are 4 main components to every MCQ. Objective –important part and best place to start Stem and Lead-in Options Explanation

Best Evidence in practice for creating effective and defensible MCQ’s – Content/STEM 1. Focused, items should test specific content/objective and single specific mental behavior 2. Based on important content; avoid trivial content 3. Avoid using exact textbook language – “buzz words” (recall/recognition) 4. Content of each item should be independent 5. Avoid over-specific or over-general language (usually, frequently, commonly, often, always, never) 6. No opinion based items Table 7. 3 Taxonomy of Multiple-Choice Item Writing Guidelines. Adapted summary from Haldyna, Downing, and Rodriquez 2002

Best Evidence in practice for creating effective and defensible MCQ’s - STEM 7. Should NOT be tricky or unnecessarily complex (thyroid ex. ) 8. Keep vocabulary simple (avoid all abbreviations/acronyms) 9. Be very clear and direct in the stem question (i. e. which of the following is the MOST likely diagnosis) “best next vs next best” 10. Central idea should be in the stem, not the options 11. Avoid window dressing (excessive verbiage, as short as possible, all sentences should be relevant) 12. Word the stem positively (avoid NOT, EXCEPT, etc. ) Table 7. 3 Taxonomy of Multiple-Choice Item Writing Guidelines. Adapted summary from Haldyna, Downing, and Rodriquez 2002

A few more of my rules 1. Be as realistic as possible (case-based) 2. Should be answerable without options OR options must be 100% true or false 3. Patients in MCQ don’t lie 4. Try to test the application of knowledge (avoid recall of facts) …. . if while writing the question you think this is a straightforward concept – how can I make it complicated, this is a reason to pause!

Best Evidence in practice for creating effective and defensible MCQ’s - OPTIONS As many effective choices as you can (research says three is adequate) One indisputably correct option (doesn’t mean other options are completely wrong) Randomize choices Choices should be related but independent, not overlapping Homogenous choices in content, grammar, structure Keep length of choices equal Avoid none or all of the above Phrase choices positively (avoid negatives) All distractors should be plausible but clearly incorrect Use typical errors of learners to write distractors Avoid clues • always, never, completely, absolutely • clang associations (choices are identical or resemble words in stem) • grammatical inconsistencies • conspicuous correct choice • pairs, triplets of options • absurd or ridiculous options Table 7. 3 Taxonomy of Multiple-Choice Item Writing Guidelines. Adapted summary from Haldyna, Downing, and Rodriquez 2002

Explanation • Be very clear and concise • Explain why the correct option was correct first (Main Explanation) • For each distractor/incorrect answer briefly explain why it was specifically incorrect • When possible attach a reference for your source and where someone can go to learn more

What’s wrong? Which of the following is true about pseudogout? A. It occurs frequently in women. B. It is seldom associated with acute pain in a joint. C. It may be associated with a finding of chondrocalcinosis. D. It is clearly hereditary in most cases. E. It responds well to treatment with allopurinol.

What’s wrong? Too Vague? Not focused Which of the following is true about pseudogout? A. It occurs frequently in women. B. It is seldom associated with acute pain in a joint. C. It may be associated with a finding of chondrocalcinosis. D. It is clearly hereditary in most cases. E. It responds well to treatment with allopurinol.

What’s wrong? A 60 -year-old man is brought to the emergency department by the police, who found him lying unconscious on the sidewalk. After ascertaining that the airway is open, the first step in management should be intravenous administration of A. examination of cerebrospinal fluid B. glucose with vitamin B 1 (thiamine) C. CT scan of the head D. phenytoin E. diazepam

What’s wrong? A 60 -year-old man is brought to the emergency department by the police, who found him lying unconscious on the sidewalk. After ascertaining that the airway is open, the first step in management should be intravenous administration of A. examination of cerebrospinal fluid B. glucose with vitamin B 1 (thiamine) C. CT scan of the head D. phenytoin E. Diazepam Example: the first step should be the intravenous administration of which medication? …. (several options and maybe two meds would work but one comes in IV preparation and one does not) – Testing the wrong construct!

Ms. James is a 57 yo AAF with PMH DM-2 (a 1 c 8. 2%), HLD, HTN, and obesity. She presents to your office for follow-up. She has no complaints. She takes ASA 81 daily, atorvastatin 40 daily recently increased, and metoprolol tartrate 12. 5 bid but sometimes forgets to take metoprolol. BP is 154/82, HR 87 today. Previous blood pressures the last three visits over the year have been 148/79, 158/86, and 156/90. She has trace swelling of the lower extremities and PE is otherwise unremarkable. What is the next best step in managing Ms. James’ BP? A. Encourage diet and lifestyle modification and switch metoprolol 12. 5 bid to HCTZ 25 daily. B. Encourage diet and lifestyle modification and increase metoprolol from 12. 5 bid to 25 bid. C. Encourage diet and lifestyle modification and switch metoprolol 12. 5 bid to HCTZ 25 daily and amlodipine 10 daily. D. Switch metoprolol 12. 5 bid to Lisinopril 40 daily. E. Start amlodipine 2. 5 daily.

Ms. James is a 57 yo AAF with PMH DM-2 (a 1 c 8. 2%), HLD, HTN, and obesity. She presents to your office for follow-up. She has no complaints. She takes ASA 81 mg daily, atorvastatin 40 daily recently increased, and metoprolol tartrate 12. 5 bid but sometimes forgets to take metoprolol. BP is 154/82, HR 87 today. Previous blood pressures the last three visits over the year have been 148/79, 158/86, and 156/90. She has trace swelling of the lower extremities and PE is otherwise unremarkable. What is the next best step in managing Ms. James’ BP? A. Encourage diet and lifestyle modification and switch metoprolol 12. 5 bid to HCTZ 25 daily. B. Encourage diet and lifestyle modification and increase metoprolol from 12. 5 bid to 25 bid. C. Encourage diet and lifestyle modification and switch metoprolol 12. 5 bid to HCTZ 25 daily and amlodipine 10 daily. D. Switch metoprolol 12. 5 bid to Lisinopril 40 daily. E. Start amlodipine 2. 5 daily.

What’s wrong? In patients with advanced dementia, Alzheimer’s type, the memory defect A. can be treated adequately with phosphatidylcholine (lecithin) B. could be a sequela of early parkinsonism C. is never seen in patients with neurofibrillary tangles at autopsy D. is never severe E. possibly involves the cholinergic system

What’s wrong? In patients with advanced dementia, Alzheimer’s type, the memory defect A. can be treated adequately with phosphatidylcholine (lecithin) B. could be a sequela of early parkinsonism C. is never seen in patients with neurofibrillary tangles at autopsy D. is never severe E. possibly involves the cholinergic system Main problem with this question is the lack of focus in the stem! The verb should be in the lead in of the stem not the options….

What’s wrong? A 58 -year-old man with a history of heavy alcohol use and previous psychiatric hospitalization is confused and agitated. He speaks of experiencing the world as unreal. This symptom is called A. depersonalization B. derailment C. derealization D. focal memory deficit E. signal anxiety

What’s wrong? A 58 -year-old man with a history of heavy alcohol use and previous psychiatric hospitalization is confused and agitated. He speaks of experiencing the world as unreal. This symptom is called A. depersonalization B. derailment C. derealization D. focal memory deficit E. signal anxiety

What’s wrong with these options? Secondary gain is 1. synonymous with malingering 2. a frequent problem in obsessive-compulsive disorder 3. a complication of a variety of illnesses and tends to prolong many of them 4. never seen in organic brain damage

What’s wrong with these options? Secondary gain is 1. synonymous with malingering 2. frequent problem in obsessive-compulsive disorder 3. a complication of a variety of illnesses and tends to prolong many of them 4. never seen in organic brain damage Longer, more specific, more complete one is correct It is also the only double option

Stem critique A 60 yo man with pmhx of htn, dm, hld and seasonal allergies presents with difficulty swallowing. He has difficulty with solids only. He only has trouble swallowing meat and potatoes but not juice or water. He states, “this has been going on for about 2 years. ” He also has recurrent food impaction and occasional burning. He takes hydrochlorothiazide, lantus, Crestor and Claritin. His greatgrandfather had lung cancer. He claims to not smoke or drink. What is the most likely diagnosis? 30 year old man with a history of seasonal allergies and recurrent food impaction presents to clinic with dysphagia to solids. He denies a history of heartburn. He does not smoke or drink alcohol. What is the most likely diagnosis? Objective: Diagnose Eos Esophagitis Answer options: A. GERD induced peptic stricture B. Eosinophilic esophagitis C. Achalasia D. Esophageal cancer

What’s wrong? Which of the following is correct regarding when to initiate antibiotic prophylaxis in an HIV infected patient? A. CD 4 conuts <50 starts Bactrim B. CD 4 counts <50 start Azithromycin C. Initiate azithromycin when the CD 4 count is less than 150 D. CD 4 count <100 start isoniazid E. CD 4 count <200 start azithromycin F. None of the above correct answer is the option that has the most in common with the other options

What’s wrong with these options? 1. Admit to general medicine floor for hypertensive emergency/crisis and restart home medications – HCTZ 25 daily and Lisinopril 40 daily, lower DBP to <160/90 mm. Hg in 16 hours. 2. Admit to ICU for hypertensive emergency/crisis and start nicardipine 5 mg/hour and titrate to lower DBP by 10 -15% (or 110 mm. Hg) in 30 -60 minutes. 3. Admit to ICU for hypertensive emergency/crisis and bolus with Lasix 80 mg IV x 1 and start Lasix gtt at 10 mg/h. 4. Admit to general medicine floor for hypertensive urgency and use hydralazine 10 mg iv every 4 -6 hours PRN for SBP >180 or DBP >105 mm. Hg. 5. Admit to ICU for hypertensive emergency/crisis and start nicardipine 5 mg/hour and titrate to lower DBP by 25% over 16 hours.

What’s wrong? Following a second episode of PID, what is the likelihood that a woman will become infertile? A. Less than 20% B. 20 to 30% C. Greater than 50% D. 90% E. 75%

What’s wrong? Severe obesity in early adolescence A. usually responds dramatically to dietary regimens B. often is related to endocrine disorders C. has a 75% chance of clearing spontaneously D. shows a poor prognosis E. usually responds to pharmacotherapy and intensive psychotherapy Research has shown that vague frequency terms are not consistently defined or interpreted, even by experts.

A 51 yo Caucasian male presents to your office for follow-up of slightly elevated aminotransferases. His past medical history includes obesity (BMI 40), hyperlipidemia, and type 2 diabetes mellitus. He drinks 4 -5, 12 oz. beers on Sundays. Medications include metformin 1000 mg twice daily, atorvastatin 40 mg daily, HCTZ 12. 5 mg daily, and aspirin 81 mg daily. Vital signs: afebrile, HR 79, BP 138/94. Physical exam reveals normal cardiac and pulmonary exam, notable central adiposity. Labs: WBC 4. 5, Hgb 13. 7, Platelets 129, 000, AST 68, ALT 32, Total protein 7. 2, Total bilirubin 1. 4, Alkaline phosphatase: 145, Albumin 3. 5, INR 1. 2, Hgb A 1 c 7. 9%, Na 139, K 3. 7, Cl 105, HCO 3 25, BUN 18, SCr 0. 9, Ca 8. 2, Mg 1. 9 Right upper quadrant ultrasound is pending. What would you tell this patient regarding the diagnosis of non-alcoholic fatty liver disease (NAFLD)? A. Recommend lifestyle modification including weight loss of 5 -10%, reduce alcohol intake, continuation of statin and metformin for hyperlipidemia and diabetes mellitus, respectively, and start vitamin E. B. Recommend lifestyle modification including weight loss of 5 -10%, reduce alcohol intake, continuation of statin and metformin for hyperlipidemia and diabetes mellitus, respectively, and start pioglitazone. C. Recommend lifestyle modification including weight loss of 5 -10%, reduce alcohol intake, continuation of metformin for diabetes mellitus and stop atorvastatin. D. Recommend lifestyle modification including weight loss of 5 -10%, reduce alcohol intake, continuation of statin and metformin for hyperlipidemia and diabetes mellitus, respectively, and he may need a liver biopsy. E. NAFLD is a less common cause of liver disease in the United States and he should not be worried about this.

Questions?

What do you think about the stem? Diagnose multiple myeloma 66 year black man is seen in clinic complaining of back pain that is made worse with movement. He has no signs of sensory loss, bowel/bladder incontinence, radiculopathy or neuropathy. He has had a 6 lb weight loss, generalized weakness, and fatigue. Spinal Xrays reveal a lytic lesion. Which of the following diagnostic tests would be best to order next?