Pitfalls of Writing Learning Objectives and Writing Multiple

Pitfalls of Writing Learning Objectives and Writing Multiple Choice Exam Questions Beth De. Jongh, Pharm. D. , BCPS, BCPP Assistant Professor of Pharmacy Practice 1 Susan Gallanis, MS, CI and CT Instructional Development Specialist, CELT

Learning Objectives �Evaluate the components of learning objectives using the ABCD model. �Revise learning objectives based on the common pitfalls described during this session. �Revise multiple choice exam questions based on the common pitfalls described during this session. �Correctly link learning objectives and multiple choice exam questions to more effectively assess student learning. 2

Resources The CELT Blog: http: //bit. ly/Write. Learn. Obj (bit. ly/Write. Learn. Obj) �Blooms Taxonomy �Sample Verbs �Writing Learning Objectives �How to Write Better Tests �Writing Multiple Choice Questions 3

Learning Objectives 4

Points to Ponder �Is this learning objective measurable, observable, and specific? �Will the learner know what is expected of them after reading the objective? �Does this learning objective focus on important information the student should know? �Guides writing of exam questions �Is this learning objective written at an appropriate level of learning? �Highlights importance of Bloom’s Taxonomy 5

Writing Learning Objectives: A Brief Review �A = Audience �Almost always the student �B = Behavior �What do you want students to be able to do? �Measurable, observable �C = Condition �What the learner will use, have access to, real world condition �“Given the…” �D = Degree �How well, measurement of criteria �Might be quality, accuracy, based on time/speed, or simply “without error” 6

Pitfalls of Writing Learning Objectives �Including more than one verb per learning objective �Lacks focus �Using verbs that are not measurable/observable or specific �Makes assessment difficult �Missing “condition” �Missing “degree” when not assumed �Do not cover a variety of levels of Bloom’s Taxonomy 7

Pitfalls with LOs �Do not cover a variety of levels of Bloom’s Taxonomy �Easy exam versus a very difficult exam �May not discriminate between those who know the material well and those who do not �May not allow lower level learners to demonstrate their knowledge 8

Active Learning Time 9

Pitfalls with LOs: Examples 1&2 �Including more than one verb per learning objective �Lacks focus 10 Multiple verbs One verb Classify a patient with an anxiety disorder (GAD, SAD, PTSD, OCD) and recommend a first-line treatment for the disorder. Classify a patient with an anxiety disorder (GAD, SAD, PTSD, OCD) based on the patient’s presenting symptoms and DSM-5 diagnostic criteria. Evaluate a patient’s characteristics (age, comorbid conditions, concomitant medications, renal/hepatic function) to select an antidepressant when given a patient case. Design a pharmacologic treatment plan for a patient with depression using patient specific characteristics (age, comorbid conditions, concomitant medications, renal/hepatic function).

Pitfalls with LOs: Examples 3&4 �Using verbs that are not measurable/observable or specific �Makes assessment difficult Vague/Ambiguous Verbs Measurable/observable and specific Know the potential drug-drug and drugfood interactions associated with antidepressants (SSRIs, SNRIs, MAOIs, TCAs). Describe the potential drug-drug and drugfood interactions associated with a specific antidepressant (SSRI, SNRI, MAOI, TCA) when given a medication profile. Understand strategies for managing the extrapyramidal symptoms associated with antipsychotics. Recommend strategies for managing the extrapyramidal symptoms associated with antipsychotics based on a patient’s presenting symptoms. Avoid words like: know, understand, comprehend, appreciate, believe, recognize, and memorize 11

Pitfalls with LOs: Examples 5 -8 �Missing “condition” Missing Condition Included Develop a monitoring plan for metabolic side effects associated with second-generation antipsychotics. Apply the monitoring protocol recommended by the 2004 ADA/APA consensus on antipsychotic drugs and obesity and diabetes to a patient taking a second generation antipsychotic for schizophrenia. Describe strategies for managing the adverse effects associated with stimulants. Explain relevant monitoring parameters for the tricyclic antidepressants. Design treatment regimens for supratherapeutic INRs or active bleeds. 12 Describe strategies for managing common adverse effects associated with stimulants for patients with ADHD. Decide which patients taking a tricyclic antidepressant for depression should receive an EKG based on their age and comorbidities. Design a treatment regimen for a supratherapeutic INR or active bleed when given a patient case on warfarin.

Pitfalls with LOs: Examples 9 -11 �Missing “degree” when not assumed Missing Degree Included Calculate a patient’s creatinine clearance using a calculator. Calculate a patient’s creatinine clearance within 5 m. L/min of the correct value when using a calculator. List monitoring parameters required with lithium according to the APA guidelines. List three monitoring parameters required with lithium according to the APA guidelines. Educate a patient on the adverse effects associated with the SSRIs. Educate a patient on the common adverse effects associated with SSRIs within 10 minutes. 13

Multiple Choice Exam Questions 14

Points to Ponder �Does this exam question test important information the examinee should know or does it test a trivial detail or obscure fact? �Would a typical pharmacist in your field be able to answer this question or would they have to look it up? �Highlights importance of linking exam questions to learning objectives �Does this exam question reflect the appropriate level of training? �Is this something a student should know or would this be more appropriate for a resident to understand? �Highlights importance of carefully selecting Bloom’s verbs in learning objectives �Is this exam question written in a manner that is defensible, focused, clear, and unambiguous? 15 Boland R, Lester N, Williams E. Writing Multiple-Choice Questions. Academic Psych 2010; 34(4): 310 -316.

Pitfalls with MC Exam Questions � Writing unfocused stems �Reward recognition over recall �Confusing �Contain irrelevant details �Test more than one concept � Superfluous information in the stem �Detracts from the primary focus �Increases reading time �Increases complex sentences and awkward wording � Negatively phrased questions �Makes question unnecessarily difficult �May not discriminate between those who know the material well and 16 those who do not � Distractors that differ in content, length, or grammar �May not discriminate between those who know the material well and those who do not

Pitfalls with MC Exam Questions Unfocused Stem Focused Stem Which of the following is true of fluoxetine? a. It is structurally a heterocyclic compound. b. A common side effect is weight loss. c. It is fully metabolized in 48 hours. d. It requires a 4 -week wash-out period before starting an MAOI. Switching antidepressant agents from fluoxetine to phenelzine requires a 4 -week wash-out period due to increased risk of developing which of the following? a. Stevens-Johnson Syndrome b. Serotonin Syndrome c. Neuroleptic malignant syndrome d. Agranulocytosis 17 Boland R, Lester N, Williams E. Writing Multiple-Choice Questions. Academic Psych 2010; 34(4): 310 -316.

More Appropriate Use of Cases A 27 -year-old patient with a history of bipolar disorder presents to the hospital with an 8 -day history of insomnia, pressured speech, flight of ideas, hyperactivity and delusions. The first manic episode began at the age of 22. Since then, the patient has had numerous manic episodes (~2 episodes per year) with occasional depressive episodes (~1 episode per year). The past medical history is significant for lower back pain that he adequately treats with as needed naproxen (Naprosyn®) The laboratory work-up was unremarkable other than a sodium level of 130 m. Eq/L (normal 135 -145 m. Eq/L). Which one of the following medications would be MOST appropriate for this patient based on his presenting symptoms, past medical history, and lab results? a. Carbamazepine (Tegretol®) b. Divalproex sodium (Depakote®) c. Lamotrigine (Lamictal®) d. Lithium (Lithobid®) A 17 -year-old comes to your clinic for follow-up monitoring of attention deficit hyperactivity disorder (ADHD) medications. The past medical history is significant for partial complex seizures. The patient’s mother is present at the appointment and reports he/she was caught selling Adderall® (mixed amphetamine salts) at school and is no longer going to receive prescriptions for controlled substances. She asks for your recommendation for an alternative treatment option. Physical Exam: Weight: 81 kg Height: 70 inches Blood Pressure: 100/68 mm. Hg Heart Rate: 52 bpm Respiratory Rate: 16 rpm 18 Which of the following medications is the BEST alternative medication option for this patient? a. Atomoxetine (Strattera®) b. Bupropion (Wellbutrin®) c. Guanfacine ER (Intuniv®) d. Methylphenidate (Concerta®)

Pitfalls with MC Exam Questions Examples Venlafaxine works by reuptake of which of the following neurotransmitters? a. Serotonin and histamine b. Dopamine and norepinephrine c. Serotonin and norepinephrine d. Norepinephrine and acetylcholine A patient with Parkinson’s disease presents to your pharmacy to pick up a refill of their carbidopa/levodopa 10/100 that they take three times daily. You note they should have run out 2 weeks ago and politely question the patient about their adherence. They tell you that they often skip doses because it makes them nauseous. What can be done to help this patient? a. Adjunctive therapy should be considered. b. The dose of carbidopa should be increased to prevent the peripheral conversion of levodopa to dopamine. c. The patient can take this 30 minutes before meals. d. Evaluate for an infectious cause of the nausea. A 23 -year-old patient presents to his psychiatrist with complaints of an increased number of panic attacks over the past month and worries about having more. The patient has been drinking 1/5 th of vodka on a daily basis to “take the edge off”. Which of the following is the BEST treatment option? a. Clonazepam as needed and escitalopram b. Propranolol and escitalopram c. Clonazepam bridge and escitalopram d. 19 Escitalopram

Active Learning Time 20

Pitfalls with MC Exam Questions: Example 1 Unfocused Stem Focused Stem Which of the following pharmacokinetic statements about mood stabilizers is TRUE? a. Carbamazepine (Tegretol®) and oxcarbazepine (Trileptal®) induce their own metabolism which affects the time to steady state concentration. b. Carbamazepine (Tegretol®) decreases divalproex sodium (Depakote®) concentrations through competition at protein binding sites. c. Unbound divalproex sodium (Depakote®) may produce signs and symptoms of toxicity without changes in total drug concentration. d. Divalproex sodium (Depakote®) significantly induces the metabolism of lamotrigine (Lamictal®) thus requiring careful dose titration to prevent rash. Which of the following mood stabilizers increases a patient’s risk for developing a serious rash by inhibiting the metabolism of lamotrigine (Lamictal®)? a. Carbamazepine (Tegretol®) b. Divalproex sodium (Depakote®) c. Lithium (Lithobid®) d. Oxcarbazepine (Tegretol®) 21

Pitfalls with MC Exam Questions: Ex 2 Superfluous Non-superfluous A 30 yo physically healthy patient with a diagnosis of major depressive disorder is currently experiencing another episode of depression. The patient admits to missing doses of his medications 2 -3 times per week due a hectic home and work schedule. A request is made for a long-acting shot, “like with Haldol. ” On assessment, the patient appears generally healthy with stable vital signs and an unremarkable physical examination. You explain that, although there is not currently an antidepressant injection, you can offer the medication with the longest half-life. Which of the following would be prescribed to this patient? a. Citalopram b. Fluvoxamine c. Paroxetine d. Fluoxetine Which of the following SSRIs has the longest elimination half-life? a. Citalopram b. Fluvoxamine c. Paroxetine d. Fluoxetine 22 Boland R, Lester N, Williams E. Writing Multiple-Choice Questions. Academic Psych 2010; 34(4): 310 -316. Question 101 Writing Video

Pitfalls with MC Exam Questions: Ex 3 Superfluous You are working in CUW’s research laboratory and identify a group of people who appear to have a genetic hypercoagulability disorder, in which they produce excess Factor II. You decide you would like to help these people by inventing an oral medication that directly inhibits thrombin (Factor IIa). You begin the drug development process by basing the mechanism off an injectable anticoagulant that is able to directly inhibit thrombin. Which of the following injectable anticoagulants is a direct inhibitor of thrombin? a. Bivalirudin (Angiomax®) b. Enoxaparin (Lovenox®) c. Fondaparinux (Arixtra®) d. Unfractionated heparin 23 Non-superfluous Which of the following injectable anticoagulants is a direct inhibitor of thrombin? a. Bivalirudin (Angiomax®) b. Enoxaparin (Lovenox®) c. Fondaparinux (Arixtra®) d. Unfractionated heparin

Pitfalls with MC Exam Questions: Ex 4 24 Negatively Phrased Positively Phrased A 40 -year-old woman with a history of breast cancer is taking tamoxifen. She presents for treatment of depression. Which of the following antidepressants should not be prescribed? A 40 -year-old woman with a history of breast cancer is taking tamoxifen. She presents for treatment of depression. Which of the following antidepressants is MOST appropriate for this patient? a. b. c. d. e. Desvenlafaxine Amitriptyline Mirtazapine Paroxetine Venlafaxine Sertraline Fluoxetine Paroxetine Venlafaxine Escitalopram Boland R, Lester N, Williams E. Writing Multiple-Choice Questions. Academic Psych 2010; 34(4): 310 -316.

Additional Pitfalls with MC Exam Questions �Using “all of the above” or “none of the above” as distractors �Low ability to discriminate �Can correctly answer without fully understanding the material �All of the above � Can rule it out if you can rule out one distractor �None of the above � Can select it if you know two alternatives are correct 25 Boland R, Lester N, Williams E. Writing Multiple-Choice Questions. Academic Psych 2010; 34(4): 310 -316. Question 101 Writing Video

Additional Pitfalls with MC Exam Questions �Implausible distractors �Avoid silly distractors �Avoid distractors that do not or cannot exist �No consensus on 4 versus 5 answer choices �Use only as many distractors as are plausible � Utilize psychometrics in Exam Soft to help make determination �Research suggests 4 answer choices is optimal � Additional distractors increase difficulty but not discrimination 26 Tresivan M. Estimating the optimum number of options per item using an incremental option paradigm. Educ Psychol Meas 1994; 54: 86 -91. Taylor A. Violating conventional wisdom in multiple choice test construction. College Student J 2005; 39: 141. Swanson D, Holtzman K, Clauser B, et al. Psychometric characteristics and response times for one-best-answer questions in relation to number and source of options. Acad Med 2005; 90: S 93 -96. Boland R, Lester N, Williams E. Writing Multiple-Choice Questions. Academic Psych 2010; 34(4): 310 -316.

Linking Learning Objectives and Exam Questions 27

Points to Ponder �Does the exam question reflect the behavior expected from the learning objective? �Identify the content the students are interacting with �Identify what students are doing with the content �Reevaluate the Bloom’s level and the appropriateness of the specific verb 28

Compare the following: � Learning Objective � Determine State the appropriate washout period required when switching to or from an MAOI to help prevent serotonin syndrome. � The Exam A 36 yo female presents to your clinic with worsening depression. She has been taking the maximum recommended dose of citalopram (Celexa®) for the past three months and has noticed improvement in her symptoms. She has already tried and failed multiple other first-line antidepressants. After speaking to the team psychiatrist, you decide to switch the patient to phenelzine (Nardil®). What is the minimum washout period required between discontinuing citalopram and initiating phenelzine? (1 point) (Objective 5) (Evaluation – Blooms level 6) 2 weeks What potentially life-threatening condition is the patient at risk for if the washout period between citalopram and phenelzine is too short? (1 point) (Objective 3) (Comprehension – Blooms level 2) Serotonin syndrome 29

Pitfalls with Linking LOs and Exam Questions �Exam question and LO do not match Learning Objective Exam Question Revised Exam Question Design a pharmacologic treatment plan for an agitated patient based on patient specific characteristics (e. g. age, comorbid conditions, concomitant medications, behavior). Which of the following agents is available as a short acting injectable? a. Aripiprazole b. Lurasidone c. Quetiapine d. Risperidone An 80 yo patient is brought to the ER by family for increased paranoia. The patient is confused and screaming at family members. Which of the following medications would be the BEST option to start with? Describe potential drug-drug and drug-food interactions when presented with a specific antidepressant and medication profile. Which of the following antidepressants is MOST likely to be involved in CYPmediated drug-drug interactions? a. Bupropion b. Citalopram c. Fluoxetine d. Venlafaxine A patient presents to your clinic with worsening depression despite taking the maximum dose of citalopram for the past 6 months. The patient has already tried and failed multiple other first-line antidepressants. A decision is made to switch the patient to phenelzine. What type of foods should the patient avoid while taking phenelzine? What potentially life-threatening condition is the patient at risk for if he/she does not follow the dietary 30

Pitfalls with Linking LOs and Exam Questions �Intended level of Blooms does not match the difficulty level of the exam question Learning Objective Intended Bloom’s Level Exam Question Compare and contrast the significant adverse effects associated with SSRIs, SNRIs, bupropion, mirtazapine, MAOIs, TCAs, and the serotonin antagonist/reuptake inhibitors for patients with depression. Analysis Which of the following SSRIs is MOST Comprehension likely to cause or worsen constipation in an elderly patient? a. Escitalopram b. Fluoxetine c. Paroxetine d. Sertraline 31 Actual Bloom’s Level

Learning Objectives �Evaluate the components of learning objectives using the ABCD model. �Revise learning objectives based on the common pitfalls described during this session. �Revise multiple choice exam questions based on the common pitfalls described during this session. �Correctly link learning objectives and multiple choice exam questions to more effectively assess student learning. 32

Questions 33