Taking the Guesswork out of Writing Better Multiple

Taking the Guesswork out of Writing Better Multiple Choice Questions Presented to AMATYC February 2, 2015 by Patricia L. Gregg, Ph. D (Patti) Georgia Perimeter College

Multiple choice tests have their place … … but it isn’t necessarily “everywhere”

Question “stem” Should give a clear indication of what Don’t be too broad or generic the question is about (e. g. Which of the following is true? ) Phrase in form of a question Don’t use incomplete statements, fill in the blank, etc. Be consistent with grammar and syntax Don’t give clues about right answer based on the wording of the stem Use simple and positive constructions Avoid convoluted and negative (e. g. Name the most common cause of …) constructions (e. g. Name the least uncommon cause of …) Use scenarios when possible but Don’t include a lot of distracting or irrelevant information (red herrings)

Answer options Categories should be mutually exclusive Avoid overlapping categories Have one clear right (or best) answer Avoid “all of the above” “none of the above” “both a and c” etc. Be consistent with grammar and syntax Don’t give clues about right answer based on the wording of the options Should be same length and construction Don’t make the right answer longer or shorter than other options, or phrased differently Should be equally plausible No obvious wrong answers

Aren‘t plausible distractors “trick questions”? Plausible distractors help to • minimize lucky guesses • ensure that you are truly measuring students’ learning, not their ability to recognize blatantly irrelevant information • diagnose key concepts that students may be misunderstanding or confusing with one another

Poll #1

ITEM ANALYSIS can help you understand • which of your incorrect answer options are plausible “distractors” • whether your test question effectively “discriminates” between high-performing and low-performing students

Can we test higher order thinking with multiple choice questions? Yes, but we need to be vigilant! Too often we claim to be assessing application, analysis, interpretation, etc. but we are really assessing recognition, computation or calculation.

• Not all word problems test higher order skills! • Examples of question items that could measure higher order skills would require students to: Ø Interpret data or graphs Ø Recognize spurious conclusions Ø Distinguish between relevant and irrelevant information

_____triangles have three sides of equal length. a. b. c. d. Isosceles Equilateral Right Scalene Leading with the blank is an awkward construction, especially for non-native English speakers and others with reading challenges.

Which type of triangle has all three sides of equal length? a. Isosceles b. Equilateral c. Right d. Scalene Adapted from Cheung & Bucat, How Can We Construct Good Multiple Choice Items? This question tests the same knowledge as the previous example, but the construction is more familiar.

The mean of a distribution of test scores is the: a. Most frequently occurring score b. 50 th percentile c. Arithmetic average d. Measure of score range

A university developed an aptitude test to use for admission into its Honors Program. The test was administered to a group of seven applicants who obtained the following scores: 70, 72, 80, 89, 94, 98. The mean score on the aptitude test is: a. 72 c. 82 b. 80 d. 90 from Zimmaro, Writing Good Multiple-Choice Exams In order to correctly answer this question, the student must still know the definition of “mean” but also must be able to calculate the mean. However, the question does not assess higher order skills such as application or interpretation.

We want students to see the relevance of course content, but too many details in a word problem end up testing reading speed and comprehension, instead of math skills!

Suppose you are a mathematics professor who wants to determine whether or not your teaching of a unit on probability has had a significant effect on your students. You decide to analyze their scores from a test they took before the instruction and their scores from another exam taken after the instruction. Which of the following t-tests is appropriate to use in this situation? a. Dependent samples. b. Heterogenous samples. c. Homogenous samples. d. Independent samples.

When analyzing your students’ pretest and posttest scores to determine if your teaching has had a significant effect, an appropriate statistic to use is the t-test for: a. b. c. d. Dependent samples. Heterogenous samples. Homogenous samples. Independent samples. Burton et. al. , How to Prepare Better Multiple Choice Items: Guidelines for University Faculty cited in Duquesne University, Good, Better, Best: Multiple Choice Exam Construction

Test items with carefully constructed answer options can help you diagnose what concepts your students may be confusing and/or misunderstanding.

The number of bacteria N(t) present in a culture at time t hours is given by N (t) = 2200(2)t. Find the number of bacteria present when t = 4 hours. (a) 35200 (b) 17600 (c) 70400 (d) 19360000 Test item courtesy of a GPC faculty member

The correct answer is (a). A student would select (b) if they multiplied 2 x 4 instead of raising 2 to the 4 th power. A student would get (c) if they multiplied 2200 by 32, thinking that (2)4 is 32 rather than 16. The student who selects (d) demonstrates no understanding of the order of operation. In this case, the student multiplies 2200 by 2 and then raises it to 4. Since that is not one of the choices, 19360000 (4400 raised to 2) is the closest.

http: //www. act. org/compass/sample/pdf/numerical. pdf

How could this item be stronger? Do we really need 5 answer options? The COMPASS publishers probably do this to minimize the probability of guessing the correct answer, BUT it is really hard to come up with 5 plausible alternatives, so if one is blatantly wrong, we may as well just have 4 options.

Do all the answer options contribute to diagnosis? It seems unlikely that a student would select 0% or 45% unless they had almost no grasp of simple arithmetic, even if they didn’t understand how to calculate percentages.

Does the number of math courses taken contribute to the strength of the question? It may be an unnecessary complication for weaker readers. The skill being testing by this item is calculation of percentages. We want the student to demonstrate their understanding that the second group is 60% of 25% not 60% of the total class; therefore the last remaining group is 40% of 25%.

One possible re-write: This year, 75% of the graduating class of Harriet Tubman High School had been accepted to college. Of the remaining class members, 60% had applied to college but had not been accepted. What percent of the graduating class did not apply to college? The calculation is exactly the same, but the red herrings are removed.

http: //www. act. org/compass/sample/pdf/numerical. pdf

Poll #2

How could this item be stronger? Ensure that all answer options contribute to diagnosis. Eliminate extraneous information e. g. exercise 3 times per week for 20 minutes. IF this test item were intended to measure higher order skills, we would keep the exercise information and possibly add some additional “what if” scenarios with different variables.

PAPERS Burton, Sudweeks, Merrill & Wood (1991, 33 pp. ) https: //testing. byu. edu/handbooks/betteritems. pdf Cheung & Bucat (2002, 10 pp. ) http: //www 3. fed. cuhk. edu. hk/chemistry/files/construct. MC. pdf Zimmaro (2010, 41 pp. ) https: //ctl. utexas. edu/sites/default/files/documents/Writing-Good-Multiple-Choice-Exams-04 -28 -10. pdf INSTITUTIONAL WEBSITES Brigham Young University: https: //testing. byu. edu/handbooks/14%20 Rules%20 for%20 Writing%20 Multiple-Choice%20 Questions. pdf Duquesne University: http: //www. duq. edu/about/centers-and-institutes/center-for-teaching-excellence/teaching-andlearning/multiple-choice-exam-construction Indiana University: http: //www. iub. edu/~best/pdf_docs/better_tests. pdf University of Texas: http: //ctl. utexas. edu/teaching/assessment/planning/multiple-choice Vanderbilt University: http: //cft. vanderbilt. edu/guides-sub-pages/writing-good-multiple-choice-test-questions/ COMMERCIAL PUBLISHERS Censeo Corporation, Guidelines for Writing Effective Tests: A Practical “Short Course” for Test Authors (2007, 9 pp). Document not available online; appears company may no longer be in business; I have a PDF copy to share. Magna Publications: http: //www. facultyfocus. com/articles/educational-assessment/seven-mistakes-avoid-writing-multiple-choice-questions/

Patricia (Patti) Gregg Associate Director, Assessment and Reporting Georgia Perimeter College Patricia. Gregg@gpc. edu 678 -891 -2571 @Dr. Patti. G www. linkedin. com/pub/patricia-gregg