Crafting the Question Get the Most Out of

Crafting the Question: Get the Most Out of Your Student Response System Ann Agee, San Jose State University LOEX 2016, May 5 -7, Pittsburgh, Pennsylvania

Crafting the Question • • Create effective multiple-choice questions Create questions at different levels of learning Use a variety of question formats Understand time management and best practices

Clickers

Free Software

None of the Above

Why Use Them? • • Encourage participation Check for understanding Direct instruction Promote discussion (Vanderbilt University Center for Teaching)

Anatomy of a Multiple-Choice Question

1. Choose the resource that has details about an event as it happens. A) Book B) Scholarly Article C) Encyclopedia D) Internet news site STEM OPTIONS

1. Choose the resource that has details about an event as it happens. A) Book B) Scholarly Article DISTRACTORS C) Encyclopedia D) Internet news site KEY

Stem Options Distractors Key = = Question All of the answers Wrong answers Right answer

Crafting the Stem

A stem should relate directly to a learning objective. (Mc. Donald, 2002, p. 88)

How is this question related to my objective? What do I want to achieve by asking this question? (Premkumar & Coupal, 2008, p. 146)

Stem Guidelines • Keep the central idea in the stem, not the options • Avoid complex language • Avoid negatives, such as NOT or EXCEPT (Haladyna, Downing, & Rodriguez, p. 312)

Effective Stems • • • Complete Succinct Focused Positive Grammatically correct (Mc. Donald, p. 92 -94)

Stem Formats • Question What is a resource that has details about an event as it happens called? • Completion A resource that has details about an event as it happens is called… • Instructional Choose the resource that has details about an event as it happens. (Mc. Donald, 88 -89)

Crafting the Options

Keep the Options Balanced DISTRACTORS KEY (Mc. Donald, p. 90)

Effective Options • • Positive Succinct Grammatically correct Distinct (Mc. Donald, p. 95 -98)

Effective Key • Make sure there is just one • Make sure it is distinct • Vary the placement (Mc. Donald, p. 107)

Effective Distractors • Make them plausible • Use common misconceptions • Avoid “All of the above” and “None of the above” (Mc. Donald, p. 107; Haladyna et al. , p. 319)

How Many? (Haladyna et al. , p. 318)

Pop Quiz

u Problem? Paraphrasing: A. Synthesizing the original passage and writing it in your own words. B. Changing the sentence order of the original passage. C. Replacing certain words from the original passage.

u Better Which of the following best describes acceptable paraphrasing? A. Synthesizing the original passage and writing it in your own words. B. Changing the sentence order of the original passage. C. Replacing certain words from the original passage.

v Problem? I would have liked to spend more time on: A. B. C. D. How to search Psyc. INFO How to use Ref. Works How to search the library catalog None of the above. I feel ready to research.

v Better I would have liked to spend more time on how to: A. B. C. D. Search Psyc. INFO Use Ref. Works Search the library catalog None of the above. I feel ready to research.

w Problem? Which of the following media do you NOT need to cite? A. A chart from your textbook. B. A video you saw on MTV. C. A photo you took with your digital camera.

w Better You do not need to cite one of the following media. Which one? A. A chart from your textbook. B. A video you saw on MTV. C. A photo you took with your phone.

x Problem? Using AND to join search terms has the effect of A. Narrowing your search results B. Broadening your search results C. Searching for two terms at the same time

x Better Using AND to join search terms has the effect of: A. Narrowing your search results B. Broadening your search results C. Removing a term from your search results

Levels of Learning

• • Level 1 Level 2 Level 3 Level 4 Remember Understand Apply Analyze

Remember The library service that allows you to borrow material from another library is called: A. Interlibrary Loan B. Get. Text C. Scholar. Works

Verbs for the Remember Level • • Define Identify Know List Name Recall Recognize • • • Relate Reproduce Select State Tell (Mc. Donald, 2002, p. 233)

Understand What is an empirical study? A. An article that surveys the current literature on a topic. B. An article that provides an analysis of previously published statistics. C. An article that is based on observed facts or behaviors rather than theory.

Verbs for the Understand Level • • Change Compare Convert Describe Differentiate Discuss Distinguish Estimate • • Explain Extrapolate Illustrate Infer Interpret Outline Rank Rearrange (Mc. Donald, 2002, p. 234) • • • Reorder Rephrase Reword Summarize Translate Transform

Apply Which of the options below is an academic journal article citation? A. Gladstein, M. R. (2000). Atlas Shrugged: Manifesto of the mind. New York: Twayne. B. Gladstein, M. R. (2000). Atlas Shrugged: Manifesto of the mind. American Communist History, (9)1, 104106. C. Gladstein, M. R. (2000, April 2). Atlas Shrugged: Manifesto of the mind. Retrieved from http: //randfans. com/2007/04/atlas_shrugs_ba. html/

Verbs for the Apply Level • • Arrange Avoid Classify Construct Develop Demonstrate Determine • • Ensure Examine Identify Illustrate Locate Modify Relate (Mc. Donald, 2002, p. 235) • Revise • Utilize

Analyze Which of the following searches will give the best results for this topic: Is there a relationship between obesity and online gaming? A. obesity OR online gaming B. obesity AND online gaming C. obesity AND online gam*

Verbs for the Analyze Level • • • Analyze Associate Categorize Compare Contrast Correlate Delineate Deduce Detect • • • Determine Differentiate Diagram Distinguish Divide Estimate Examine Infer Investigate (Mc. Donald, 2002, p. 236) Look for trends Order Question Recognize error • Separate • Solve • Verify • •

Question Formats

Picture Questions Which of these buttons will often take you to the full text of an article? A. B. C.

Picture Questions With this search, your results would contain which of the following words? A. Teaches, Teaching, Teacher B. Teaches, Teaching, Teak C. Teaches, Instructs, Tutors

Video Questions What is the question portion of a multiplechoice question called? A. Stem B. Leaf C. Branch

Confidence Level Questions How do you feel about your ability to find peerreviewed articles on your topic? A. Not confident B. Sort of confident C. Confident (Bruff, 2016)

Student Perspective Questions How many library research classes have you had at SJSU? A. B. C. D. None One Two Three or More (Bruff, 2016)

Student Perspective Questions Doing research in the library databases makes me feel: A. Anxious B. Nothing special C. Exhilarated (Bruff, 2016)

Choose Your Own Adventure Your roommate asks if she can turn in a paper you wrote last semester for an assignment she’s really behind on. What do you say? A. “Sure. What are friends for? ” B. “No. That’s plagiarism. ” (Bruff, 2016)

Let’s Talk About This Turning in someone else’s work as your own violates the campus policy on academic honesty. By letting your roommate use your paper, she could fail her assignment or the course. Both of these are consequences of plagiarism at SJSU. How would anyone know it was your paper? SJSU uses Turnitin, a plagiarism detection service. It compares student papers with information found on the Internet as well as its own database of previously submitted papers, including yours.

Good Choice! Turning in someone else’s work as your own violates the campus policy on academic honesty. How would anyone know it was your paper? SJSU uses Turnitin, a plagiarism detection service. It compares student papers with information found on the Internet as well as its own database of previously submitted papers, including yours.

Choose Your Own Adventure What would you like to learn about first? A. Setting up Google Scholar to search the databases B. Using Ref. Works to automatically generate your Works Cited page C. Using the library catalog to find books and articles

Assessment Questions I would have liked to spend more time on how to: A. B. C. D. Search Psyc. INFO Use Ref. Works Search the library catalog None of the above. I feel ready to research.

Clicker Pyramid Category F Scores 1 2 Final Category D Category A Category E Category B (ITS, The University of Iowa, 2016) Category C

Trivia Multiple-choice tests were first used on a large scale in: A. World War I B. World War II C. Korean War

Trivia Multiple-choice tests were first used on a large scale in: A. World War I B. World War II C. Korean War

Time Management

The One-Shot (By Wyatt 915 (Own work) [Public domain], via Wikimedia Commons)

(By Wyatt 915 (Own work) [Public domain], via Wikimedia Commons)

Time Challenges • Creating the questions (Bruff, 2016)

Time Challenges • Creating the questions • Technical problems (Bruff, 2016)

Time Challenges • Creating the questions • Technical problems • Adapting lesson plan (Bruff, 2016)

Time Challenges • • Creating the questions Technical problems Adapting lesson plan Set-up (Bruff, 2016)

Time Challenges • • • Creating the questions Technical problems Adapting lesson plan Set-up Just-in-time teaching (Bruff, 2016)

Proceed With Caution

Best Practices

Best Practices • Make the commitment (University of Wisconsin, 2016)

Best Practices • Make the commitment • 3 -4 questions at 10 -20 minute intervals per 50 -minute class session (University of Wisconsin, 2016)

Best Practices • Make the commitment • 3 -4 questions at 10 -20 minute intervals per 50 -minute class session • Troubleshooting (University of Wisconsin, 2016)

Best Practices • Make the commitment • 3 -4 questions at 10 -20 minute intervals per 50 -minute class session • Troubleshooting • Low expectations (University of Wisconsin, 2016)

Best Practices • Make the commitment • 3 -4 questions at 10 -20 minute intervals per 50 -minute class session • Troubleshooting • Low expectations • 30 seconds to 1 minute per question (Carl Wieman Science Education Initiative (2009), p. 20)

A stem should relate directly to a learning objective. (Mc. Donald, p. 88)

“Just as there can be no set formulas for producing a good story or a good painting, so there can be no set of rules that will guarantee the production of good test items. Principles can be established and suggestions offered, but it is the item writer’s judgment in the application (and occasional disregard) of these principles and suggestions that determine whether good items or mediocre ones are produced. ” Robert L. Ebel (Ebel, 1951, p. 185)

References • • Bruff, D. (2016). Vanderbilt Center for Teaching: Classroom response systems ("Clickers”). Retrieved from https: //wp 0. its. vanderbilt. edu/cft/guides-sub-pages/clickers/ Burton, S. J. , Sudweeks, R. R. , Merrill, P. F. , and Wood, B. (1991). How to prepare better multiplechoice test items: Guidelines for university faculty. Retrieved from https: //testing. byu. edu/handbooks/betteritems. pdf Carl Wieman Science Education Initiative, University of British Columbia (2009). Clicker resource guide: An instructor's guide to the effective use of personal response systems (clickers) in teaching. Retrieved from http: //www. cwsei. ubc. ca/resources/files/Clicker_guide_CWSEI_CU-SEI. pdf Haladyna, T. M. , Downing, S. M. , & Rodriguez, M. C. (2002). A review of multiple-choice itemwriting guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309 -334. doi: 10. 1207/S 15324818 AME 1503_5 Krathwohl, D. R. . (2002). A revision of Bloom's Taxonomy: An overview. Theory into Practice, 41(4), 212– 218. Retrieved from http: //www. jstor. org/stable/1477405 Mc. Donald, M. E. (2002). Systematic assessment of learning outcomes: Developing multiple-choice exams. Studbury, MA: Jones and Bartlett Publishers. Premkumar, K. & Coupal, C. (2008). Rules of engagement – 12 tips for successful use of “clickers” in the classroom. Medical Teacher, 30(2), 146 -149. doi: 10. 1080/01421590801965111 University of Wisconsin. (2016). Student response systems (SRS): Best practices. Retrieved from http: //www 4. uwm. edu/ltc/srs/faculty/best_practices. cfm

Do you have any questions? A. Yes, a lot B. A few C. No, I think that covers it ann. agee@sjsu. edu

1. Choose the resource that has details about an event as it happens. STEM OPTIONS A) Book B) Scholarly Article DISTRACTORS C) Encyclopedia D) Internet news site KEY