Dr Angelika Pohl Workshop 39 Writing Questions That

Dr. Angelika Pohl – Workshop #39 “Writing Questions That Elicit What You’re Looking For – And Questions That Can Reveal Unexpected, Important Information!” Dr. Angelika Pohl www. Better. Testing. com 1

A Welcome Quiz n n n The “Welcome Quiz” presents you with 10 multiple-choice questions that have one or more common flaws. Discuss the questions with colleagues at your table and identify the flaws. Work quickly and win some prizes! Dr. Angelika Pohl www. Better. Testing. com 2

Multiple-Choice Questions n n Not as easy to write as they look Good “Distractors” are key n n Relevant Plausible Mutually exclusive Etc. Dr. Angelika Pohl www. Better. Testing. com 3

Writing Multiple Choice Items A few – of the many – rules to know! n Avoid n n n n “all of the above, ” “none of the above” “b and c” [unless “Mark all that apply!”] negatively worded questions silly distracters [wrong answer choices have to be plausible, relevant to the topic] clues in the question to the answer ambiguous answer choices overlapping answer choices more than one correct answer! Dr. Angelika Pohl www. Better. Testing. com 4

My survey for workshop participants n n Thank you for responding! Thank you for sending me surveys. Response rate was 79% (77 out of 98 surveys sent). Why was it this high? Dr. Angelika Pohl www. Better. Testing. com 5

What You Told Me: n Participants are highly diverse in n n Work contexts Types of survey needs Survey experience What you want from this workshop: n Everything!Basics! Advanced! Statistics! Dr. Angelika Pohl www. Better. Testing. com 6

Workshop Plan n n Multiple-Choice Questions n Take Quiz & Discussion Mixed-Method Survey n n Sample Survey & Discussion Survey Development n Considerations, Hints, and Rules Dr. Angelika Pohl www. Better. Testing. com 7

Another Sample Survey n n Please answer the questions for yourself – no one else will see your answers. (Uncomfortable? Just skip. ) We will then discuss how effective the questions are – and how they could be improved. Dr. Angelika Pohl www. Better. Testing. com 8

General Survey Considerations n n Surveys are voluntary Establish a relationship n n n Trust Comfort Credibility Mutual Respect Visual appeal Dr. Angelika Pohl www. Better. Testing. com 9

What Type of Questions to Use? n Binary: Yes/No n n Multiple-Choice n n n If No, go to question #18 Can be 3, 4, or more options, depending Likert Scaled Open-Ended WHICH TO USE? Cost in time and dollars n for respondent and for researcher Dr. Angelika Pohl www. Better. Testing. com 10

Likert Scales n n n “Strongly agree” to “strongly disagree” “Excellent” to “poor” Other possibilities (a lot – a little. . . ) Use 4 or 5 options (no more) include “neutral” choice? When relevant, include “not applicable” and/or “Don’t know” Dr. Angelika Pohl www. Better. Testing. com 11

How often is “often”? Busy physicians were asked how often they: 1. Advise smokers to stop smoking 2. Refer smokers to smoke-cessation program • • • Never Rarely Often Most of the time Always Dr. Angelika Pohl www. Better. Testing. com 12

Some Definitions During the past year, how often did you interact with XYZ Project staff? 1. . . Frequently (4 or more times per month) 2. . . Sometimes (1 -3 times per month) 3. . . Rarely (Less than 1 time per month) 4. . . Never (No interaction during the past year) Dr. Angelika Pohl www. Better. Testing. com 13

Problems in Formatting & Scaling Strongly Disagree Neutral Agree Strongly Agree 1. The staff was helpful to me. 5 4 3 2 1 2. I didn’t have to wait too long. 5 4 3 2 1 Dr. Angelika Pohl www. Better. Testing. com 14

A Better Format and Scale Strongly disagree neutral 1. The staff was helpful to me. 2. I didn’t have to wait too long. agree Strongly agree N/A 1 2 3 4 5 N/A Dr. Angelika Pohl www. Better. Testing. com 15

What’s wrong here? [An Excerpt] PLEASE INDICATE YOUR LEVEL OF AGREEMENT: Disagree Agree Strongly Strongly The instructor showed expertise in this field. 1 2 3 4 5 The instructor welcomed questions and comments. 1 2 3 4 5 Excellent Poor PLEASE PROVIDE A RATING: 1 2 3 4 5 The content of the workshop 1 2 3 4 5 The workshop leader 1 2 3 4 5 Dr. Angelika Pohl www. Better. Testing. com 16

Ask only 1 thing at a time! Rate the degree of your agreement or disagreement with these statements: 1. Instructional materials were available and pertinent. 2. The instructor reviewed treatment options and identified the advantages and disadvantages associated with each. Dr. Angelika Pohl www. Better. Testing. com 17

A difficult task for the respondent “Imagining that you were to be named principal of a school tomorrow, how would you rate your abilities right now in the following areas? ” Scale: 1 – Weak (critical priority for self-improvement) 2 – Not quite adequate (high priority for selfimprovement) 3 – Adequate (area for self-improvement) 4 – More than adequate 5 – Very good 6 – Expert (can mentor others) Dr. Angelika Pohl www. Better. Testing. com 18

Something to think about: Survey length! (How long will it take? Be honest with Respondents!) 37 Statements like these, with those 6 answer options Develop a school organization where faculty & staff understand that every student counts and where every student has the support of a caring adult. Understand the change process and use leadership & facilitation skills to manage it effectively. Understand how adults learn and advance meaningful change through quality, sustained professional development that leads to increased student achieveme Dr. Angelika Pohl www. Better. Testing. com 19

Open-ended questions n Instead of n n “How did you feel about the program? ” Give structure: n n n “List 2 -3 things you learned that you will use” “Please describe anything you particularly liked and/or disliked” “What question(s) should have been included in this survey, and what would be your answer? ” Dr. Angelika Pohl www. Better. Testing. com 20

Strengths of “Mixed Methods” n Teachers were asked to n n Rate their “morale” on scale of 1 -5 (low to high). Elaborate on and explain their rating. Teachers had very different ideas of what the numbers meant! “ 4” elicited: n n “Everything’s going really well; the kids are now learning and my principal is supportive” “Still struggling with discipline; worried about upcoming tests; administrators not there for me” Dr. Angelika Pohl www. Better. Testing. com 21

Validity n It’s not the survey itself that is “valid. ” It’s the validity of the inferences we make – and the uses to which we put the results – that matters. n Different types of validity: n n n Criterion-related validity Construct-related Etc. Is it useful for its intended purpose? Dr. Angelika Pohl www. Better. Testing. com 22

Reliability n n Another word for it: “Consistency” Answers aren’t affected by irregular circumstances of taking the survey n (pressure, bias, illegibility, poor instructions, stock market crash, headache, etc. ) Dr. Angelika Pohl www. Better. Testing. com 23

Key Questions To Ask Yourself: Part 1 n n n 1. What is the purpose of this survey? 2. Do the respondents understand the purpose of the survey? 3. Does every question relate directly to this purpose? (no fishing expeditions) 4. Is the survey as short as it can be? (Is every question necessary? ) 5. Are questions phrased so they hold the respondent’s attention? (Is it interesting? ) Dr. Angelika Pohl www. Better. Testing. com 24

Key Questions To Ask Yourself: Part 2 n n n 6. Have you avoided buzzwords, jargon, cultural bias and insensitivity? 7. Have respondents been assured of confidentiality or anonymity? (Know the difference) 8. Have you taken appropriate steps to ensure confidentiality & anonymity? 9. Do you have clear plans for how to summarize, analyze, interpret the data? (Who, when, how, why, what) 10. How will you effectively report the results to different stakeholders? Dr. Angelika Pohl www. Better. Testing. com 25

Key Questions To Ask Yourself: Part 3 n n n 11. Will the results likely help you answer important questions? 12. Will the results give a fair (persuasive? ) picture of the progress that program participants have made? 13. Will the results likely provide a basis for improving the program? 14. Will the information collected have been worth the effort (by evaluators and by respondents)? 15. Is there a feasible way for you to share (some) results of the survey with respondents? Dr. Angelika Pohl www. Better. Testing. com 26

The Cardinal Rule! Pilot-test your survey with willing “innocents” n n People who resemble your intended respondents, who will give you honest feedback, who aren’t clued in to the buzzwords & the jargon, who aren’t well informed about your research. Note: It takes more than one pair of eyes to find ambiguities and other problems. Dr. Angelika Pohl www. Better. Testing. com 27

Remember. . . "Not everything that counts can be counted; and not everything that can be counted counts. " (Einstein) Dr. Angelika Pohl www. Better. Testing. com

Practice What You Learned n QUICKLY: 5 minutes n n n Discuss at your table what tasks you’ve each prepared Decide which tasks to do Decide who will work with whom Get to Work! 15 minutes I will circulate to be at your service. Dr. Angelika Pohl www. Better. Testing. com 29

Thank You for Your Attention! n Feel free to contact me with further questions and comments – as well as with your consulting needs! Dr. Angelika Pohl Founder and Senior Consultant Better Testing & Evaluations Info@Better. Testing. com Located right here in hot but beautiful Atlanta Dr. Angelika Pohl www. Better. Testing. com 30