Introduction to Questionnaire Design Spring 2005 Seminar Series

Introduction to Questionnaire Design Spring 2005 Seminar Series Survey Research Laboratory University of Illinois at Chicago www. srl. uic. edu

Questionnaire design encompasses: 1. How to write questions 2. How to draft and organize a questionnaire

The Art of Asking Questions üYou must ask the right question üRespondents must understand your question üRespondents must know the answer üRespondents must be willing and able to tell you the answer

Three Simple Rules for Writing a Good Questionnaire 1) Think through your research questions and objectives before you write questions 2) Prepare an analysis plan before you write questions 3) Ask yourself, in relation to points #1 and #2 above, if each question on your list is necessary? Even if the data would be ‘interesting’ it has to ultimately be used in analysis to make the cut!

Types of Survey Questions 1) Those that ask about behaviors or facts § Non-threatening behavior questions § Threatening behavior questions § Demographics 2) Those that ask about psychological states or attitudes 3) Those that ask about knowledge

What Is A Good Question? § § § § One that yields a truthful, accurate answer One that asks for one answer on one dimension One that accommodates all possible contingencies of response One that uses specific, simple language One that has mutually exclusive response options One that produces variability in response One that minimizes social desirability One that is pretested

Ask questions one at a time Bad question: In the past 6 months, what major appliances has your household purchased new from the store? Better question: Now I’m going to read a list of household appliances. As I read each one, please tell me whether or not your household has purchased this type of appliance new from the store during the past 6 months. How about… a refrigerator? a kitchen range or oven? a microwave?

Ask questions one at a time Bad Question: Compared to one year ago, are you paying more, less, or about the same for your auto and life insurance? Better Question: Compared to one year ago, are you now paying more, less, or about the same for … a. auto insurance? b. life insurance?

Specify who, what, when, where and how. For example, whose income? What’s included? Over what period of time? Example: “In 2002, what was your total household income, before taxes? Please count income from all members of your household, including wages from employment, disability, social security, and public aid. ”

Specify through cues Example People drink beer in many places – for example, at home, at restaurants, at bars, sporting events, at friends’ homes, etc. During the past 30 days, did you drink any beer?

Use words with singular meanings Ambiguous: How would you compare how close you are to family in your current neighborhood to how close you were in your old neighborhood? Would you say your family is closer here, further here, or the same? More Clear: Compared to your last neighborhood, do you now live closer to your family, are you further from your family, or are you about the same distance?

Social Desirability Ø Respondents will try to represent themselves to the interviewer in a way that reflects positively on them ØAs questions become more threatening, respondents are more likely to overstate or understate behavior, even when the best question wording is used

Minimizing Social Desirability • For socially desirable behavior, it is better to ask whether respondents have ever engaged in the behavior before asking whether they currently engage in the behavior • For socially undesirable behavior, it is better to ask about current behavior first, rather than ask about their usual or typical behavior • Train interviewers to maintain a professional attitude • Self-administered computer-assisted procedures can reduce question threat and improve reporting on sensitive questions • Longer questions reduce sensitivity when obtaining information on frequencies of socially undesirable behavior

Open vs. Closed Questions General rule: closed questions are usually better – Easier for the respondent – Less coding later – Better to have respondent do categorizing – Categories help define the question

Disadvantages of Closed Questions • Categories may be leading to respondents • May make it too easy to answer without thinking • May limit spontaneity • Not best when – asking for frequency of sensitive behaviors – there are numerous possible responses

General Principles for Response Options • Response categories should be consistent with the question • Categories must be exhaustive, including every possible answer • Categories must be mutually exclusive (no overlap) • If appropriate, include a “don’t know” category

Constructing Response Scales • Respondents can generally only remember a maximum of 5 responses unless visual cues are used • Using graphic images such as thermometers and ladders and using card sorting for compex ratings is effective • Number of points in scale should be determined by how you intend to use the data • With scales with few points, every scale can be labeled; in longer scales, only the endpoints are labeled

Ordering Response Categories • Usually better to list responses from the lower level to the higher level • Associate greater response levels w/ greater numbers • Start with end of a scale that is least socially desirable

Should you use a middle category? – Common practice is to omit it to push respondents (Rs) toward one end or the other, on theory that few individuals are truly in the middle on a particular issue – Evidence from empirical studies shows that use of an explicit middle alternative will often be taken by Rs in a forced choice situation if offered; at the same time, it does not affect the ratio of ‘pro’ to ‘con’ responses or the size of the don’t know category – Our usual recommendation is to include it unless there are persuasive reasons to exclude

Part 2: How to draft and organize your questionnaire

Ordering the Questions • • • Segment by topic Ask about related topics together Salient questions take precedence over less salient ones Ask recall backwards in time Use transitions when changing topics – give a sense of progress through the questionnaire • Leave objectionable questions (e. g. , income) for the end • Put demographic questions at the end

How to Start the Questionnaire • Start with easy questions that all respondents can answer with little effort • First questions should also be non-threatening • Don’t start with knowledge or awareness questions • First questions should be directly related to the topic as described in the introduction or advance/cover letter

Survey Intro/Cover Letter Introduction should indicate: – who is conducting the survey – the topics to be covered in the survey – an assurance of confidentiality – any IRB stipulations – whether you offer how long it will take depends on mode, topic, population

Physical Format of the Self. Administered Questionnaire • Careful formatting is necessary to decrease errors and increase motivation • Respondent’s needs must always take priority, followed by interviewer and data processors

Physical Format Checklist • Number all questions sequentially • Use large, clear type; don’t crowd • ‘White space: ’ Place more blank space between questions than between subcomponents of questions • List answer categories vertically instead of horizontally • Avoid double/triple ‘banking’ of response choices • Be consistent with direction of response categories • Be consistent with placement of response categories

Physical Format, Continued • Don’t split questions across pages. If necessary (e. g. , question requires 1. 5 pages), restate question and response categories on next page • Put special instructions on questionnaire as needed, next to question • Distinguish directions from questions • Precode the questionnaire (vs. check boxes)

Mail questionnaires • Include a cover letter and contact information if the respondent needs help • Use a booklet format – – – Easier to turn pages Prevents lost pages Permits double-page formats Looks more professional Include a title, graphic, name/address of sponsor on cover

Testing the Questionnaire • Preferable to test the questionnaires with people like those in your main study population • Test in same mode to be used for main study • Consider cognitive pretesting

Questionnaire Design Steps 1. 2. 3. 4. 5. 6. 7. Decide what information is needed Search for existing questions Focus groups Draft new questions/revise existing ones Sequence the questions Get peer evaluation Revise and test on self/co-workers

Questionnaire Design Steps 8. 9. 10. 11. 12. Think-aloud interviews Revise/eliminate questions Prepare interviewer instructions for pilot test Pilot test (10 -20 cases) Revise eliminate questions based on respondent & interviewer comments

Questionnaire Design Steps 13. 14. 15. 16. 17. Pilot test again, if necessary Prepare final interviewer instructions Be prepared to modify questionnaires if interviewer training raises problems After interviewing is complete, debrief interviewers for potential problems Use experience from one study for future planning

Key References To Get Started Dillman, Don. Mail and Internet Surveys: The Tailored Design Method. New York: John, Wiley & Sons, Inc, 2000. Bradburn, N, Sudman, S. and Wansink, B. . Asking Questions: The Definitive Guide to Questionnaire design – for Market Research, Political Polls, and Social and Health Questionnaires. San Francisco: Jossey Bass, 2004. See also past issues of Public Opinion Quarterly