RMK Survey Research l l l l Types

RMK: Survey Research l l l l Types of Surveys (Q & I) Types of Questions Decisions about Question Content Decisions about Response Format Decisions about Question Wording Decisions about Question Placement Conducting Interviews Advantages / Disadvantages of Methods

Types of Surveys Questionnaires and Interviews

Questionnaires Group administration Mail administration Household drop off Web-based

Interviews Personal Telephone

Types of Questions

Type of Questions l Unstructured » Open ended / short comment box / transcript of interview / narrative l Structured » Details in following slides

Structured Questions Dichotomous ____Yes ____No Male Female

Structured Questions Nominal Occupational Class: 1 = Truck driver 2 = Lawyer 3 = etc.

Structured Questions Ordinal Rank the candidates in order of preference from best to worst. . . ___ Bob Dole ___ Bill Clinton ___ Newt Gingrich ___ Al Gore

Structured Questions Interval Likert Response Scale 1 strongly disagree 2 disagree 3 neutral 4 agree 5 strongly agree

Structured Questions Interval Semantic Differential very much somewhat neither interesting somewhat very much boring simple complex uncaring useful useless etc.

Structured Questions Interval Guttman (Cumulative) Scale ___Are you willing to permit immigrants to live in your country? ___Are you willing to permit immigrants to live in your community? ___Are you willing to permit immigrants to live in your neighborhood? ___Would you be willing to have an immigrant live next door to you? ___Would you let your child marry an immigrant?

Filter or Contingency Questions Have you ever smoked marijuana? yes no

Filter or Contingency Questions Have you ever smoked marijuana? yes no If yes, about how many times have you smoked marijuana? once 2 to 5 times 6 to 10 times 11 to 20 times more than 20 times

Filter or Contingency Questions l l Try to avoid having more than three levels (two jumps) for any question. If only two levels, use graphic to jump (for example, arrow and box). If all responses can’t fit on the page, then jump to a new page. Electronic advantages (web pages) – Can be programs to move forward in a specific way based on response.

Decisions About Question Content

Decisions About Question Content Is the question necessary / useful? l Example – Do you need the age of each child or just the number of children under 16? l Example – Do you need to ask income or can you estimate?

Decisions About Question Content Are several questions needed? Don’t combine two issues in one. . . l Examples: Bad Question – What are your feelings towards African. Americans and Hispanic-Americans? – What do you think of proposed changes in benefits and hours?

Decisions About Question Content Are several questions needed? Did you get all the info desired. . . l l If you ask about earnings, might not mention all income If ask if they’re in favor of public TV, ask for whom (might not mention it’s because of Sesame Street for the kids)

Decisions About Question Content Are several questions needed? Do you need more context to interpret answer. . . l If you ask research participants about attitudes towards a specific religion, can you interpret the results without finding out about their attitudes towards religion?

Decisions About Question Content Are several questions needed? Need more info to determine the intensity. . . l If they say they support public TV … – do they watch it? – would they be willing to have their tax dollars spent on it?

Decisions About Question Content Do respondents have the needed info? l Examples – ‘Do you think Dean Rusk acted correctly in the Bay of Pigs crisis? ’ -- can’t say if they don’t know who he is. – In TV surveys, ‘regular’ watchers can be asked about program content, ‘never watch’ should not be asked.

Decisions About Question Content Does question need to be more specific? l l l How well did you like the book? Versus. . . Did you recommend the book to others? Did you look for other books by that author?

Decisions About Question Content Is question sufficiently general? l For instance, seasonal specificity. – If you ask what they usually watch and it’s summer, you’ll get a different answer than in the winter.

Decisions About Question Content Is question biased or loaded? l l Ask about the benefits of a tax cut without asking about the disadvantages Ask about the disadvantages of eliminating welfare without asking about the benefits

Decisions About Question Content Will respondent answer truthfully? l To find out age or income, give brackets. – [18 -22], [23 -27], etc. l l To find out about charitable contributions, ask how much “people you know” typically give. To find out what magazines they read, offer to buy used ones.

Decisions About the Response Format

Should the Response Format be. . . § § Single option vs. multi-option Structured § Dichotomous (two possibilities) § Multiple choice § Fill-in-the-blank § Scale - interval (e. g. , Likert response format, semantic differential, Guttmann scale) § Checklist – see next slide… § Unstructured § Open ended / free answer / narrative

If Checklist. . . § § § are all alternatives covered? is it of reasonable length? is the wording impartial? is the form of the response easy, uniform? If its possible that there may be some other alternatives, provide “other” option and allow open ended response.

Decisions About Question Wording

Decisions About Question Wording l l l Can the question be misunderstood? Questions about ‘nationality’ or ‘marital status are too vague. Questions about ‘mass media’ are too general. What kind of headache remedy do you use? – Want brand name? – Want type? (Aspirin, pill, or capsule? )

Decisions About Question Wording What assumptions does question make? l l If you ask what social class someone’s in, you assume that they know what social class is and that they think of themselves as being in one. Check assumption in a previous question.

Decisions About Question Wording Is the time frame specified? l l l Do you think Congress will cut taxes? Do you think Congress will successfully resist tax cuts? Neither of these specifies a time frame.

Decisions About Question Wording How personal is the wording? Ensure Appropriate Context for the study l Are working conditions satisfactory or not satisfactory in the plant where you work? » Objective l Do you feel that working conditions are satisfactory or not satisfactory in the plant where you work? » Feeling l Are you personally satisfied with working conditions in the plant where you work? » Personal satisfaction

Decisions About Question Wording Is the wording too direct? l l l How did you feel about being in the war? - May be too direct How well did the equipment hold up in the field? How well were new recruits trained?

Decisions About Question Wording Some Additional Issues. . . l l Does question contain difficult or unclear terminology? Does the question make each alternative explicit? Is the wording objectionable? Is the wording loaded or slanted?

Question Placement and Sequence

Decisions About Placement l l l Is the answer influenced by prior questions? Does question come too early or too late to arouse interest? Does the question receive sufficient attention?

The Opening Questions l These are particularly true in the case of an interview. – Should be easy to answer – Should not be sensitive material – Should get the respondent “rolling” l In a survey, consider – putting basic demographic questions towards the end of a survey because these are easy to complete and thus it may overcome “survey fatigue. ”

Sensitive Questions l l l Only after trust is developed Should make sense in that section of the survey (not “out of left field”) Precede with warm-up questions

A Checklist of Considerations (slide 1) 4 4 Start with easy, nonthreatening questions. Put more difficult, threatening questions near end. Never start mail survey with an openended question. Put demographics at end (unless needed to screen).

A Checklist of Considerations (slide 2) 4 4 4 Avoid demographics at beginning. For historical demographics, follow chronological order. Ask about one topic at a time. When switching topics, use a transition. Reduce response set. For filter or contingency questions, make a flowchart.

Important Rule Keep questionnaire as short as possible!

Interviews

The Role of the Interviewer l l l Locate and enlist cooperation of respondents. Motivate respondents to do good job. Clarify any confusion/concerns. Observe quality of responses. Conduct a good interview.

Training the Interviewers (slide 1) Ô Ô Ô Describe the entire study. State who is sponsor of research. Teach enough about survey research (should respect method and be motivated). Explain the sampling logic and process. Explain interviewer bias. “Walk through” interview.

Training the Interviewers (slide 2) Ô Explain selection procedures, including – Reading maps – Identifying households – Identify respondents Ô Ô Ô Rehearse interview Explain supervision Explain scheduling

The Interviewer’s Kit l l l Official-looking 3 -ring notebook Maps Sufficient copies of survey Official identification A cover letter A phone number to call

The Interview â â â Opening remarks Asking the questions Obtaining adequate answers Recording the response Concluding the interview More Details on Following Slides

Opening Remarks l l Gaining entry -- appearance, confidence, non-threatening Doorstep technique -- brief, suggest don’t ask Introduction -- name, organization, credentials Explaining the study -- general, stress confidentiality

Asking the Questions l l l Use questionnaire carefully, but informally. Ask questions exactly as written Follow the order given. Ask every question. Don’t finish sentences. Don’t leave any questions blank.

Obtaining Adequate Responses - The Probe l l l Silent probe - a pause Overt encouragement Elaboration – How do you mean? Anything else? l l Ask for clarification Repetition

Recording the Response l l l Record responses immediately Include all probes (write them in) Use abbreviations where possible (for example – R = respondent; DK = don’t know)

Concluding the Interview l l l Thank respondent. Don’t be brusque or hasty. Immediately after leaving -- write down any notes about how the interview went.

Advantages and Disadvantages of Survey Methods (general assessment)

Visual Presentations Possible? Questionnaire Group Mail Interview Drop-Off Personal Phone

Visual Presentations Possible? Questionnaire Group Mail Yes Interview Drop-Off Personal Phone Yes No

Long Response Categories Possible? Questionnaire Group Mail Interview Drop-Off Personal Phone

Long Response Categories Possible? Questionnaire Group Mail Yes Interview Drop-Off Personal Phone Yes ? No

Privacy? Questionnaire Group Mail Interview Drop-Off Personal Phone

Privacy? Questionnaire Group Mail No Yes Interview Drop-Off Personal Phone No No ?

Flexible? Questionnaire Group Mail Interview Drop-Off Personal Phone

Flexible? Questionnaire Group Mail No No Interview Drop-Off Personal Phone No Yes

Open-ended Questions Feasible? Questionnaire Group Mail Interview Drop-Off Personal Phone

Open-ended Questions Feasible? Questionnaire Group Mail No No Interview Drop-Off Personal Phone No Yes

Reading and Writing Needed? Questionnaire Group Mail Interview Drop-Off Personal Phone

Reading & Writing Needed? Questionnaire Group Mail ? Yes Interview Drop-Off Personal Phone Yes No No

Can You Judge Quality of Response? Questionnaire Group Mail Interview Drop-Off Personal Phone

Can You Judge Quality of Response? Questionnaire Group Mail Yes No Interview Drop-Off Personal Phone ? Yes ?

High Response Rates? Questionnaire Group Mail Interview Drop-Off Personal Phone

High Response Rates? Questionnaire Group Mail Yes No Interview Drop-Off Personal Phone Yes No

Explain Study in Person? Questionnaire Group Mail Interview Drop-Off Personal Phone

Explain Study in Person? Questionnaire Group Mail Yes No Interview Drop-Off Personal Phone Yes ?

Low Cost? Questionnaire Group Mail Interview Drop-Off Personal Phone

Low Cost? Questionnaire Group Mail Yes Interview Drop-Off Personal Phone No No No

Staff and Facilities Low? Questionnaire Group Mail Interview Drop-Off Personal Phone

Staff and Facilities Low? Questionnaire Group Mail Yes Interview Drop-Off Personal Phone No No No

Access to Dispersed Samples? Questionnaire Group Mail Interview Drop-Off Personal Phone

Access to Dispersed Samples? Questionnaire Group Mail No Yes Interview Drop-Off Personal Phone No No No

Respondent Has Time to Formulate Answers? Questionnaire Group Mail Interview Drop-Off Personal Phone

Respondent Has Time to Formulate Answers? Questionnaire Group Mail No Yes Interview Drop-Off Personal Phone Yes No No

Personal Contact? Questionnaire Group Mail Interview Drop-Off Personal Phone

Personal Contact? Questionnaire Group Mail Yes No Interview Drop-Off Personal Phone Yes No

Long Survey Feasible? Questionnaire Group Mail Interview Drop-Off Personal Phone

Long Survey Feasible? Questionnaire Group Mail No No Interview Drop-Off Personal Phone No Yes No

Quick Turnaround? Questionnaire Group Mail Interview Drop-Off Personal Phone

Quick Turnaround? Questionnaire Group Mail No Yes Interview Drop-Off Personal Phone No No Yes