Survey Research Neuman and Robson Ch 8 Introduction

Survey Research Neuman and Robson Ch. 8

Introduction to Survey Research l l Survey research is a very old, and a very popular, research technique 1880 survey by Karl Marx in France l l 25000 questionnaires sent to workers Late 19 th century “Chocolate Sociologists” l l Rowntree and Cadbury Community surveys to study poverty

Topics appropriate to survey research: l descriptive, explanatory, and exploratory research purposes l best method available to collect original information about a population l To measure attitudes and orientations

General features of survey research l respondents are chosen through probability sampling procedures l Systematic questionnaire or interview procedures l Sophisticated statistical techniques

Strengths and Weaknesses of Survey Research: l Advantages l l l Reliability Can describe large population Flexibility Standardized Disadvantages l l Validity Superficiality Cannot modify questionnaire in field Artificiality

Public Opinion Polling Canada l l l No official or mandatory regulation i. e. election polls Could influence voters “bandwagon” phenomenon “free will” effect Problems and discrepancies: l l l 20 -40% answer “don’t know” or “no answer” Results often published as if out of 100% misrepresentation

Mail Surveys and Self-Administered Questionnaires l l l Hand-administered to a group or mailed to sample Mail distribution and return practices Monitoring returns Follow-up mailings Response rates: 50% adequate, 70% very good Additional factors: l l Sponsorship Inducement to respond

Disadvantages of mail surveys: l l l Requires simple questions Cannot observe respondents No opportunity for probe No control over conditions or who responds Not suited for people who are illiterate or for whom English not first language Low response rate can be a problem

Face-to-Face Interview Surveys l l l l Very expensive Interviewers require training Use interview schedule Appearance and demeanour crucial Need familiarity with questionnaire Must follow question wording exactly Must record responses exactly Also record other events during interview

Advantages and Disadvantages of Face-to-Face Interview Surveys l Advantages: l Can probe for responses l Can keep respondent “on task” l Disadvantages: l Cost and training l Interviewer bias

Telephone Surveys l l In past, could reach 95% of households Now, between 20 and 26% of Canadians have no landline (see next slide) Use of telephone interview schedules Selecting a sample for telephone surveys l l l Telephone directory Random Digit Dialing (RDD) Selecting a respondent from a household

Percentage of households in Canada that use a cell phone only (Source: Stats Can “The Daily”, 2014) All households Households under 35 years of age Households aged 55 and older 2008 2010 2013 8 12. 8 21 26. 1 39. 3 60. 6 1. 9 3. 8 6. 4

Telephone Surveys l Can achieve higher response rates l Can decrease "don't knows" and "no answers” l Interviewers can help clarify l Telephone surveys have many advantages of faceto-face but without high cost

Advantages and Disadvantages of telephone surveys l Advantages: l l cheaper to carry out than face-to-face interviews less time and effort more impersonal Disadvantages: l l large number of tries to achieve a small number of successes. less motivation generated among respondents No visual cues/aids possible Limited interview length

Computer-assisted telephone interviewing (CATI) l l interviewers sit at a computer terminal with a screen answers given are directly entered into the computer Very fast Often used in polling (i. e. Video: Ask a Silly Question)

Internet (Web) Surveys l l l In Canada, 95 per cent of Canadians in the highest income quartile are connected to the Internet, yet only 62 per cent in the lowest income quartile have Internet access (2014) Use same basic design as mail surveys Advantages: l l l Fast, cheap Can use graphic and visual effects Disadvantages: l l Coverage – not all groups use computers Privacy – need for encryption

Comparison of Survey Methods Criteria Interview Mail Telephone _________________________________ Web 1. Cost high low moderate very low 2. Response rate high low high moderate high low moderate high 3. Level of control of situation 4. Applicability to geographically dispersed populations moderate high 5. Applicability to heterogeneous populations high low high moderate 6. Obtaining detailed information high moderate low 7. Speed slow fastest

Secondary Analysis of Survey Data l l l New analysis of data collected for another purpose Use of shared data and data archives (i. e. Census and GSS) Advantages l l l cheaper and faster Access to larger datasets Disadvantages l l l problems of validity Questions may not meet your needs Sample may not be adequate

Questionnaire Construction l usually includes a variety of demographic questions, and one or more scales and indexes aimed at collecting data on attitudes, beliefs, or behaviours l purpose of a survey is to try to collect information about a sample in order to make generalizations about the larger population

The general structure of a good questionnaire: l l l l 1. Title 2. A brief introduction and explanation of research. 3. A section of general demographic questions re: gender, age, marital status, education. Note: more sensitive demographic questions (i. e. income, home value, etc. ) might be better placed at the end of the questionnaire. 4. Section of non-threatening or less sensitive behavioural and attitudinal questions. 5. A section near the end with any sensitive or threatening questions. 6. Any additional demographic information. 7. Conclusion with brief statement again thanking respondents for their time and effort

Additional Considerations: l l l 1. Keep questionnaire as short as is reasonably possible. 2. Appearance of the questionnaire is important. Don't crowd your questions, and use an easy-toread font. Leave some space between questions. 3. Use bold font and underlining for titles/headings. 4. Use examples and sample questions for clarity, but be careful not to introduce bias. 5. Include an open-ended question at the end of the questionnaire asking for any additional information and/or feedback. This question sometimes provides some interesting information!

Constructing the questions: l l Create questions that use indicators which will specifically address your hypotheses. Use existing scales or measures whenever possible. You can also modify a scale or index to meet the needs of your particular study if an existing scale is not quite appropriate. The advantage in using an existing scale is that its reliability and validity are already known.

Types of questions to use: l a. Simple questions that use a concrete indicator (i. e. What is your age in years? ___). l b. Likert style questions are used for measuring simple attitudes, beliefs, emotions, or behaviours (i. e. When you think back to your high school years, you feel: 1. Very unhappy 2. Somewhat unhappy 3. Don't feel anything 4. Happy 5. Very happy) l c. Matrix questions are useful when you have a number of related items. Matrix questions are composed of a series of questions that have identical response categories which are presented in table format with the response category labels at the top of the chart.

Types (cont. ) l c. Guttman scales: These are useful for measuring the strength or intensity of an attitude. l d. Rank-ordering: Rank ordering is a method that works well with a small number of objects, such as 10. For example, instead of rating how interesting each of a set of subject areas are, you could ask the respondent to simply sort them in order of most to least interesting. l e. Paired comparisons: In this method, you present items two at a time, and ask respondent to pick which one has more of some attribute. For example, you can present behavioural problems and ask which one is more serious.

Closed or open-ended question responses l A closed response set refers to those questions which have fixed categories for answers (uses quantitative analysis) l Open-ended questions allows for free responses (qualitative analysis)

Filter questions and contingency questions l Filter questions are useful for sorting respondents. l Contingency questions allow respondents to skip portions of the survey that may not be relevant to them.

Principles of Question-Writing l l l l 1. Keep questions as short and concise as possible. 2. Choose your wording carefully. 3. Try not to ask questions beyond a respondent's capabilities. 4. Avoid emotional language and the use of "loaded" words. 5. Watch out for prestige bias in your questions. 6. Don't use leading questions like "You don't smoke, do you? " 7. Avoid ambiguity and vagueness.

Principles (cont. ) l l l 8. Don't use double-barrelled questions. 9. Avoid false premises or assumptions. 10. If possible, don't ask respondent about future intentions. The link between intentions and future behaviour is tenuous. 11. Try not to use negatives and double negatives in your questions. 12. Watch for explanatory statements that may bias the answer to the question. Watch out also for questions that might influence the answers to subsequent questions.

Other Issues l Aiding recall l Provide special instructions or extra time Ask month by month, week by week, etc. Threatening questions l l People under-report illness, disability, deviance, illegal activity, income/wealth Create “enhanced questions” or embed activity in list of more serious activities

Issues (cont. ) l Social desirability bias l l l People tend to over-report socially desirable behaviours (being cultured, voting, giving to charity, being good spouses or parents, etc. ) Try to minimize importance of these activities or present alternatives in questions Knowledge questions l l Make sure questions are appropriate level Phrase questions so respondents feel comfortable not knowing the answer