NonExperimental Designs Survey Methods Psych 231 Research Methods

Non-Experimental Designs: Survey Methods Psych 231: Research Methods in Psychology

n n I will aim to have the exams graded and entered into Reggie. Net by Monday Remember that piloting is happening in labs this week, so be prepared with everything that you need, and attendance is important Announcements

n Sometimes you just can’t (or don’t want to) perform a fully controlled experiment n n n Because of the issue of interest Limited resources (not enough subjects, observations are too costly, etc. ). There are other kinds of designs that may be used • Surveys • Correlational studies • Quasi-Experiments • Developmental designs • Small-N designs n This does NOT imply that they are bad designs n Just remember the advantages and disadvantages of each Non-Experimental designs Crash course: Research methods (~11 mins)

n Sometimes you just can’t (or don’t want to) perform a fully controlled experiment n n n Because of the issue of interest Limited resources (not enough subjects, observations are too costly, etc. ). There are other kinds of designs that may be used • Surveys • Correlational studies • Quasi-Experiments • Developmental designs • Small-N designs n This does NOT imply that they are bad designs n Just remember the advantages and disadvantages of each Non-Experimental designs

§ Popular methodology: seem to be everywhere these days. Collecting the data Surveys Reporting the results

§ Popular methodology: substantial changes over the years Characteristic 1978 2000 2009 2014 Through the 1960 s 1970 s through 1980 s 1990 s to the present Human interaction High: face-to-face through inperson visits to respondent homes Medium: Remote through a telephone connection Low: Encounter is more likely to be with a machine or its products Trust that the survey is legitimate High: Encouraged by interviewer presence, appearance, and sincerity Medium: Encouraged through voice inflection and ability to listen and request additional info Low: Because of possibility that survey is fake and potentially harmful to respondent Time investment with each respondent High: Interviewer goes to respondent and obtains information one-on-one Medium: One-on-one, but contact effort is minimal Low: Minimal to no time with individual respondents Attention given to each respondent High: Because of time to find the interview each respondent Medium: Because of placing calls one after another Low: Mass e-mails Respondent control over access Low: Households generally accessible Medium: Unlisted numbers, voice mail, and call monitoring High: Caller ID, call blocking, e-mail filters Respondent control over whether to respond Low: Required breaking off human interaction Medium: Ease of hanging up telephone High: increased disclosures required to be communicated and social support for refusing Table 1. 1 from Dillman, Smyth, & Christian (2009 ). Internet, mail, and mixedmode surveys. The tailored design method. 3 rd edition Surveys

n n n Instruments used What are survey methods? Questionnaires and interviews that ask people to provide information about themselves Note: these instruments can be used as DVs in many research designs (observational, experimental) Why conduct them? n Best way to collect some kinds of information: • Descriptive, behavioral, and preferential • e. g. , demographic information, recreational behavior, personal preferences, and attitudes n To compliment experimental work • Good/common first step, can collect a lot of data about a lot of variables • Do not have to directly observe behaviors n Surveys A good resource: http: //www. people-press. org/ n Check out the Methodology Pages

n Advantages n One can investigate internal events • e. g. , attitudes & opinions n n Can generalize about an entire population based on relatively small samples of individuals Large amounts of data can be collected quickly with relatively little cost (effort, time, etc. ) • But they’re often not as “cheap” as you may think Surveys

n Disadvantages n Correlational: causal claims shouldn’t be made • Often used for descriptive and predictive research questions • Note: Survey instruments are sometimes used as part of an experimental design, which may allow testing of causal claims n Non-response bias • Why doesn’t everybody respond? • Does response rate interact with variables of interest? n Self-reports may not be truthful • Response set - tendency to respond from a particular perspective • Social desirability bias (e. g. , how a “moral” person would answer) n Large data sets are sometimes difficult to analyze Surveys

n Stage 1) Identify the focus of the study and select your research method n n n What are the objectives of the research? Is a survey method the best approach? What kind of survey should be used? • • • Group administration Mail surveys Internet surveys Telephone surveys Face-to-face interviews Focus group interviews 11 Stages of survey research

n n Stage 2) Determining the research schedule and budget Stage 3) Establishing an information base n Find out what’s been done, what’s known • E. g. , Find other related surveys n Stage 4) Identify the sampling frame n The actual population that the sample is drawn from (as opposed to the ideal population) • Think of it as operationalizing the conceptual level population • Be aware of potential coverage error – when the sampling doesn’t lead to a good representativeness 11 Stages of survey research Kahn Academy: Examples of bias in surveys

n Stage 5) Determining the sample method and sampling size n Review Probability and Non-Probability methods • Voluntary response method n Importance of sample size 11 Stages of survey research Kahn Academy: Reasonable Samples

A kind of convenience sampling methods commonly used Should leftover Halloween candy be given out to students who get an A on the exam? n Call 231 -4 YES if you think YES Call 231 -NONO if you think NO • Problem: Typically only individuals with strong opinions respond, so the results are often extremely biased (A coverage error issue? Nonresponse issue? ) Daily show clip (8/17/2009, 5: 30 mins) Voluntary response methods

n Sampling error - how is the sample different from the population? Often focus on this part But this part is important too n Confidence intervals • An estimate of the mean or percentage of the population, based on the sample data • “John Doe has 55% of the vote, with a margin of error ± 3%” • Margin of error (that “± 3%” part) • The larger your sample size, the smaller your margin of error will be. • Which would you be more likely to believe • “We asked 10 people …” • “We asked 1000 people …” Importance of sample size More than just sampling error (10/5/16)

n Sampling error - how is the sample different from the population? n Response rate • What proportion of the sample actually responded to the survey? • Hidden costs here - what can you do to increase response rates • Non-response error (bias) • Is there something special about the data that you’re missing (From the people who didn’t respond)? Importance of sample size

n Stage 6) Designing the survey instrument n Question construction: How the questions are written is very important • Clearly identify the research objectives • Do your questions really target those research objectives (think Internal and External Validity)? • Take care wording of the questions • Keep it simple, don’t ask two things at once, avoid loaded or biased questions, etc. • How should questions be answered (question type)? 11 Stages of survey research • 7 tips for good survey questions

Poor Was the FDC negligent by ignoring the warnings about Vioxx during testing and approving it for sale? Yes Problem: a)emotionally b) No charged c) words Unsure Do you favor eliminating the wasteful excess in the public school budget? a) b) c) Yes No Unsure Good If the FDC knew that Vioxx caused serious side effects during testing, what should it have done? a)Ban it from ever being sold b)Require more testing before approving it c)Unsure Do you favor reducing the public school budget? a)Yes b)No c)Unsure Good and poor questions

Poor Should senior citizens be given more money for recreation centers and food assistance programs? a) Yes b) No Problem: asks c) Unsure two different questions Good Should senior citizens be given more money for recreation centers? a) b) c) Yes No Unsure Should senior citizens be given more money for food assistance programs? a) b) c) Yes No Unsure Good and poor questions

Poor Are you against same sex marriage and in favor of a constitutional amendment to ban it? Good What is your view on same sex marriage? a) a) b) c) Yes No Unsure Problem: Biased in more than one direction b) c) I think marriage is a matter of personal choice I’m against it but don’t want a constitutional amendment I want a constitutional amendment banning it Problem: Asks two questions Good and poor questions

n Question types n Open-ended (fill in the blank, short answer) • Can get a lot of information, but • Coding is time intensive and potentially ambiguous n Close-ended (pick best answer, pick all that apply) • Easier to code • Same response alternatives for everyone • Take care with your labels • Decide what kind of scale • Decide number/label of response alternatives Survey Questions What is the best thing about ISU? (choose one) n 1. Location n 2. Academics n 3. Dorm food n 4. People who sell things between Milner and the Bone

n Decide what kind of rating scales • Rating: e. g. , Likert scale PSY 231 is an important course in the major. 1 Strongly Agree 2 Agree 3 Neutral 4 Disagree 5 Strongly Disagree • Semantic differential: Rate how you feel about PSY 231 on these dimensions Important _____: _____: Unimportant Boring _____: Interesting _____: • Nonverbal scale for children: Point to the face that shows how you feel about the toy. Survey Questions: Close-ended

n Decide number/label of response alternatives • Use odd number (mid point and equal # of responses above and below the mid point) • Questions should be uni-dimensional (each concerned with only one thing) • Labels should be clear ü Strongly Disagree No Opinion Strongly Agree 1 - 2 - 3 - 4 - 5 - 6 - 7 Strongly Agreement? Strongly Yes No 1 - 2 - 3 - 4 - 5 - 6 Strongly support and would vote for? Strongly support (Yes / No) Yes No 1 - 2 - 3 - 4 - 5 Survey Questions: Close-ended

n Stage 7) Pre-testing the survey instrument n n Stage 8) Selecting and training interviewers n n n Fix what doesn’t seem to be working For telephone and in-person surveys Need to avoid interviewer bias Stage 9) Implementing the survey Stage 10) Coding and entering the data Stage 11) Analyzing the data and preparing a final report 11 Stages of survey research