NonExperimental designs Surveys Correlational Psych 231 Research Methods

Non-Experimental designs: Surveys & Correlational Psych 231: Research Methods in Psychology

n n Mean = 75. 2 Median = 78 Max = 98 Min = 47 n Most common errors n Between vs. within designs Independent vs. dependent vars Scales of measurement n n Exam 2 results n n Confounds vs. extraneous variables Main effects vs. interactions

n Sometimes you just can’t perform a fully controlled experiment n n Because of the issue of interest Limited resources (not enough subjects, observations are too costly, etc). • • • n Surveys Correlational studies Quasi-Experiments Developmental designs Small-N designs This does NOT imply that they are bad designs n Just remember the advantages and disadvantages of each Non-Experimental designs

n n n Stage 1) Identify the focus of the study and select your research method Stage 2) Determining the research schedule and budget Stage 3) Establishing an information base Stage 4) Identify the sampling frame Stage 5) Determining the sample method and sampling size n Review Probability and Non-Probability methods • Voluntary response method n Importance of sample size Stages of survey research

n Stage 6) Designing the survey instrument n Question construction: How the questions are written is very important • Clearly identify the research objectives • Do your questions really target those research objectives (think Internal and External Validity)? • Take care wording of the questions • Keep it simple, don’t ask two things at once, avoid loaded or biased questions, etc. • How should questions be answered (question type)? Stages of survey research cont.

Poor Was the FDC negligent by ignoring the warnings about Vioxx during testing and approving it for sale? Yes Problem: a)emotionally b) No charged c) words Unsure Do you favor eliminating the wasteful excess in the public school budget? a) b) c) Yes No Unsure Good If the FDC knew that Vioxx caused serious side effects during testing, what should it have done? a)Ban it from ever being sold b)Require more testing before approving it c)Unsure Do you favor reducing the public school budget? a)Yes b)No c)Unsure Good and poor questions

Poor Should senior citizens be given more money for recreation centers and food assistance programs? a) Yes b) No Problem: asks c) Unsure two different questions Good Should senior citizens be given more money for recreation centers? a) b) c) Yes No Unsure Should senior citizens be given more money for food assistance programs? a) b) c) Yes No Unsure Good and poor questions

Poor Are you against same sex marriage and in favor of a constitutional amendment to ban it? Good What is your view on same sex marriage? a) a) b) c) Yes No Unsure Problem: Biased in more than one direction b) c) I think marriage is a matter of personal choice I’m against it but don’t want a constitutional amendment I want a constitutional amendment banning it Problem: Asks two questions Good and poor questions

n Question types n Open-ended (fill in the blank, short answer) • Can get a lot of information, but • Coding is time intensive and potentially ambiguous n Close-ended (pick best answer, pick all that apply) • Easier to code • Same response alternatives for everyone • Take care with your labels • Decide what kind of scale • Decide number/label of response alternatives Survey Questions What is the best thing about ISU? (choose one) n 1. Location n 2. Academics n 3. Dorm food n 4. People who sell things between Milner and the Bone

n Decide what kind of rating scales • Rating: e. g. , Likert scale PSY 231 is an important course in the major. 1 Strongly Agree 2 Agree 3 Neutral 4 Disagree 5 Strongly Disagree • Semantic differential: Rate how you feel about PSY 231 on these dimensions Important _____: _____: Unimportant Boring _____: Interesting _____: • Nonverbal scale for children: Point to the face that shows how you feel about the toy. Survey Questions: Close-ended

n Decide number/label of response alternatives • Use odd number (mid point and equal # of responses above and below the mid point) • Questions should be uni-dimensional (each concerned with only one thing) • Labels should be clear Survey Questions: Close-ended

n Stage 7) Pre-testing the survey instrument n n Stage 8) Selecting and training interviewers n n n Fix what doesn’t seem to be working For telephone and in-person surveys Need to avoid interviewer bias Stage 9) Implementing the survey Stage 10) Coding and entering the data Stage 11) Analyzing the data and preparing a final report Stages of survey research cont.

n Sometimes you just can’t perform a fully controlled experiment n n Because of the issue of interest Limited resources (not enough subjects, observations are too costly, etc). • • • n Surveys Correlational Quasi-Experiments Developmental designs Small-N designs This does NOT imply that they are bad designs n Just remember the advantages and disadvantages of each Non-Experimental designs

n Looking for a co-occurrence relationship between two (or more) variables n We call this relationship a correlation. n 3 properties: form, direction, strength Correlational designs

Linear Form Non-linear

Positive Negative Y • X & Y vary in the same direction Direction Y X • X & Y vary in opposite directions X

r = -1. 0 “perfect negative corr. ” -1. 0 r = 0. 0 “no relationship” r = 1. 0 “perfect positive corr. ” 0. 0 The farther from zero, the stronger the relationship Strength +1. 0

n Looking for a co-occurrence relationship between two (or more) variables n Used for • Descriptive research • do behaviors co-occur? • Predictive research • is one behavior predictive of another? • Reliability and Validity • Does your measure correlate with others (and itself)? • Evaluating theories • Look for co-occurrence posited by theory. Correlational designs

n Looking for a co-occurrence relationship between two (or more) variables n Example 1: Suppose that you notice that the more you study for an exam, the better your score typically is n At a descriptive level this suggests that there is a relationship between study time and test performance. n For our example, which variable is explanatory and which is response? And why? n It depends on your theory of the causal relationship between the variables n n Explanatory variables (Predictor variables) Response variables (Outcome variables) Correlational designs

Y 6 Hours study Exam perf. X Y 5 6 1 6 2 4 5 6 2 3 4 1 3 2 Scatterplot n 3 1 2 3 4 For this example, we have a linear relationship, it is positive, and fairly strong 5 6 X

n n For descriptive case, it doesn’t matter which variable goes where n Correlational analysis For predictive cases, put the response variable on the Y axis n Regression analysis Y 6 Response (outcome) variable 5 4 3 2 1 1 2 3 4 5 6 X Explanatory (predictor) variable Scatterplot

n Advantages: n Doesn’t require manipulation of variable • Sometimes the variables of interest can’t be manipulated n n Allows for simple observations of variables in naturalistic settings (increasing external validity) Can look at a lot of variables at once Example 2: The Freshman 15 (CBS story) • • Is it true that the average freshman gains 15 pounds? Recent research says ‘no’ – closer to 2. 5 – 3 lbs Looked at lots of variables, sex, smoking, drinking, etc. Also compared to similar aged, non college students Correlational designs

n Disadvantages: n Don’t make casual claims • Third variable problem • Temporal precedence • Coincidence (random co-occurence) n Correlational results are often misinterpreted Correlational designs

n Example 3: Suppose that you notice that kids who sit in the front of class typically get higher grades. n This suggests that there is a relationship between where you sit in class and grades. Daily Gazzett Children who sit in the back of the classroom receive lower grades than those who sit in the front. Possibly implied: “[All] Children who sit in the back of the classroom [always] receive worse grades than [each and every child] who sits in the front. ” Better: “Researchers X and Y found that children who sat in the back of the classroom were more likely to receive lower grades than those who sat in the front. ” Misunderstood Correlational designs Example from Owen Emlen (2006)

n Sometimes you just can’t perform a fully controlled experiment n n Because of the issue of interest Limited resources (not enough subjects, observations are too costly, etc). • • • n Surveys Correlational Quasi-Experiments Developmental designs Small-N designs This does NOT imply that they are bad designs n Just remember the advantages and disadvantages of each Non-Experimental designs

What are they? n n n Almost “true” experiments, but with an inherent confounding variable General types 1) An event occurs that the experimenter doesn’t manipulate • Something not under the experimenter’s control • (e. g. , flashbulb memories for traumatic events) 2) Interested in subject variables – high vs. low IQ, males vs. females 3) Time is used as a variable Quasi-experiments

n Advantages n n n Allows applied research when experiments not possible Threats to internal validity can be assessed (sometimes) Disadvantages n n n Threats to internal validity may exist Designs are more complex than traditional experiments Statistical analysis can be difficult • Most statistical analyses assume randomness Quasi-experiments

Program evaluation n – Research on programs that is implemented to achieve some positive effect on a group of individuals. – – e. g. , does abstinence from sex program work in schools Steps in program evaluation – – – Needs assessment - is there a problem? Program theory assessment - does program address the needs? Process evaluation - does it reach the target population? Is it being run correctly? Outcome evaluation - are the intended outcomes being realized? Efficiency assessment- was it “worth” it? The the benefits worth the costs? Quasi-experiments

n Nonequivalent control group designs n with pretest and posttest (most common) (think back to the second control lecture) Independent Non-Random Dependent Variable Assignment Measure Experimental group Dependent Variable Measure participants Measure Control group Measure – But remember that the results may be compromised because of the nonequivalent control group (review threats to internal validity) Quasi-experiments

n Interrupted & Non-interrupted time series designs n Observe a single group multiple times prior to and after a treatment Obs Obs Treatment Obs Obs • Look for an instantaneous, permanent change • Interrupted – when treatment was not introduced by researcher, for example some historical event n Variations of basic time series design • Addition of a nonequivalent no-treatment control group time series OOOTOOO & OOO_OOO • Interrupted time series with removed treatment • If treatment effect is reversible Quasi-experiments