NonExperimental designs Surveys QuasiExperiments Psych 231 Research Methods

Non-Experimental designs: Surveys & Quasi-Experiments Psych 231: Research Methods in Psychology

Announcements n Lab attendance is critical this week because group projects are being administered Attendance will be taken. n Turn in the group project rating sheet 1 n

Non-Experimental designs n Sometimes you just can’t perform a fully controlled experiment – Because of the issue of interest – Limited resources (not enough subjects, observations are too costly, etc). • • Surveys Quasi-Experiments Developmental designs Small-N designs

Surveys n What are they (review chpt 7)? – Questionnaires and interviews that ask people to provide information about themselves n Why conduct them? – To compliment experimental work • Good/common first step, can collect a lot of data about a lot of variables – Best way to collect some kinds of information: • Descriptive, behavioral, and preferential – (e. g. demographic information, recreational behavior, and attitudes)

Surveys n Advantages – Can generalize about an entire population based on relatively small samples of individuals – Large amounts of data can be collected quickly with relatively little cost (effort, time, etc. ) • But they’re often not as “cheap” as you may think – One can investigate internal events (for example, attitudes)

Surveys n Disadvantages – Correlational: causal claims shouldn’t be made – Non-response bias • Why doesn’t everybody respond? • Does response rate interact with variables of interest? – Large data sets are sometimes difficult to analyze – Self-reports may not be truthful • Response set - tendency to respond from a particular perspective (e. g. , how a “moral” person would answer)

Stages of survey research n Stage 1) Identify the focus of the study and select your research method – What are the objectives of the research? – Is a survey method the best approach? – What kind of survey should be used?

Surveys methods n Many different methods are used to administer surveys – – – Group administration (e. g. MASS testing session) Mail surveys Internet surveys Telephone surveys Face-to-face interviews Focus group interviews

Stages of survey research Stage 2) Determining the research schedule and budget n Stage 3) Establishing an information base n – Find out what’s been done, what’s known • E. g. , Find other related surveys n Stage 4) Identify the sampling frame – The actual population that the sample is drawn from (as opposed to the ideal population) • Think of it as operationalizing the conceptual level population

Stages of survey research n Stage 5) Determining the sample size and sampling method – Review Probability and Non-Probability methods

Voluntary response methods n A kind of convience sampling methods commonly used • TV uses a lot of these – call XXX-YYYY if you support Y – call XXX-ZZZZ if you support Z • Problem: You typically get only individuals with strong opinions to respond, so the results are often extremely biased

Importance of sample size n Sampling error - how is the sample different from the population? – Confidence intervals • An estimate of where the mean or percentage in the overall population is, based on the sample data – “John Doe has 55% of the vote, with a margin of error ± 3%” • Margin of error (that “± 3%” part) – Which would you be more likely to believe » We asked 10 people … » We asked 1000 people … – The larger your sample size, the smaller your margin of error will be.

Survey Questions n Stage 6) Designing the survey instrument – Question construction: How the questions are written is very important • Clearly identify the research objectives – Do your questions really target those research objectives? • Take care wording of the questions – Keep it simple, don’t ask two things at once, avoid loaded or biased questions, etc. • How should questions be answered?

Survey Questions n Question types – Open-ended (fill in the blank, short answer) • Can get a lot of information, but • Coding is time intensive and potentially ambiguous – Close-ended (pick best answer, pick all that apply) • Easier to code • Response alternatives are the same for everyone – Rating scales • Used for “how much” judgments – e. g. , Likert scale – measures attitudes, agree/disagree • Take care with your labels – Range of scores, anchors

Stages of survey research cont. n Stage 7) Pre-testing the survey instrument – Fix what doesn’t seem to be working n Stage 8) Selecting and training interviewers – For telephone and in-person surveys – Need to avoid interviewer bias Stage 9) Implementing the survey n Stage 10) Coding and entering the data n Stage 11) Analyzing the data and preparing a final report n

Error in survey research n Sampling error n Response rate – What proportion of the sample actually responded to the survey • Hidden costs here - what can you do to increase response rates – Non-response error (bias) • Is there something special about the data that you’re missing? From the people who didn’t respond n Measurement error – Are your questions really measuring what you want them to?

Quasi-experiments What are they? n – n Almost “true” experiments, but with an inherent confounding variable General types 1) An event occurs that the experimenter doesn’t manipulate – Something not under the experimenter’s control » (e. g. , flashbulb memories for traumatic events) 2) Interested in subject variables 1. high vs. low IQ, males vs. females 3) Time is used as a variable

Quasi-experiments n Advantages – Allows applied research when experiments not possible – Threats to internal validity can be assessed (sometimes)

Quasi-experiments n Disadvantages – Threats to internal validity may exist – Designs are more complex than traditional experiments – Statistical analysis can be difficult • Most statistical analyses assume randomness

Quasi-experiments Program evaluation n – Research on programs that is implemented to achieve some positive effect on a group of individuals. – – e. g. , does abstinence from sex program work in schools Steps in program evaluation – – – Needs assessment - is there a problem? Program theory assessment - does program address the needs? Process evaluation - does it reach the target population? Is it being run correctly? Outcome evaluation - are the intended outcomes being realized? Efficiency assessment- was it “worth” it? The the benefits worth the costs?

Quasi-experiments n Nonequivalent control group designs – with pretest and posttest (most common) (think back to the second control lecture) Non-Random Assignment Dependent Variable Measure Independent Variable Experimental group Dependent Variable Measure participants Measure Control group Measure – But remember that the results may be compromised because of the nonequivalent control group (review threats to internal validity)

Quasi-experiments n Interrupted time series designs – Observe a single group multiple times prior to and after a treatment Obs Obs Treatment Obs Obs • Look for an instantaneous, permanent change n Variations of basic time series design – Addition of a nonequivalent no-treatment control group time series OOOTOOO & OOO_OOO – Interrupted time series with removed treatment • If treatment effect is reversible

Next time Go to labs this week, attendance will be taken n Non experimental designs cont. n – Read chapters 9 & 13 n Reminder, journal summary 2 (Ariely, D. & Wertenbroch, K. (2003). Procrastination, deadlines, and performance… is coming up (due next week in class). Please put your GA’s name on it