Introduction to Research Design External Validity Simple Research

Introduction to Research Design External Validity Simple Research Designs Extraneous Variable Control

External Validity The extent to which the results of the study can generalize beyond the specifics of the experimental situation. • Would you get the same results if you – used different subjects – manipulated the IV in a different way – measured the DV in a different way – The setting was different – etc.

Threats to External Validity: Testing x Treatment Pretest Postest O X O Control Group Design O O • Is the effect of the treatment the same for pretested subjects as for subjects not pretested? • Can use a Solomon Four Group Design • Or a Posttest Only Control Group Design

Threats to External Validity Selection x Treatment : Does the treatment interact with characteristics of the subjects? • Can you generalize your results to different sorts of subjects? – from laboratory rats to feral rats? – from laboratory rats to humans? – from US college students to those in China? – from college students to farmers?

Threats to External Validity: Reactive Effects of Experimental Arrangements Can you generalize the results found with observed subjects to subjects who are not being observed? • Difficult or impossible to observe without affecting. • Heisenberg uncertainty principle. • Demand characteristics.

Threats to External Validity: Multiple Treatment Interference Do results found with subjects used in many experiments generalize to subjects without multiple treatment experience? • Expensive monkeys used in medical research. • Trained human subjects used in perceptual research.

Simple Research Designs: One-Shot Case Study (X) O • • • No variable is manipulated. Just find group that has experienced X. Example: Post tornado research here. No pretest data, no comparison group. Of little if any scientific value.

Randomized Pretest-Posttest Control Group Design R O X O R O O • Random assignment to groups – can assume groups equivalent prior to treatment. • Thus, no selection threat and • No Selection x Maturation threat.

Randomized Pretest-Posttest Control Group Design R O X O R O O • Effects of history, maturation, testing, instrumentation can be assessed by prepost comparison in control group. • They are controlled if they exist to the same extent in both treatment and control groups.

Randomized Pretest-Posttest Control Group Design R O X O R O O • Testing x Treatment remains a threat to external validity. • Problems with random assignment – Compare the two pretest means – The Elmira College Study

Randomized Pretest-Posttest Control Group Design R O X O R O O • Factorial ANOVA: Time x Group – Mixed between subjects and within subjects – Primary interest is in the Time x Group interaction. • Independent t comparing the two groups’ change scores (equivalent to interaction analysis)

Randomized Pretest-Posttest Control Group Design R O X O R O O • Analysis of Covariance – pretest scores as covariate – posttest scores as dependent variable • All three of these have more power than independent t on posttest scores.

Switching Replications Design R O X O O O R O O X O • This is a modification of the basic design. • The control group gets the treatment too, but after the other group gets it. • This is useful when the treatment is something that should be of value to all subjects. • Also known as the waitlist control group design.

Randomized Switching Replications Design Intervention Group Pre (no X) Post X Followup Waitlist Control Time 1 (no X) Time 2 (no X) Time 3 (no X) Followup 2 Followup 3 Post X Followup

Randomized Posttest Only Control Group Design R X O R O • For controlling threats previously discussed, this is the strongest design. • But it usually has less power than designs that include within subjects comparisons. • This is a threat to statistical conclusion validity.

Independent Samples Designs • Cases are assigned to experimental groups in such a way that no correlation is expected between the scores in any one group and the scores in any other group. • AKA, Between Subjects Design • Completely Randomized Design – subjects are randomly sampled from a population of interest and randomly assigned to treatment groups.

Correlated Samples Designs • Scores are obtained in a way that produces a substantial positive correlation between the scores in any one condition and those in any other condition. • Randomized Blocks Designs – Matched Pairs – Repeated Measures (Within Subjects) – Enhanced Power

Extraneous Variable Control • Important to control extraneous variables to – Reduce noise – eliminate confounds • I’ll discuss five methods of controlling extraneous variables.

Constancy • Hold values of EVs constant across subjects and groups. – for example, use only female participants – test all subjects at same time of day, etc. • Eliminates confounds. • Reduces noises, raising power. • May reduce external validity – do results generalize to men, etc?

Balancing • Distribution of EV constant across groups – if 60% female in one group, same in others – if 70% tested during day in one group, same in others • Prevents confounding. • Does not reduce noise.

Randomization Assumed to produce balancing, thus • Confounds are eliminated. • Noise is not reduced.

Matching • Subjects matched up on EV(s) into blocks of k, where k = # of treatments. • Within each block, randomly assign one subject to each treatment. • Matched pairs, randomized blocks. • Balances groups, eliminating confounds. • Allows power-enhancing statistical analysis.

Statistical Control • Statistical techniques, such as ANCOV and ANOVA, can be used to remove the effect of extraneous variables and thus increase power. • I’ll discuss these in more detail in the next unit of this class.