Experimental Design Making your experiment more valid and

Experimental Design Making your experiment more valid and more efficient

So, the basic experimental procedure in social science is: • Gathering of a subject pool and then random assignment to conditions – Note: the sample is not random, but once you have a subject pool you flip a coin, etc. to determine which subjects receive which condition • Depending upon the experimental design, a measure of the dependent variable may be taken from the groups prior to the manipulation

• Subjects are exposed to the ‘treatment’ assigned to their group – May be some level of the factor or a zero-level (control) treatment – For example, the ‘’experimental group” may play a video game with lots of violence while the control group plays a non-violent game • Both groups are then measured on the dependent variable of interest – In this case, some measure of aggressiveness • The data are then analyzed to test the research hypothesis

Basic design (After-only, control group design) R X R R=randomization, X=manipulation, O=observation O O

What’s the effect? • The difference between group scores on the dependent variable is assumed to be due to the manipulation • But what if the groups were different at the start? – Randomization should have taken care of that – However, pure chance can sometimes lead to differences at the outset • So, measure them before the manipulation as well as after

Before-after, control group design R O 1 X R=randomization, X=manipulation, O=observation O 2

Before-after, control group design • Now there’s a new problem: – Measuring the subjects and then measuring them again later may cause them to respond differently to the second measure than they otherwise would have (true for both experimental and control subjects) – Measuring the experimental subjects before exposing them to the treatment may change their response to the manipulation

Problems with the basic design • The difference between control and experimental conditions includes more than violence – Experimenters may have control group play a nonviolent video game • Many other variables may affect the outcome – For example, gender • Measure and include in statistical analyses • Use factorial design

So: • There will always be some problems with any procedure • The goal is to minimize threats to validity as best your circumstances (including budget) allow – Especially internal validity

How do we do that? • Experimental design • Statistical analyses • Careful application of procedures

Experimental design • Two facets – Choices in identifying the independent and dependent variables to be measured and/or manipulated – Choices in developing the procedures for manipulation, measurement, environment of testing, etc.

Choosing the variables • Theory determines variables manipulated and measured • Experimental designs where the research manipulates more than one independent variable are known as factorial designs – Mediating concepts often are included

Choosing the measures • The measures are chosen to best represent the concepts in the original theory – Measurement validity • Must work in an experimental setting – Lengthy questionnaires unlikely to work – Physical measures more easily applied in a laboratory setting

Factorial designs • Factorial designs are useful because they: – Are efficient • With two independent variables we can test three hypotheses instead of just one – Don’t need three times as many subjects • However, there is a practical limit to independent variables – As the number rises, the required number of subjects increases quickly (5 per cell) – Help us avoid coming to invalid conclusions • They allow us to control for potentially confounding variables and interactions – We can test the impact of third variables • Factorial designs can be more difficult to administer

An example • We may want to test the following hypothesis: – “Exposure to Dancing with the Stars will lead to greater liking for the contestants” • We might, however, expect that men and women will be affected differently by the show • So, we might add gender as another independent variable in the design

Factorial design See Dancing with the Stars Don’t see Dancing with the Stars Men Women All groups are measured on liking for contestants

What we can test • “Main effect” of exposure to Dancing with the Stars • “Main effect” of gender • “Interaction effect” of exposure by gender • Remember: the dependent variable is liking for contestants

Results (% who like contestants) 90% 80% 70% 60% 50% Men 40% Women 30% 20% 10% 0% Didn't see DWTS Saw DWTS

Conclusions • Main effect of gender – Yes: Gender affects liking for celebs • Main effect of exposure to DWTS – Yes: Exposure increases liking • Interaction effect of exposure by gender: – Yes: Exposure has a clear positive effect for women, little or even negative effect for men

Procedural choices • How many times will you measure the subjects? – You can account for chance fluctuation by increasing the number of times the groups are measured • You increase the likelihood of testing reactivity, though

R O 1 O 2 X O 3 O 4

Additional design concerns • How many times/how long will the manipulation be presented? – Testing a single exposure to DWTS may not be valid • Viewers may watch many shows during a season

R O 1 X 1 O 2 X 2 O 3

Removal of the treatment R O 1 X O 2 -X O 3

• How many levels of the independent variable will be presented? – May want to test high v. medium v. low levels of violence, etc.

R O X(high) O R O X(medium) O R O X(low) R O O O

Factorial nomenclature • The number of treatments on each factor are crossed – 2 X 3 factorial design has 2 treatment levels on one factor and 3 treatment levels on another factor – If we tested the effects of soft, medium and loud music and gender upon ease in a social situation, we would have a 2 X 3 factorial study • Factor 1: music volume 3 levels: soft, medium and loud • Factor 2: gender 2 levels: male, female • Dependent variable: ease in a social situation

Additional design concerns • Artificiality of the testing environment – Labs allow for the greatest control over third variables • Volume, distance from screen, prevent other people from walking by, etc. – Lab experience can sometimes be made less artificial • Some commercial-testing services provide the subjects with a simulated living room for exposure • Loss of control over third variables weighed against increase in generalizability (external validity)

Statistical analyses • If you measure potential confounds you can use statistical procedures to remove their impact on the dependent variable

Careful application of the design • Unplanned variations among treatments can easily occur when conducting experiments in the real world – The person you wanted to greet everyone and provide instructions may have to leave for an emergency – Subjects may talk to each other outside the lab before they have all completed their part of the experiment – And so on

Remember: • Experiments are the most effective quantitative research method for testing hypothesized causal relationships • Experiments emphasize internal validity but are often weak with regard to external validity • Many different experimental designs exist, each having a set of advantages and disadvantages • Your goal is to provide the most valid test of theory your circumstances allow you