Internal Validity 8 things to look out for

Internal Validity 8 things to look out for before you jump to (cause-effect) conclusions Web Links

Internal validity – Definition: Can we make cause-effect conclusions? – Importance: » If we want to understand people, we need to be able to explain their behavior. Any good explanation is a causal explanation. (People answer “why? ” questions with “Because …) » If we want to help people (be effective), we need treatments that work.

3 prerequisites for internal validity 1. Know that the behavior changed (more helping when you are smiling than when not)

Three prerequisites for internal validity One reason we want to manipulate the treatment 1. Know that the behavior changed helping) 2. Know that the treatment came before the behavior changed (smiling came before helping, not after)

Three prerequisites for internal validity 1. Know that the behavior changed 2. Know that the treatment came before the behavior changed 3. Know that everything except treatment stayed the same (helping and smiling weren’t due to sunny weather)***

Keeping everything else the same is very difficult! Millions of things can change. Fortunately, these millions of things fall into eight categories (Campbell & Stanley’s 8 threats to internal validity):

Eight Threats • • History Maturation Testing • Instrumentation • • • Regression Selection by maturation Mortality

History: Changes due to outside (environmental) events other than the treatment. – Example*:

December 26, Rudolph is no longer so sensitive about his nose!

Maturation: Changes due to inside (biological) events other than the treatment (fatigue, illness, physical development, other changes inside the person) – Examples of physical maturation: » Clay Matthews As freshman As senior 6’ 1”, 170 6’ 3”, 240

The duckling matures!

(Re)Testing: The act of being tested causes changes in the participant that are reflected when participants are retested. – Example: Taking SATs, GREs multiple times – Example*

Bob has taken the eye test before. Indeed, he has a “cheat” sheet from last time.

Instrumentation: The measuring instrument—not the participant-- changes during the course of the study and these changes cause participants’ scores to change. Not testing (person changing due to taking test), it is scoring changing. – Example 1: Changes in raters or interviewers. – Other examples*

Example of Instrumentation: SAT recentering This table converts individual SAT I: Reasoning Verbal Test scores on the original scale to equivalent scores on the recentered scale. Original Recentered 800 500 580 790 800 490 570 780 800 480 560 770 800 470 550 760 800 460 540

A would-be dieter creates an instrumentation bias by hitting the scale.

Regression: Participants who get extreme scores have less extreme scores when retested, because extreme scores are extreme, in part, due to random error (e. g. , guessing on a test, random fluctuations when shooting free throws). Since random error is inconsistent, chances are that random error will not make participants scores quite so extreme the second time around.

Examples of Regression • Time-outs in basketball after the other team has gone on a run will seem to be effective • Rewards after exceptionally good performance will seem to backfire, whereas punishment after exceptionally bad performances will seem to work.

Selection: The groups were different to start with.

These individuals don’t understand selection.

People don’t just differ on 1 characteristic. Example: Smokers Vs. Nonsmokers Smokers eat more white bread. Smokers eat more sugar. Smokers eat more cooked meat. Smokers drink more whole milk. Smokers eat fewer fruits & vegetables. Smokers drink more beer. Smokers are less likely to lie on social desirability scales. Smokers are 3 X more likely to have had sex by age 19. These are only a few of the differences between these two groups!

Selection by maturation interaction: Groups that are similar (but not identical) to start with may naturally grow apart. Ex: – Arm strength of 4 th grade boys and girls – Rates of re-arrest for young versus older convicts

Mortality: Overall scores change because of people (often lower scoring participants) dropping out of the study.

§ Now that we know about the individual threats, let’s see how they affect two basic kinds of designs: § The treatment group versus no-treatment group design § The before (treatment)-after (treatment) design

Overview: Problems in comparing two groups: Two big problems: Group differences may not be due to the treatment, but to 1. mortality* 2. selection* Matching doesn’t solve selection problems because of – interactions with selection* –regression to different means*

Mortality: What if people drop out of the treatment group, but not out of the no-treatment group? Ex: AA? College? Any demanding treatment Treatment group: beginning Treatment group: end No-treatment group: beginning No-treatment group: end

Selection*: Groups being different before study began

Sources of selection bias 1. Self-assignment to group Volunteers for no-shock group Volunteers for shock group

Sources of selection bias 1. Self-assignment to group 2. Researcher assignment to group “You will be in the notreatment group. ” A biased anti-depressant study

Sources of selection bias 1. Self-assignment to group 2. Researcher assignment to group “You will be in the treatment group. ” A biased anti-depressant study

Sources of selection bias 1. Self-assignment to group 2. Researcher assignment to group 3. Arbitrary (not random) assignment to group: Selecting on differences gives you different groups. An example of why it matters: The misleading results from Brady’s study of “executive monkeys”

Why matching doesn’t work Are there identical participants? No. Can you match on all relevant variables? No. Matching on pretest scores makes you vulnerable to: 1. Selection by maturation (not matched on other factors that affect maturation) 2. Regressing to different means (matched on scores) rather than on the actual characteristic. Example of problems with matching: “Head Start”

Summary of problems with 2 -group design Assume groups are identical. Try to make groups identical Lose to selection Too many variables Lose to selection Grow apart Lose to selection. Random maturation measurement error fools you Lose to regression

Problems with before-after designs (Why these designs are the stars of infomercials) 1. Participants may change even without the treatment* 2. Participants’ scores may appear to change even without any change in the participant*

3 nontreatment reasons participants may change 1. Maturation (e. g. , juvenile delinquents may outgrow criminal behavior) 2. History (e. g. , measuring prejudice after 9 -11) 3. Testing (e. g. , more experience with test makes participant more knowledgeable or relaxed)

Three ways measurement changes may cause participants’ scores to change 1. Instrumentation* 2. Regression* 3. Mortality *

Instrumentation Changes in how participants are measured Examples: “You’re looking fairly happy now. ”

Regression Changes in the extent to which measurement is affected by random error – Ex: Extreme scorers, people in slumps, person going to doctor – “Nothing recedes like success. ” --Paul Rosenzweig

Mortality Changes in how many participants are measured – Real problem if treatment is long or intense: May be comparing the whole group (pretest group) with an elite subgroup who did not quit (post-test group)

Chapter Exercises Pages 332 -333

Conclusions: Like Treatment-No Treatment Group Design, Pretest. Posttest Design Lacks Internal Validity Biological changes: Maturation Environmental changes: History Pretesting effects: Testing Instrument changes Regression Mortality Participants change Participants’ scores change

Chapter 9 Chapter Websites Quick Visuals Quizzes Teacher Chapter Outline Quiz 1 Student Concept Map Quiz 2 Chapter Summary Quiz 3 Demonstrations/Labs