The Strength of Instrumental Variables and Their Sensitivity

The Strength of Instrumental Variables and Their Sensitivity to Unobserved Biases Dylan Small Department of Statistics, Wharton School, University of Pennsylvania Joint work with Paul Rosenbaum

A Picture of the IV Argument: WWII Veteran Status and Earnings • Does military service raise or lower earnings? • Angrist and Krueger (1994) studied this in context of WWII military service and 1980 earnings (using 5% public use sample of US Census). • Lower earnings? Military service in WWII interrupts education or career. • Higher earnings? Labor market might favor veterans, GI Bill increases education.

WWII Vets (76% of men) earned on average $4500 more in 1980 than Non-Vets. This is association not causation: WWII Vets might not be comparable to Non-Vets in terms of health, criminal behavior…

We created matched triples: men matched on quarter of birth, race, age, education up to 8 years and location of birth. This figure provides reason to doubt military service increases earnings by $4500. From 1924 to 1926, the proportion of veterans stayed about constant and the earnings stayed about the same. From 1926 to 1928, the proportion of veterans decreased by 50% but earnings increased, suggesting military service decreases earnings.

Instrumental Variables Strategy Y=Outcome W=Treatment Z=IV X=Covariates Y X W|X Z|X X Unobserved Variables Extract variation in W from Z that is free of unobserved confounders and use this variation to estimate the causal effect of W on Y. Key IV Assumptions: (1) Z independent of unobserved variables; (2) Z does not have direct effect on outcome.

Outline 1. Illustration of IV method: effect of WWII military service on earnings. 2. Randomization inference using IVs. 3. Sensitivity analysis. 4. Strength of IVs and the effect of the strength of IV on the sensitivity analysis. 5. Practical consequences.

Matched Pair Encouragement Design

Randomization Inference

95% CI for effect of military service: (-$1, 445, -$500)

Relationship to Angrist, Imbens and Rubin Setup

Instrumental Variables in Health Care Policy Applications

Sensitivity Analysis

Model for Sensitivity Analysis

Carrying out Sensitivity Analysis

Sensitivity Analysis for WWII Study

Sensitivity to Exclusion Restriction • Sensitivity to the exclusion restriction assumption (encouragement has no direct effect) is also an important issue and can be incorporated into the sensitivity analysis (as done by Angrist, Imbens and Rubin) • For time purposes, will not discuss further in this talk. See me for details.

Strength of IV • • An IV is strong if encouragement has a strong effect on treatment received; An IV is weak if encouragement has only a weak effect on treatment received. Effects of Weak IVs 1. Increased Variance 2. Increased Sensitivity to Bias

Effect of Weak IVs I: Increased Variance Y X W|X Z|X X Unobserved Variables If Z is a weak IV, then the variance of the IV estimate will be higher because less variation in W from Z can be extracted.

95% CI for effect of military service using 1926 vs. 1928 IV: (-$1, 445, -$500). 95% CI for effect of military service using 1924 vs. 1926 IV: (-$10, 130, $10, 750)

Effect of Weak IVs II: Increased Sensitivity to Bias

Model for Power Analysis

Summary: Increased Sensitivity to Bias of Weak IVs

Practical Consequences

Practical Consequences Continued

Summary • The IV method can be a powerful strategy for observational studies when there are confounders that are hard to measure and there is a “random” encouragement to receive treatment. • When encouragement is not actually random, it is important to do a sensitivity analysis. • Strong IVs are much less sensitive to bias. – Consider two studies, a small study with a strong IV and a large study with a weak IV, which would have the same power if both IVs are valid. When there is concern about the validity of IVs, the small study with a strong IV has considerable advantages.