Using the Instrumental Variables Technique in Educational Research

Using the Instrumental Variables Technique in Educational Research By Larry V. Hedges Northwestern University

Outline • The place of IV in educational research methodology • The classical econometric justification of IV • The modern statistical approach to IV and causal inference • Implementing IV analyses • What can go wrong • Practical problems in IV

Disclaimer This talk is intended to be non-technical, therefore: No matrix algebra will be used Some technical details will be glossed over For example, I will speak of bias and accuracy in situations where the actual moments of estimates do not exist The object is to build intuition and understanding not to be rigorously technically correct

Estimating Treatment Effects Consider treatment assignment (dummy variable) X and outcome Y Regress Y on X Y i = β 0 + β 1 X i + ε i The estimate of β 1 is just the difference between the mean Y for X = 1 (the treatment group) and the mean Y for X = 0 (the control group) Thus the OLS estimate is = β 1 +

Estimating Treatment Effects (With Random Assignment) If the treatment is randomly assigned, then X is uncorrelated with ε (X is exogenous) If X is uncorrelated with ε if and only if But if , then the mean difference is = β 1 + = β 1 This implies that standard methods (OLS) give an unbiased estimate of β 1, which is the average treatment effect That is, the treatment-control mean difference is an unbiased estimate of β 1,

What goes wrong without randomization? (Simple Case) If we do not have randomization, there is no guarantee that X is uncorrelated with ε (X may be endogenous) Thus the OLS estimate is still = β 1 + If X is correlated with ε, then Hence does not estimate β 1, but some other quantity that depends on the correlation of X and ε If X is correlated with ε, then standard methods give a biased estimate of β 1

What goes wrong without randomization? When you regress Y on X, Y = β 0 + β 1 X + ε and the OLS estimate of β 1 can be described as But since X and ε are correlated, b. OLS does not estimate β 1 but some other quantity that depends on the correlation of X and ε

Instrumental Variables Natural experiments are naturally occurring situations where we want to know the effect of variable X on Y and there is a variable Z related to X, but not ε Another way so say this is: Z effects Y only through X This variable Z is called an instrumental variable It can be shown that is an unbiased estimator of β 1 in large samples but not in small samples (b. IV is consistent)

Instrumental Variables One way to see this is in terms of two regression equations Y i = β 0 + β 1 X i + ε i Xi = γ 0 + γ 1 Zi + ηi Note that, in this model X is endogenous (may be correlated with ε) The instrumental variables model requires that: 1. γ 1 ≠ 0 so that Z predicts X, and 2. Z uncorrelated with ε (Z is exogenous) [Cov{ε, Z} = 0]

Instrumental Variables You can see the logic of IV as follows

Instrumental Variables Recall the two regression equations Yi = β 0 + Xiβ 1 + εi Xi = γ 0 + Ziγ 1 + ηi This is why instrumental variables methods are associated with simultaneous equations methods in econometrics In this formulation, Zi and Xi can be vectors, so you can have • several X variables, only some of which are endogenous and • several Z variables only some of which are instruments (but you must have more instruments than endogenous X variables) The instrumental variables model requires that γ 1 ≠ 0 and Z uncorrelated with ε

Instrumental Variables Remember: To be an instrument Z must be: • Relevant (Z must be related to the endogenous variable X) • Exogenous (Z must be related to the outcome Y only through X) Failure of either condition is a problem! But both conditions can be hard to satisfy at the same time

Example Experiments with imperfect compliance Effect of intent to treat, versus treatment on the treated Intent to treat estimate Compare Y for all those assigned to treatment 1 to those assigned to treatment 0 This estimates the causal effect on Y of assignment to treatment It does not measure the effect of actually receiving the treatment unless there is perfect compliance Experimental methods cannot estimate the effect of receiving the treatment, because that cannot be randomly assigned (without perfect compliance) For example, families that use vouchers may be systematically different than those who do not in ways that affect Y

Example Experiments with imperfect compliance Voucher experiments We may want to know the causal effect of using vouchers But not all families assigned vouchers use them Because of vouchers is not randomly assigned, it may be correlated with residuals Random assignment to receive vouchers (is? ) an instrument because • Voucher assignment is related to voucher use • Voucher assignment may affect school achievement only through voucher use

Example Experiments with imperfect compliance This same idea can be applied to study the effect of receiving treatment (the effect of treatment on the treated) in many settings It can also be used to study the effect of the “active ingredients” in imperfectly implemented treatments It can (more cautiously) be used to study effects of a treatment where there is an instrument that does not arise via random assignment

Other examples of IV Studies

Estimating Causal Effects The Rubin-Holland-Rosenbaum model starts with 2 potential responses for each unit r 1 i = outcome unit i experiences in treatment 1 r 0 i = outcome unit i experiences in treatment 0 The causal effect of treatment 1 versus 0 on unit i is defined as τi = r 1 i – r 0 i You can’t estimate τi directly, but you can estimate the average causal effect in some circumstances, like a randomized experiment

Estimating Causal Effects (Randomized Experiments) Let Z = {0, 1} be a variable that expresses treatment assignment In a perfectly implemented randomized experiment, treatment assignment (Z) is uncorrelated with both r 1 i and r 0 i, so E{r 1 i | treatment 1 (Z = 1)} = E{r 1 i} E{r 0 i | treatment 0 (Z = 0)} = E{r 0 i} Thus E{r 1 | Z = 1} – E{r 0 | Z = 0} = E{r 1 – r 0} = So the estimate of the treatment effect is unbiased

Estimating Causal Effects (IV Studies) Consider IV within randomized experiments Random assignment Z, with endogenous X (believed to be the efficacious causal component of treatment) We want to know the causal effect of the endogenous variable X on outcome Y For example • Effect of voucher use in randomized choice studies • Effect of treatment implementation • Effect of using specific instructional methods

Estimating Causal Effects (IV Studies) IV can estimate causal effects of X on Y, if the following assumptions hold: 1. SUTVA 2. Random assignment of Z 3. Exclusion restriction (exogeneity of Z) 4. Nonzero causal effect of Z on X 5. Monotonicity (no defiers) Then the IV estimate is an estimate of the average treatment effect for those who comply with assignment

Unit’s Reaction to Treatment We can characterize unit’s reaction to treatment into four categories • Compliers (do what they are assigned to do) • Always takers (get treatment regardless of assignment) • Never takers (never get treatment regardless of assignment) • Defiers (always do the opposite of what is assigned) [Note that we ruled out defiers by hypothesis] Note that we cannot necessarily identify individuals are which

Estimating Causal Effects (IV Studies) Note that the causal effect of treatment on always takers and never takers is 0 by definition We can also see the IV estimate as the ratio of two causal effects (two intent to treat estimates)

Carrying Out IV Analyses Recall the description of IV in terms of two regression equations Y i = β 0 + β 1 X i + ε i Xi = γ 0 + γ 1 Zi + ηi Two-stage least squares estimation involves • Regressing X on Z to get estimates • Regressing Y on of X to get an estimate of β 1 Specialized programs are also available in many packages (e. g. , STATA or SAS) There also other, more complex procedures (such as LIML)

What Can Go Wrong In the Use of IV Failure of the assumptions! Failure of exogeneity (Z influences Y though other variables than X) Failure of relevance (Z has only a weak relation to X) Both of these kinds of failures are quantitative, not qualitative Choice of instruments may involve a tradeoff between these two kinds of failures But also, IV is a large sample procedure, even when assumptions are met it is only guaranteed to be unbiased in large samples

Violation of IV Assumptions It is important to distinguish between two situations: 1. The assumption of exogeneity is met exactly and the relevance may be small (but nonzero) [weak instruments] In this case the only bias is due to small sample bias in estimation 2. The exogeneity assumption is not met exactly In this case there is additional (large sample) bias due to direct causal effect of Z on Y The analysis of bias is quite different in these two cases!

Exogenous, but Weak Instruments Even when assumptions are perfectly met, IV is not unbiased in small (finite) samples Finite sample bias can be non-negligible (e. g. , 20 - 30%), even when the sample size is over 100, 000 if the instrument is weak (Z is only weakly correlated with X) The relative bias of b. IV (versus b. OLS) is approximately 1/F where F is the F-statistic for testing the relation between the instrument (Z) and endogenous variable (X) A small value of F, even if it is large enough to be statistically significant signals possible large bias in b. IV

Exogenous, but Weak Instruments Measuring strength of instruments: The concentration parameter One interpretation of the concentration parameter is related to the Ftest statistic in the regression of X on Z is a test of the hypothesis that γ = 0: k(F – 1) estimates λ where k is the number of instruments The accuracy of b. IV (2 SLS) estimate depends on λ, (λ functions like a sample size)

Testing for Weak Instruments It is not sufficient that the relation between Z and X is statistically significant Need to test whether λ/k exceeds a threshold (below which instruments are weak enough to imperil inference) Two definitions of ‘weak enough to imperil inference, ’ and both can be tested with first stage F for relation of Z and X (Stock & Yugo, 2005): 1. Bias of b. IV exceeds 10% of the bias of b. OLS Requires F > 10 2. Actual level of 5% significance test exceeds 15% Requires F > 24

Exogenous, but Weak Instruments Exact (small sample) results are available, but very complex (almost to the point of being uninformative) In general, more instruments increases the relevance of the instrument set (increases the first stage F) But, too many instruments increases small sample bias (compared to few instruments) In general it is best to have as few instruments as possible, and for them to be strongly correlated with X (the endogenous variable)

There are Several IV Methods I focused on 2 SLS, the most widely used IV method There are more complex competitors, such as the Limited Information Maximum Likelihood (LIML) estimation Analyses of these methods are difficult too. Large sample methods can help, but There at least 4 different large sample (asymptotic) models for analyzing IV (and they often give different results) One of these suggests that 2 SLS is equivalent to LIML Small sample studies (not definitive) suggest that LIML may be superior to 2 SLS in small samples

There are Several IV Methods But the full story is not completely clear (e. g. , how much this finding depends on normality) and it is not simple Although it is generally found that 2 SLS has particularly poor finite sample behavior, each alternative estimator seems to have its own pathologies when instruments are weak. (Andrews & Stock, 2005, p. 2)

Failure of Exogeneity Let H be the direct causal effect of Z on Y Then if the exclusion restriction (exogeneity) is violated, the (large sample, large λ) bias in b. IV is This shows that bias is reduced when the instrument is relevant (strong correlation between Z and X), so the odds of being a noncomplier are small

Failure of Exogeneity Failure of exogeneity may introduce large biases that are hard to quantify precisely because they depend on unobservables Usually, this assumption will be (somewhat) false The best we can do is often to be skeptical and to make sure exogeneity is highly plausible in the setting to which we apply IV

IV Can Provide Valid Estimates There applications in which IV does provide credible estimates Krueger’s (1999) IV estimate of the effects of actual class size on achievement using randomization as an instrument Howell et al. ’s (2000) IV estimate of the effects of using school vouchers on achievement using randomization as an instrument Bloom et al. ’s (1997) IV estimate of the effects of JTPA on earnings using randomization as an instrument

Practical Problems with IV How do we know if Z is exogenous? Isn’t randomization always a good instrument? No! Consider a randomized experiment to change instruction (using many sites or schools)

Practical Problems with IV Z is assignment to treatment to change instruction X is a measure of the instruction targeted by treatment Is Z relevant (a strong instrument)? Hard to tell a priori (e. g. , if Z is dichotomous, X is continuous, Z may not explain much variance in X) Is Z (exogenous)? Why should Z not influence Y through other unmeasured instructional practices?

Practical Problems with IV Possible Solution Include other instructional practices as covariates or endogeneous variables But the number of instruments must exceed the number of endogenous variables—now we need more instruments We could include Z-by-site interactions as instruments But now we have increased the number of instruments, which may increase bias

Practical Problems with IV Assignment may have direct effects on Y if volunteers want the treatment (Shadish, Cook, & Campbell, 2002) Assignment may influence units to get alternatives • Tutoring • Teacher induction • Health care • After school programs Assignment may have a discouraging effect on control group

Conclusions IV can make possible estimates of causal effects without random assignment in some cases But it is no panacea Often, it will be difficult to find instruments that are both relevant (strong enough) and exogenous IV estimation is a complicated subject and good theory for all of the relevant issues is not available For example, all of theory I have mentioned assumes simple random sampling so it does not take clustered sampling (of the kind in most education experiments) into account

Select Bibliography Causal Inference Rubin, D. B. (1974). Estimating causal effects in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688701. Angrist, J. D. , Imbens, G. W. , & Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91, 444 -455. Imbens, G. W. & Angrist, J. D. (1994). Identification and estimation of local average treatment effects. Econometrica, 62, 467 -475. Natural Experiments Angrist, J. D. & Krueger, A. B. (2000). Instrumental variables and the search for identification: From supply an demand to natural experiments. The Journal of Economic Perspectives, 15, 69 -85.

Select Bibliography Weak Instruments Bound, J. , Jaeger, D. A. , & Baker, R. M. (1995). Problems with instrumental variables estimation when the correlation between the instruments and the endogenous explanatory variable is weak, Journal of the American Statistical Association, 90, 443 -450. Staiger, D. , & Stock, J. H. (1997). Instrumental variables regression with weak instruments. Econometrica, 65, 557 -586. Nelson, C. R. & Startz, R. (1990). Some further results on the exact small sample properties of the instrumental variable estimator. Econometrica, 58, 967 -976. Stock, J. H. , Wright, J. H. , & Yogo, M. (2002). A survey of weak instruments and weak identification in generalized method of moments. Journal of Business and Economic Statistics, 20, 518 -529 Buse, A. (1992). The bias of instrumental variable estimators. Econometrica, 60, 173 -180.