EVALUATION NOTES BERGEN COURSE SPRING 2010 Petra Todd

EVALUATION NOTES BERGEN COURSE SPRING, 2010 Petra Todd University of Pennsylvania Department of Economics

The Evaluation Problem Will study econometric methods for evaluating effects of active labor market programs Employment, training and job search assistance programs School subsidy programs Health interventions

Key questions Do program participants benefit from the program? Do program benefits exceed costs? What is the social return to the program? Would an alternative program yield greater impact at the same cost?

Goals Understand the identifying assumptions needed to justify application of different estimators Statistical assumptions Behavioral assumptions Assumptions with regard to heterogeneity in how people respond to a program intervention

Potential Outcomes Y 0 – outcome without treatment Y 1 – output with treatment D=1 if receive treatment, else D=0 Observed outcome Y=D Y 1+(1 -D) Y 0 Treatment Effect Δ= Y 1 -Y 0 Δ not directly observed, missing data problem

Other parameters of interest Proportion of people benefiting from the program Pr(Y 1>Y 0|D=1)=Pr(Δ>0|D=1) Distribution of treatment effects F(Δ|D=1, X) Selected quantile Inf {Δ: F(Δ|D=1, X)>q}

Model for potential outcomes with and without treatment Model: Y 1=Xβ 1+U 1 Y 0=Xβ 0+U 0 E(U 1|X)=E(U 0|X)=0 Observed outcome: Y=Y 0+E(Y 1 -Y 0) Y= Xβ 0+D(Xβ 1 - Xβ 0)+U 0+D(U 1 -U 0)

Distinction between TT and ATE TT=E(Δ|D=1, X)=Xβ 1 - Xβ 0+E(U 1 -U 0|D=1, X) ATE= E(Δ|X)=Xβ 1 - Xβ 0 TT depends on structural parameters as well as means of unobservables Parameters are the same if (A 1) U 1=U 0 (A 2) E(U 1 -U 0|D=1, X)=0 Condition (A 2) means that D is uninformative on U 1 -U 0, , i. e. ex post heterogeneity but not acted on ex ante

Three Commonly Made Assumptions from least to most general Coefficient on D is fixed (given X) and is the same for everyone (most restrictive) U 1=U 0 Y=Xβ+Dα(X)+U E(Y 1 -Y 0|X, D)= α(X)

Coefficient on D is random given X, but U 1 -U 0 does not help predict participation in the program Pr(D=1| U 1 -U 0 , X)=Pr(D=1|X) which implies E(U 1 -U 0 |D=1, X)= E(U 1 -U 0 |X) Coefficient on D is random given X and D helps predict program participation (least restrictive) E(U 1 -U 0 |D=1, X)≠E(U 1 -U 0 |X)

How Can Randomization Solve the Evaluation Problem? Comparison group selected using a randomization devise to randomly exclude some fraction of program applicants from the program Main advantage – increase comparability between program participants and nonpartcipants Have same distribution of observables and of unobservables Satisfy program eligibility criteria

What problems can arise in social experiments? Randomization bias – occurs when introducing randomization changes the way the program operates Greater recruitment needs may lead to change in acceptance standards Individuals may decide not to apply if they know they will be subject to randomization

Contamination bias – occurs when control group members seek alternative forms of treatment Ethical considerations – there may be opposition to the experiment and some sites may refuse to participate, which poses a threat to external validity Dropout – some of the treatment group members may drop out before completing the program Sample attrition – may have differential attrition between the treatment and control groups

At what stage should randomization be applied? Randomization after acceptance into the program Randomization of eligibility Let R=1 if randomized (treatment group), R=0 if randomized out (control group) Let Y 1* and Y 0* denote outcomes Let D* denote someone who applies to the program and is subject to randomization

How does program dropout affect experiments? Can define treatment as “intent-to-treat” or “offer of treatment, ” in which case dropout not a problem If dropout occurs prior to receiving the program (i. e. dropouts do not get treatment), then could treat it like randomization on eligibility.

Because eligibility is randomized, Pr(D=1|X, e=1)=Pr(D=1|X, e=0) Pr(D=0|X, e=1)=Pr(D=0|X, e=0) E(Y 0|X, e, D=1)= E(Y 0|X, D=1) E(Y 1|X, e, D=1)= E(Y 1|X, D=1) Thus, difference in previous two equations gives Pr(D=1|X, e=1){E(Y 1|X, e, D=1)-E(Y 0|X, D=1)}

What about control group contamination? Not necessarily a problem if willing to define benchmark state as being excluded from the program

What about sample attrition? Attrition is a problem that is common to both experimental and nonexperimental studies Attrition occurs when some people are not followed in the data (maybe due to nonresponse) If attrition is nonrandom with respect to treatment, then attrition requires the use of nonexperimental evaluation methods

Sources of bias in estimating E(Δ|X), E(Δ|X, D=1)

Traditional (Simple) Regression Estimators Cross-section Before-after Difference-in-differences

“Ashenfelter’s Dip” Mean Y D=1 D=0 T=0

Before-after estimators

Drawbacks and Advantages of before-after approach Drawbacks Identification breaks down in the presence of timespecific intercepts Can be sensitive to choice of time periods because of Ashenfelter Dip pattern Advantage minimal data requirements - only requires data on participants.

Cross-section estimators

Difference-in-difference estimators

Advantages Allows for time-specific intercepts that are common across groups Consistent under fixed effect error structure – therefore allows for time-invariant unobservables to affect participation decisions and program outcomes

Matching Estimators Assume have access to data on treated and untreated individuals (D=1 and D=0) Assume also have access to a set of X variables whose distribution is not affected by D F(X|D, YP)=f(X|YP) where YP=(Y 0, Y 1) “potential outcomes”

Matching estimators pair treated individuals with observably similar untreated individuals Usually assumed that (Y 0, Y 1) ╨ D | X (M-1) or Pr(D=1|X, Y 0, Y 1) = Pr(D=1|X) and 0<Pr(D=1|X)<1 (M-2) To justify this assumption, individuals cannot select into the program based on anticipated treatment impact

Let n denote number of observations in the treatment group A typical matching estimator for TT takes the form:

is an estimator for the matched no treatment outcome Recall, that (M-1) implies

How does matching compare to a randomized experiment? Distribution of observables will by construction be the same matched control group as in the treatment group However, distribution of unobservables not necessarily balanced across groups Experiment has full support (M-2), but with matching there can be a failure of the common support condition (when matches cannot be found)

Practical problems in Matching Problems How to construct match when X is of high dimension How to choose set of X values What do to if Pr(D=1|X)=1 for some X (violation of common support condition (M-1))

Rosenbaum and Rubin (1983) Theorem Provide a solution to the problem of constructing a match when X is of high dimension Show that (Y 0, Y 1) ╨ D | X Implies (Y 0, Y 1) ╨ D | Pr(D=1|X) Reduces the matching problem to a univariate problem, provided Pr(D=1|X) can be parametrically estimated Pr(D=1|X) is known as the propensity score

Matching can be implemented in two steps Step 1: estimate a model for program participation, estimate the propensity score P(Xi) for each person Step 2: Select matches based on the estimated propensity score

Ways of constructing matched outcomes Define a neighborhood C(Pi) for each person i Є{Di=1} Neighbors are persons in {Dj=0} for whom Pj Є C(Pi) Set of persons matched to i is Ai={jЄ{Di=0} such that Pj Є C(Pi)}

Nearest Neighbor Matching C(Pi)=min || Pi-Pj || j jЄ{Di=0} => Ai is a singleton set Caliper matching Matches only made if || Pi-Pj ||<ε for some prespecified tolerance (tries to avoid bad matches)

Kernel Matching Estimate matched outcomes by nonparametric regression

Local Linear Regression Matching

Difference-in-difference matching Assume (Y 0 t-Y 0 t’) ╨ D | Pr(D=1|X) 0<Pr(D=1|X)<1 Main advantage Allows for time invariant unobservable differences between the treatment group and the control group Selection into the program can be based on the unobservables

Should matches be reused? If don’t reuse, then results will not be invariant to the order in which observations were matched

Balancing Tests: Checking the Specification of the Propensity Score Model By R&R Theorem, for any set of variables Z

Different kinds of “balancing tests” If conditioning on estimated value of P(Z) there is additional dependence on Z then this can be viewed as misspecification in P(Z) In practice, researchers often group data according to values of P(Z) (e. g. 5 strata) and compare means of Z within each group (how to choose strata not entirely clear)

Or estimate regression…

If balancing tests fail… Refine propensity score model and reestimate Could use a semiparametric approach to estimate P(Z) If estimate P(Z) fully nonparametrically, then curse of dimensionality returns and there is no gain to using the propensity score methodology.

What variables to put in the propensity score? Econometric Models of Program Participation Assume individuals have the option of taking training in period k Prior to k, observe Y 0 j, j=1. . k After k, observe two potential outcomes (Y 0 t, Y 1 t) To participate in training, individuals must apply and be accepted, so there may be several decision-makers determining who gets training D=1 if participates, =0 else Assume participation decisions are based on maximization of future earnings

Simple model of participation n n D=1 if First term is earnings stream if participates in program C is the direct cost of training Last term is earnings stream if do not participate Ik is the information set at time k used to form expectations

Implications of this simple decision rule Past earnings are irrelevant except for value in predicting future earnings Persons with lower foregone earnings or lower costs are more likely to participate in programs Older persons and persons with higher discount rates are less likely to participate The decision to take training is correlated with future earnings only through the correlation with expected future earnings

Special case of above model Assume constant treatment effect α D=1 if expected rewards exceed costs n. If earnings temporarily low (e. g. unemployed), people are more likely to enroll in the program n. Model is consistent with Ashenfelter’s Dip Pattern

Model of the decision process Let IN=H(X)-V H(X) = expected future rewards V = costs=C+Y 0 k (assumed unknown) If V assumed to be independent of X, then could estimate by logistic or probit model: Pr(D=1|X)=e. H(X)/ {1+ e. H(X)} Pr(D=1|X)=Φ((H(X)-μ 1)/σv)

Evidence on Performance of Matching Estimators

Control function methods References: Roy (1951), Willis and Rosen (1979), Heckman and Honore (1990), Heckman and Sedlacek (1985), Heckman and Robb (1985, 1986) Allow selectivity into the program to be based on unobservables, explicitly model and control for potential selectivity bias Conventional to assume unobservables are normally distributed, but can relax normality

Model for outcomes

Comparison of Control function and matching methods

Normal model Note that assumption of normal model in inconsistent with assumption of matching estimator

Bias Function Matching assumption implies that B(P(Z))=0. Difference-in-difference matching assumes that bias function differences over time

Decomposition of Sources of Bias for B=E(Y 0|D=1)E(Y 0|D=0)

Propensity Score Distribution

Pointwise Bias and Comparison with Normal Model

Pointwise bias over time, conditional on P

Nonexperimental Estimators: Regression Discontinuity Methods Rule determining who gets treatment, but assignment is nonrandom Probability of getting treatment changes discontinuously as a function of underlying variables

Previous research Introduced in Thistlethwaite and Campbell (1960) Analyzed by Goldberger (1972) in context of evaluating education interventions Applications in Berk and Rauma (1983), van der Klaauw (1996) and Angrist and Lavy (1996) Many studies rely implicitly on nonlinearities or discontinuities in the treatment assignment rules (e. g. Black, 1996, and Angrist and Krueger, 1991)

Post Test score c Pre-test score

Questions How does the RD design provide additional sources of identifying information? How can treatment effects be recovered with minimal parametric restrictions? What is the relationship between RD and IV estimators?

Sharp design Pr(D=1|z) z

Fuzzy design Pr(D=1|z) c z

IV and Local IV (LIV) Estimators

Suppose binary instrument Z Identifying assumption is that

When does the IV estimator identify treatment-on-the-treated?

Case 1: Common effect

Case 2: heterogenous impacts

Examples Angrist (1990) uses draft lottery as instrument. Could be invalid for TT parameter Firms take into account lottery numbers in making hiring decisions Workers take actions to avoid draft Moffit (1996) uses cross-section variation in welfare benefits as an instrument for participation in a job training program Could be invalid if anticipated benefit correlated with welfare benefit

LATE Imbens and Angrist (1994, Econometrica) show that even if the assumptions that would justify application of IV for purpose of estimating TT are not valid, the IV estimator still identifies the LOCAL AVERAGE TREATMENT EFFECT (LATE) LATE is the average treatment effect for the subset of individuals induced to change their treatment status by the instrument

Distinction between “always-takers, ” “compliers, ” “never-takers” LATE is the average treatment effect for compliers only and compliers cannot be identified in the data Size of the group of compliers is unknown and the group may be instrument-dependent LATE assumes that the instrument affects everyone’s propensity to take the treatment in the same way (monotone response to the instrument)

Examples of LATE interpretation In Angrist (1990) example, get effect of military service for the subset induced to enter the military by the draft lottery (excludes those who always join or who never go, despite the draft) Angrist and Krueger (1994) study effect of schooling on earnings using compulsory schooling laws as an instrument Gives treatment effect for the subset induced to enter school by the instrument (tend to be low level schooling types) Angrist and Evans (1998) – effect of fertility on labor supply using twins as an instrument

MTE and LIV estimation (Heckman and Vytlacil, 2005 Econometrica) Develop a unifying theory for how TT, ATE, and LATE all relate to one another Propose a new concept called the “marginal treatment effect” (MTE) and show to estimate it and how to build up the other parameters from it.

Treatment effect model

Parameters of interest

Interpret LATE in terms of MTE

TT, ATE, LATE as function of MTE

Estimation strategy Uses fact that MTE is a limiting form of LATE

Bounding approaches (Heckman and Smith, 1995, Manski, 1997) Recall that from experiments, we can only learn about the marginal distributions of Y 0 and Y 1 and not about the joint distribution Let Y 0 and Y 1 denote a discrete outcome, such as employment status (Y 0, Y 1) can take on the values (E, E), (E, N), (N, E) and (N, N) Would like to know treatment effect= PNE-PEN

We only see marginals

Frechet-Hoeffding bounds Upper bound – prob of joint event cannot exceed probability of the events that compose it Lower bound – the sum of the individual cell probabilities must equal one