NonExperimental Methods Florence Kondylis Quasi Experimental Methods I
- Slides: 31
Non-Experimental Methods Florence Kondylis Quasi Experimental Methods I AADAPT Workshop Latin America Brasilia, November 16 -20, 2009
What we know so far Aim: We want to isolate the causal effect of our interventions on our outcomes of interest Use rigorous evaluation methods to answer our operational questions Randomizing the assignment to treatment is the “gold standard” methodology (simple, precise, cheap) What if we really, really (really? ? ) cannot use it? ! >> Where it makes sense, resort to non-experimental methods
Objective �Find a plausible counterfactual �Every non-experimental method is associated with a set of assumptions �The stronger the assumption, the more doubtful our measure of the causal effect ▪ Question our assumptions ▪ Reality check, resort to common sense! 3
Example: Fertilizer Voucher Program Principal Objective ▪ Increase maize production Intervention ▪ Fertilizer Vouchers distribution ▪ Non-random assignment Target group ▪ Maize producers, land over 1 Ha & under 5 Ha Main result indicator ▪ Maize yield 4
Illustration: Fertilizer Voucher Program (1) Control Group Treatment Group 14 12 10 8 (+) Impact of the program (+) Impact of external factors 6 4 2 0 5
Illustration: Fertilizer Voucher Program (2) Control Group Treatment Group 14 12 10 (+) BIASED Measure of the program impact 8 6 4 2 “Before-After” doesn’t deliver results we can believe in! 0 6
Illustration: Fertilizer Voucher Program (3) Control Group Treatment Group 14 12 10 8 6 « Before» difference btw participants and nonparticipants « After » difference btw participants and non-participants 4 2 0 >> What’s the impact of our intervention? 7
Difference-in-Differences Identification Strategy (1) Counterfactual: 2 Options 1. Non-participant maize yield after the intervention, accounting for the “before” difference between participants/nonparticipants (the initial gap between groups) 2. Participant maize yield before the intervention, accounting for the “before/after” difference for nonparticipants (the influence of external factors) � 1 and 2 are equivalent 8
Difference-in-Differences Identification Strategy (2) Underlying assumption: Without the intervention, maize yield for participants and non participants’ would have followed the same trend >> Graphic intuition coming…
Data -- Example 1 Average maize yield (T / Ha) 2007 2008 Difference (2007 -2008) Participants (P) 1. 3 1. 9 0. 6 Non-participants (NP) 0. 6 1. 4 0. 8 Difference (P-NP) 0. 7 0. 5 -0. 2 10
Data -- Example 1 Average maize yield (T / Ha) 2007 2008 Difference (2007 -2008) Participants (P) 1. 3 1. 9 0. 6 Non-participants (NP) 0. 6 1. 4 0. 8 Difference (P-NP) 0. 7 0. 5 -0. 2 11
Impact = (P 2008 -P 2007) -(NP 2008 -NP 2007) = 0. 6 – 0. 8 = -0. 2 Chart Title 2 1, 8 P 2008 -P 2007=0. 6 1, 4 1, 2 1 0, 8 0, 6 0, 4 0, 2 0 NP 2008 -NP 2007=0. 8 2007 Participants 2008 Non-Participants 12
Impact = (P-NP)2008 -(P-NP)2007 = 0. 5 - 0. 7 = -0. 2 Chart Title 2 1, 8 1, 6 1, 4 1, 2 P-NP 2007 1 =0. 7 0, 8 0, 6 0, 4 0, 2 0 P-NP 2008=0. 5 2007 Participants 2008 Non-Participants 13
Assumption of same trend: Graphic Implication Chart Title 2 1, 8 1, 6 1, 4 1, 2 1 0, 8 0, 6 0, 4 0, 2 0 Impact=-0. 2 2007 Participants 2008 Non-Participants
Summary �Negative Impact: Very counter-intuitive: Increased input use should increase yield once external factors are accounted for! �Assumption of same trend very strong 2 groups were, in 2007, producing at very different levels ➤ Question the underlying assumption of same trend! ➤When possible, test assumption of same trend with data from previous years
Questioning the Assumption of same trend: Use pre-pr 0 gram data 2, 5 2 1, 5 participants non-participants 1 0, 5 0 2006 2007 2008 >> Reject counterfactual assumption of same trends !
Data – Example 2 Average maize yield (T / Ha) 2007 2008 Difference (2007 -2008) Participants (P) 1. 5 2. 1 0. 6 Non-participants (NP) 0. 5 0. 7 0. 2 Difference (P-NP) 1. 0 1. 4 0. 4 17
Impact = (P 2008 -P 2007) -(NP 2008 -NP 2007) = 0. 6 – 0. 2 = + 0. 4 2, 5 2 P 08 -P 07=0. 6 1, 5 participants non-participants 1 NP 08 -NP 07=0. 2 0, 5 0 2007 2008 18
Assumption of same trend: Graphic Implication 2, 5 2 Impact = +0. 4 1, 5 participants non-participants 1 0, 5 0 2007 2008
Conclusion �Positive Impact: More intuitive �Is the assumption of same trend reasonable? ➤ Still need to question the counterfactual assumption of same trends ! ➤Use data from previous years
Questioning the Assumption of same trend: Use pre-pr 0 gram data 2, 5 2 1, 5 participants non-participants 1 0, 5 0 2006 2007 2008 >>Seems reasonable to accept counterfactual assumption of same trend ? !
Caveats (1) �Assuming same trend is often problematic No data to test the assumption Even if trends are similar in the past… ▪ Where they always similar (or are we lucky)? ▪ More importantly, will they always be similar? ▪ Example: Other project intervenes in our nonparticipant villages…
Caveats (2) �What to do? >> Be descriptive! Check similarity in observable characteristics ▪ If not similar along observables, chances are trends will differ in unpredictable ways >> Still, we cannot check what we cannot see… And unobservable characteristics might matter more than observable (ability, motivation, etc)
Matching Method + Differencein-Differences (1) Match participants with non-participants on the basis of observable characteristics Counterfactual: � Matched comparison group �Each program participant is paired with one or more similar non-participant(s) based on observable characteristics >> On average, participants and nonparticipants share the same observable characteristics (by construction) �Estimate the effect of our intervention by using difference -in-differences 24
Matching Method (2) Underlying counterfactual assumptions � After matching, there are no differences between participants and nonparticipants in terms of unobservable characteristics AND/OR � Unobservable characteristics do not affect the assignment to the treatment, nor the outcomes of interest
How do we do it? �Design a control group by establishing close matches in terms of observable characteristics Carefully select variables along which to match participants to their control group So that we only retain ▪ Treatment Group: Participants that could find a match ▪ Control Group: Non-participants similar enough to the participants >> We trim out a portion of our treatment group!
Implications �In most cases, we cannot match everyone Need to understand who is left out �Example Matched Individuals Portion of treatment group trimmed out Nonparticipants Participants Score Wealth
Conclusion (1) �Advantage of the matching method Does not require randomization 28
Conclusion (2) �Disadvantages: Underlying counterfactual assumption is not plausible in all contexts, hard to test ▪ Use common sense, be descriptive Requires very high quality data: ▪ Need to control for all factors that influence program placement/outcome of choice Requires significantly large sample size to generate comparison group Cannot always match everyone… 29
Summary �Randomized-Controlled-Trials require minimal assumptions and procure intuitive estimates (sample means!) �Non-experimental methods require assumptions that must be carefully tested �More data-intensive �Not always testable �Get creative: Mix-and-match types of methods! 30
Thank You 31
- Non experimental design vs experimental
- Experimental vs non experimental
- Quasi experimental
- Quasi-experimental research designs
- Cohort meaning in research
- Quasi experimental
- Ethical issues of experimental research
- Quasi-experimental research designs
- Experimental research design
- Quasi experimental design example
- Cross-sectional correlational design
- Postnodd
- Disadvantage of quasi experimental design
- Descriptive studies
- Quasi experimental design slideshare
- Bhayu rhama
- Causal-comparative research meaning
- Quasi experimental research
- Experimental vs non experimental
- Disadvantages of experimental research
- Quasi esperimenti
- Quantum meruit
- Quasi sprzeciw kpk
- Mc 1
- Quasi linear utility function
- Supply-side market failures occur when:
- I quasi incidenti sono facili da individuare
- Disegno quasi sperimentale
- Lijst met gevoelens
- Quasi complementary push pull amplifier
- Quasi contract meaning
- Quasi rent formula