Randomized Control Trials Difficulties to Consider Costs l
- Slides: 10
Randomized Control Trials Difficulties to Consider
Costs l Cost of intervention itself often not difficult to justify l l l Providing goods/services Only including promising possibilities New data are expensive l l Quality of evaluation dependent on quality of data More money spent on data is less money spent on providing the intervention to more people
Creating the Control Group l Is it politically feasible to deny treatment to some people? l l l How important is it to measure how well the intervention works? Issue of trade-offs Ethics less contested if: l l Budget constraints would have prevented everyone from receiving the intervention anyway Everyone eventually receives the intervention and the control group is only denied it initially (phasedin rollout)
Does Everyone Benefit? l Necessary to deny control group intervention l But don’t want to actively hurt them l l Can’t deceive Can’t make them worse off than they’d otherwise be Some sort of small gift/compensation typical – careful not to make this into a second treatment Must honor promises (phased-in rollout)
Internal Validity l Was the control group valid? l l l Was the intervention consistent in all treatment areas? l l Randomization worked l Intended treatment and control groups balanced l Actual treatment and control groups same as intended Contamination from spillovers Easier to guarantee in some cases than in others Do data exist to rule out alternate hypotheses?
External Validity l Will the subjects in the experiment be representative of the entire population who will eventually receive the intervention? l l Logistically, much easier to do data collection in restricted area Less likely that experiment will generalize to entire country
Data Quality l Sensitive questions l l l Subjective questions Self-reported vs quantitative measures l l How can we encourage subjects to give honest and complete answers? “recall error” Hawthorne effect - People behave differently when they know they’re being watched l l Might be desirable to follow them closely for more data But that might make biases worse
Cost-Effectiveness Comparisons l Resources are scarce – need to pick most effective programs l Need to be able to convert impacts from various projects into one set of units l How to compare improvement in nutrition to reduction in malaria?
Scaling Up l Can intervention be implemented identically at scale? l l If not, is RCT still informative? Will the economy at large respond to the intervention at scale? (“general equilibrium effects”) l l l Prices might go down – economies of scale Prices might go up – insufficient supply Spillover effects could set in
Final Thoughts l RCT is gold-standard in terms of identifying causality l But many complications arise during implementation l Need to weigh theoretical advantages against practicalities – is it really the best method?