Randomized Control Trials Difficulties to Consider Costs l

  • Slides: 10
Download presentation
Randomized Control Trials Difficulties to Consider

Randomized Control Trials Difficulties to Consider

Costs l Cost of intervention itself often not difficult to justify l l l

Costs l Cost of intervention itself often not difficult to justify l l l Providing goods/services Only including promising possibilities New data are expensive l l Quality of evaluation dependent on quality of data More money spent on data is less money spent on providing the intervention to more people

Creating the Control Group l Is it politically feasible to deny treatment to some

Creating the Control Group l Is it politically feasible to deny treatment to some people? l l l How important is it to measure how well the intervention works? Issue of trade-offs Ethics less contested if: l l Budget constraints would have prevented everyone from receiving the intervention anyway Everyone eventually receives the intervention and the control group is only denied it initially (phasedin rollout)

Does Everyone Benefit? l Necessary to deny control group intervention l But don’t want

Does Everyone Benefit? l Necessary to deny control group intervention l But don’t want to actively hurt them l l Can’t deceive Can’t make them worse off than they’d otherwise be Some sort of small gift/compensation typical – careful not to make this into a second treatment Must honor promises (phased-in rollout)

Internal Validity l Was the control group valid? l l l Was the intervention

Internal Validity l Was the control group valid? l l l Was the intervention consistent in all treatment areas? l l Randomization worked l Intended treatment and control groups balanced l Actual treatment and control groups same as intended Contamination from spillovers Easier to guarantee in some cases than in others Do data exist to rule out alternate hypotheses?

External Validity l Will the subjects in the experiment be representative of the entire

External Validity l Will the subjects in the experiment be representative of the entire population who will eventually receive the intervention? l l Logistically, much easier to do data collection in restricted area Less likely that experiment will generalize to entire country

Data Quality l Sensitive questions l l l Subjective questions Self-reported vs quantitative measures

Data Quality l Sensitive questions l l l Subjective questions Self-reported vs quantitative measures l l How can we encourage subjects to give honest and complete answers? “recall error” Hawthorne effect - People behave differently when they know they’re being watched l l Might be desirable to follow them closely for more data But that might make biases worse

Cost-Effectiveness Comparisons l Resources are scarce – need to pick most effective programs l

Cost-Effectiveness Comparisons l Resources are scarce – need to pick most effective programs l Need to be able to convert impacts from various projects into one set of units l How to compare improvement in nutrition to reduction in malaria?

Scaling Up l Can intervention be implemented identically at scale? l l If not,

Scaling Up l Can intervention be implemented identically at scale? l l If not, is RCT still informative? Will the economy at large respond to the intervention at scale? (“general equilibrium effects”) l l l Prices might go down – economies of scale Prices might go up – insufficient supply Spillover effects could set in

Final Thoughts l RCT is gold-standard in terms of identifying causality l But many

Final Thoughts l RCT is gold-standard in terms of identifying causality l But many complications arise during implementation l Need to weigh theoretical advantages against practicalities – is it really the best method?