Government Social Research Unit Randomised Controlled Trials Conference

Government Social Research Unit Randomised Controlled Trials Conference University of York, 14 -15 September 2006 Why Governments Need RCTs Philip Davies Ph. D Government Social Research Unit HM Treasury London SW 1 A 2 HQ www. gsr. gov. uk 1

Answer Because RCTs can help get policy initiatives right more often than wrong www. gsr. gov. uk 2

Caveats • Governments have got to want to get things right • ‘Getting things right’ is more than achieving an effect size • Governments do things other than achieving impacts • RCTs are not a panacea • RCTs are only as good as their design and implementation www. gsr. gov. uk 3

Governments Have Got to Want to Get Things Right “Social science should be at the heart of policy making. We need a revolution in the relationship between government and the social research community - we need social scientists to help determine what works and why, and what type of policy initiatives are likely to be most effective …” (David Blunkett, 2000) www. gsr. gov. uk 4

Governments Have Got to Want to Get Things Right Government policy must be: • Evidence-based • Properly evaluated • Based on best practice www. gsr. gov. uk 5

Governments Have Got to Want to Get Things Right • Policy making must be soundly based on evidence of what works • Improve departments’ capacity to make best use of evidence • Improve the accessibility of the evidence available to policy makers 6

Governments Have Got to Want to Get Things Right Professional Skills for Government (2005) Analysis and Use of Evidence • • • Anticipate and secure appropriate evidence Test for deliverability of policy/practice – and evaluate Use evidence to challenge decision making Identify ways to improve policy/practice Champion a variety of tools for collecting/using evidence Ensure use of evidence is consistent with wider government requirements • Work in partnership with a wide range of experts/analysts 7

What ‘Getting It Right’ Means The ‘Experimenting Society’ (Donald T. Campbell) “…a society that would use social science methods and evaluation techniques to vigorously try out possible solutions to recurrent problems and would make hardheaded, multidimensional evaluations of outcomes, and when the evaluation of one reform showed it to have been ineffective or harmful, would move on and try other alternatives” 8

What ‘Getting It Right’ Means “A social experiment benefits society by providing better information on which to base public policy. Such information can improve policy in one of two ways: it can lead policymakers to adopt a program or policy that is found to have net social benefits, or it can lead to the termination of an existing program that is found to have a net social cost” (Orr, L. , 1999). 9

Why RCTs? (Recap) Randomised Controlled Trials: • Provide most precise estimates of the likely effects of policy interventions. • Against a strong counterfactual – i. e. the net effect/impact • Reduce/account for various biases. • Establish cause and effect of policy interventions and outcomes. • Raise the quality of evidence to support policy making. 10

So, Why Don’t Governments Use RCTS More Often? • RCTs not on most policy makers’ radar • RCTs not on many government researchers’ radar • RCTS not on many academic researchers’ radar • For some (many? ) RCTs are anathema: Ø Don’t like ‘random’ (too ‘chancey’) Ø Don’t like ‘control’ (no ‘wiggle room’) Ø Don’t like ‘trials’ (sounds inhumane / 1940 s Germany) • There are things other than ‘impact’ • There are things other than ‘evidence’ 11

Things Other Than Impact Evidence Experimental Quasi-Experimental Social Ethics Public Consultation Ethical Evidence Experimental Qualitative Theories of Change Impact Evidence Implementation Evidence Research Evidence Cost-Benefit Cost-Effectiveness Cost-Utility Economic and Econometric Evidence Statistical Modelling Surveys Admin Data Comparative Qualitative Descriptive Analytical Evidence Multivariate Analysis 12

Things Other Than Evidence “Research of any kind is seldom the determining factor in shaping public policy. Experimentation is no exception to this rule” (Orr, L. , 1999) 13

Things Other Than Evidence Pragmatics & Contingencies Lobbyists & Pressure Groups Habits & Tradition Experience & Expertise Judgement Evidence Resources Values and Policy Context 14

Different Notions of Evidence Policy Makers’ Evidence Researchers’ Evidence • Colloquial (Contextual) • ‘Scientific’ (Context free) • Anything that seems reasonable • Proven empirically • Policy relevant • Theoretically driven • Timely • As long as it takes • Clear Message • Caveats and qualifications 15

Where Do Policy Makers Go For Evidence? 16

But Not All Is Lost Examples of Recent UK Government RCTs • Employment Demonstration Project (ERA) • Job Retraining and Rehabilitation Project (JRRP) • Restart evaluation (1990) • BA Visiting Officer Pilot • Evaluation of Employment Zones • New Deal 25+ Pilots • Evaluation of the New Deal for Young People Intensive Gateway • Evaluation of the Restorative Justice programme 17

RCTs Are Only As Good As Their Design and Implementation Common Design and Implementation Faults: • • • Unclear policy objectives, populations, outcomes Insufficient time (lack of strategic policy making) Test condition often pre-determined Sample sizes too small – no statistical power Problems of randomisation, concealment, cross-over High attrition (especially in control groups) Intention to Treat analysis not used Limited capacity (in the UK) to do high quality RCTs 18 Poor counterfactual too often the result

RCTs Are Only As Good As Their Design and Implementation These faults can usually be overcome! • • • Better strategic policy making Better planning and protocol development Involvement of researchers, policy makers, front-line staff High quality piloting / pre-fieldwork testing High quality research/project management Careful real time data management Use of experienced technical experts throughout Policy Trials Service Professional development of researchers and policy makers 19

Take Home Message • RCTs have enormous potential for governments • RCTs can help governments get things right more often • RCTs are currently used considerably to UK Govnt. • But there are still non-believers • And those who remain to be convinced • RCTs, like other types of research, must compete with other notions of evidence • And with factors other than evidence • There is a need to improve the capacity to do high quality RCTs in the UK 20

Contact philip. davies@hm-treasury. x. gsi. gov. uk Government Social Research Unit HM Treasury 1 Horse Guards Road London SW 1 A HQ England Tel: +44 (0)20 7270 5156 www. gsr. gov. uk 21