Social Exclusion Task Force What works Guidance on

Social Exclusion Task Force ‘What works? ’: Guidance on evidence informed commissioning April 7 th 2008

Social Exclusion Task Force n Based in the Cabinet Office n Ministers n Successor to the Social Exclusion Unit n Working across government departments n Champion the voice of vulnerable groups n Evidence-based policy n Mission of the SETF: To extend the opportunities enjoyed by the vast majority of people in the UK today to those whose lives have been characterised by deprivation and exclusion

Five principles for tackling social exclusion 1. Better identification and early intervention 2. Identifying what works 3. Multi-agency working 4. Personalisation, rights and responsibilities 5. Supporting achievement and managing underperformance

Current work programme n Adults n n n Children and families n n New Public Service Agreement on socially excluded adults Adults facing Chronic Exclusion (ACE) pilots Families at Risk Review and Pathfinders Family Nurse Partnership (FNP) pilots Multi-Systemic Therapy (MST) pilots Research n n New Analysis of social exclusion over the life course Guidance on ‘what works’ – evidence-informed commissioning

‘What works? ’: Guidance on evidence informed commissioning – A tool for commissioners and providers of services for excluded groups

What is the guidance on ‘what works? ’ § A tool which sets out guiding principles in the use of evidence to inform and monitor services/programmes for excluded groups, by drawing upon evidence to: § Help assess whether a programme is likely to be promising; § Set objectives to evaluate a programme or service; § Review progress along the way. § Designed to support and sit alongside current commissioning frameworks. § Target audience – local authority commissioners and service providers focusing on delivery for vulnerable groups.

Aim and value of the guidance § Strengthen the capacity of commissioners and providers of services to evaluate ‘what works’ by looking at the evidence. § Improve the outcomes for vulnerable individuals by ensuring services designed to support them are based on research evidence. § Build up an evidence base on what works locally and nationally. § Ensure a desired direction of travel e. g. monitoring. § User-friendly document. § Socially Excluded Adults PSA and FARR.

What the guidance will cover Section Key questions 1. Introduction What is the guidance and who is they for? 2. Using research evidence in service planning What are the benefits of using research? How do we use research-evidence in service planning? How do we ask a research question? 3. Locating relevant research evidence How can we find the evidence we need? Reviews primary and secondary research; What databases can we use? 4. Assessing and appraising research evidence How can we know whether evidence is reliable? What is qualitative and quantitative research? How do we know what is ‘good’ research? Includes a grading tool and checklist on appraising research 5. Outcome focused evaluation How do we know whether we are achieving service outcomes? How do we plan for an evaluation? Reviews SMART outcomes – Specific, Measurable, Achievable and Realistic and Time-limited; 6. Appendices – web based What does the jargon mean? Checklist for appraising research evidence How can we ensure our work is ethical What do the numbers mean Case studies

G R A D E MINIMUM SOURCES OF EVIDENCE TYPE OF EVIDENCE DESCRIPTION STRENGTHS WEAKNESSES COMMENTS IMPACT ON SERVICE USERS 1 Positive reports from service users and practitioners at follow up Personal testimony Practice experience Account by a person intimately acquainted with or affected by a particular issue Accumulated wisdom from repeated exposure to similar situations or problems Powerful and immediate; may give vivid insights into events concealed from much of the population Enables practitioners to identify anomalous situations where counterintuitive decisions can and should be made Difficult to verify; may lead to inflation of prevalence; accounts of distressing personal trauma may inhibit critical appraisal Errors can be consistently repeated unless tested by other sources of evidence; hard for lay people to challenge Sufficient basis for pilot intervention where plausible link between activities and outcomes exists Unknown effectiveness 2 Several positive prepost studies comparing performance at baseline to follow up Client opinion study Single case design Narrative or survey accounts of user views or reported needs Repeated standardised measurement of a client’s situation or problems over time Valuable insights from those at the receiving end; compels professionals to stay focused on the client’s priorities Easy and practical; can be used by practitioners with minimal training; clients are able to collaborate and contribute Correlation between satisfaction and actual outcomes is low; needs described may not translate into actual service use More difficult to apply with nonbehavioural interventions; absence of controls weakens attribution of cause and effect Sufficient basis for pilot intervention at multiple sites where plausible link between activities and outcomes exists Limited evidence of effectiveness 3 Positive evaluations by several studies featuring comparison groups Quasi-experimental study Cross sectional study Cohort study No random allocation to groups; use of natural populations. Powerful method of exploring the impact of an intervention when randomisation is impossible; can also be applied to communities Survey can be repeated at intervals, illustrating changing trends over time Best source of evidence on association between childhood experience and adult outcomes. Difficulty in ensuring equivalence of groups and natural changes in group composition over time can result in less reliable finding Hard to detect why changes have occurred; difficult to observe trends in minority populations Data often emerge too late for effective policy making; may be drop out over time. Basis for investment where need is urgent and more robust evaluation results are not immediately available May be beneficial 4 Positive evaluations by several randomised controlled trials Randomisedcontrolled trial (RCT) One group receives an intervention, another receives none or an intervention of another type; the chance of being allocated to either group is identical Changes in the client’s situation can be attributed to the intervention with a high degree of confidence; best approach for controlling bias; transparent methodology means results can be independently checked and replicated Mistakes in randomisation can invalidate results; cannot be applied to many important social problems; without equipoise – an equal chance that benefits will occur in both groups – ethical difficulties may arise; high level of statistical skill may be required for analysis. Highly desirable for large investment in single service site Likely to be beneficial 5 Intervention positively evaluated by at least one systematic review or meta-analysis Systematic review Meta-analysis Aggregation of results from eligible studies; eligibility criteria defined in advance (usually but not necessarily RCTs); review methodology is replicable Best source of reassurance that an intervention works (or doesn’t); metaanalysis pools numerical results; large reviews carry great statistical power; pooling of results gives more accurate indication of actual effect size by moderating outliers Requires a substantial number of robust primary studies in a given area; methodology less well developed for synthesising qualitative data; strict inclusion criteria may results in few or no eligible studies; reviews are technically demanding Highly desirable for large investment in multiple service sites Highly likely to be beneficial

Key players n Barnardo’s n Research in Practice n National Foundation for Educational Research n SETF n Local authorities n Third sector n Academics n Government departments n Dissemination and publication in Spring 2008 5

Workshops – key findings Method/Consultation n 4 workshops; variety of LAs across England; linked into LARC. Findings n Commissioning appears to be very varied across LAs - despite numerous frameworks; n There is a need for a culture change in the use of research evidence; n There needs to be greater innovativeness in the selection of programmes; n There needs to be greater transparency in the selection of programmes; n Guidance needs to be accessible, user-friendly, not burdensome! n Guidance should be a tool which helps not only in commissioning but decisions to decommission.

Questions for discussion For commissioners/ service providers n What are your initial thoughts on the proposed contents of the guidance? n n Would this be helpful for you – how would you use it? Anything missing / to be added? Does this resonate with any guidance or support materials you already use? For researchers, policy makers and academics n What are your initial thoughts on the proposed contents of the guidance? n n Is there anything missing which needs to be considered? How can we bridge the gap between ‘what works’ and practice?

Next Steps n Peer Review n n n Academics Local authority commissioners Third sector Government departments Specialist organisations n Commissioning courses n Home? n Publication (Spring 08) n www. cabinetoffice. gov. uk/social_exclusion_task_force n setaskforce@cabinet-office. x. gsi. gov. uk