OUTCOMES OF CARE AN OVERVIEW Dr Mark Kerr

OUTCOMES OF CARE: AN OVERVIEW Dr Mark Kerr Centre for Outcomes of Care mark@outcomesofcare. com

Presentation Overview 1. Outcomes and the relevance to you 2. Evidence on outcomes 3. Measuring outcomes 4. Assessment of need

1. Outcomes

A priest and a taxi driver both died and went to heaven. St. Peter was at the Pearly Gates waiting for them. ‘Come with me, ’ said St. Peter to the taxi driver. The taxi driver did as he was told and followed St. Peter to a mansion. It had anything you could imagine from a bowling alley to an Olympic size pool. ‘Wow, thank you, ’ said the taxi driver. Next, St. Peter led the priest to a rugged old shack with a bunk bed and a little old television set. ‘Wait, I think you are a little mixed up, ’ said the priest. ‘Shouldn’t I be the one who gets the mansion? After all I was a priest, went to church every day, and preached God’s word. ’ ‘Yes, that’s true. But during your sermons people slept. When the taxi driver drove, everyone prayed. ’ Lesson: only results count. (Hatry, 1999)

Defining Outcomes • ‘Outcomes are about impact – the effect of care support’ (Netten, 2011) • ‘Outcomes are about the impact or end results of services on a person’s life. Outcome-focused services and support therefore aim to achieve the aspirations, goals and priorities identified by service users (and carers) – in contrast to services whose content and/or form of delivery are standardised or determined solely by those who deliver them’ (Glendinning et al. , 2006) • To evaluate how well the service performs in improving outcomes, pre and post intervention measures are required to identify ‘distance travelled’

Outcomes in the context of LACYP • Outcomes for LACYP are of importance to multiple stakeholders. • • Tensions must be resolved to ensure that outcomes chosen to be focused on are in line with the needs of the child Children and young people in care reliant on the adults around them to ensure outcomes are appropriate and relevant to their needs Outcomes implemented in children’s services need to be more evidence based. A need for better measures for emotional and mental wellbeing assessment to be routinely used as part of a full assessment for LACYP coming in to care Often LACYP have multiple service inputs making outcome assessment challenging Care IS the intervention and the most significant we make in the lives of children and families. We know very little about the efficacy of children’s services in England.

Outcomes as part of a system Evidence Outcomes Data Theory Practice

2. EVIDENCE EXAMPLES

Education, education

Education Difference LAC v Non

Progress 8 Scores (relative measure)

Higgins et al. , 2015 Systematic review concluded that there is a correlation between being in care and educational outcomes, but that this relationship is mediated by a number of individual, family and environmental risk factors. Although the evidence is mixed, there was little support for the claim that being in foster or kinship care per se is detrimental to the educational outcomes of children in care.

Attainment by time in care Source: Df. E, 2014

Absence rates

NEET rates for Care Leavers v Early outcomes for education and work non • At 19 years of age 41% of care leavers NEET – highest percentage for 13 years • General population 15% • Counting rules for statistical collections may be masking much higher number • Some categories to be classed as EET questionable

Why are EET outcomes not improving?

Psychiatric disorder British children looked after by local authorities: Comparison with children living in private households (Ford et al. , 2007) • Comparison between Looked After Children and children living at home from deprived and non-deprived backgrounds • Data combined from three surveys of Looked After Children (England, Scotland, Wales) • ONS sample of children living at home (Meltzer et al. , 2000) from the child benefit register • Age 5 - 17 Measures (used in all studies): Development and Well Being Assessment (DAWBA - Goodman et al. , 2000) • Teachers assessment • Social workers and carers assessment

Psychiatric disorder among British children looked after by local authorities: Comparison with children living in private households (Ford et al. , 2007)

Looking at care leavers - mental health Buchanan (1999) • At age 16 o Less satisfied with their lives o 25% had significant levels of maladjustment • At age 33 o Higher rates of depression than other parental groups Saunders and Broad (1997) • 16. 6% had a mental disorder • 35% had engaged in self harm since the age of 15 • 60% had felt suicidal • 31% had self referred to mental health services (of whom 77% did not find services useful)

Care Leavers MH Cont’ Dixon et al (2006) • Found in the months following leaving care CL mental health problems (self report) doubled from 12% to 24% • Using GHQ-12 poor mental well-being 22% at baseline / 25% at follow up • High incidences of stress, depression • Several suicide attempts

LAC Scholastic ability (teacher rating) Functioning Age / Actual Age 5 – 10 year old 11 – 15 year old All children 4+ years behind 7 21 14 3 years behind 7 13 10 2 years behind 16 16 16 1 year behind 27 17 22 Equivalent 29 23 26 1+ years ahead 14 11 12 Metlzer et al, 2003 Within the adolescent LAC population 50% are at least 2 years behind their peers

Progress 8 Scores (relative measure)

Educational attainment over life-course • survey of 310 care leavers • results were significantly more positive than expected: • 55 per cent had attained GCSEs, A Levels or equivalent; • 26 per cent had graduated with a degree; • six per cent had a Master’s degree and • two per cent a Ph. D. • as the age of care leavers increased, so did the level of educational attainment (Care Leavers Association (CLA) ‘Listen Up’ (CLA / Duncalf, 2010)

3. Measurement

Hard and soft indicators Quantitative indicators are measures of units or amounts Qualitative indicators are people’s judgments or perceptions about a change Some are hybrid e. g. SDQ Indicators must be: Measurable Reliable Feasible Have utility in decision making

Measures / Scales / Psychometrics • Normally self completed with or without assistance and in the case of • • children often completed by third party who knows the child Can be used for screening or to assess a specific issue e. g. self esteem, sense of belonging Variable in their validity and critical that services for LAC only use validated measures that have been subject to independent peer review Must be mindful of suitability as a ‘repeated measure’ to assess distance travelled SDQ is NOT a measure of change Example measures: • Parenting Stress Index (PSI) • Child Behavior Checklist (CBCL) • Child and Adolescent Needs and Strengths (CANS) • Child Development Inventory (CDI) • Warwick Edinburgh Mental Wellbeing Scale (WEMWBS)

Event frequency • As well as the need to incorporate measures that capture changes in development, emotional wellbeing and mental health, there is also a need to monitor the frequency of critical events. Examples could include: • Self harm • Absconding • Drug / alcohol consumption • Loss of regulatory control • But also positive events: • Attended school • Received praise • Engaged in therapy

Methodological considerations Whilst there is a need to better understand what works when, where and for which children, often this learning is hindered by insufficient methodological rigor. • When a measure becomes a target it ceases to be a good measure (Goodhart’s Law) • Measuring a system usually disturbs it. The more precise the measurement, the shorter its timescale, the greater the unpredictability of the outcome (Heisenberg’s uncertainty principle) • A need for random samples, controls or matched pairs • The attribution v contribution challenge

Attribution vs. Contribution Evaluations need to determine whether (and to what extent) the intervention is making a difference • Attribution: Determining the degree to which the intervention is ‘causing’ the observed outcomes; • Contribution: Determining the degree to which the intervention is ‘contributing to or helping to achieve’ the observed outcomes (and impacts); • In practice this distinction not critical – projects will ask: • Are outcomes & well-being (as measured by indicators) changing? • To what extent is the intervention responsible for these measured changes? • To what extent can we say intervention contributed to or caused these changes? • If evidence suggests changes would not have happened without the intervention, we can attribute changes to it; if evidence suggests the intervention played a role but was not sole driver, we can say it contributed to changes • In practice there will be a continuum from no contribution to full attribution

4. The need for data improvement

Problem • Outcomes rely on baseline data / needs assessments Outcomes Interventions Needs

Assessing need Needs assessment is the most critical factor for children’s social care, done well it: • Provides key evidence to inform the decision whether to receive a child in to care; • Informs appropriate service responses; • Facilitates fairer co-commissioning of placements based on need and potential future risks and resource demands • Provides baseline data required to evidence true outcomes from services – ‘distance travelled’ • When aggregated contributes to sufficiency and forward planning However: Recent research highlights the poor accuracy of much decisionmaking in the child protection field, with assessments being ‘only slightly better than guessing’ (Dorsey et al 2008)

Problem continued • Absence of longitudinal / time series data a longstanding problem preventing us understanding ‘what works, when and for which children’ • We need to quantify need across the LAC population to inform sufficiency and ensure the sector evolves in a needs led way • To be able to better understand the efficacy of ‘innovation’ • The introduction of results based financial structures require reliable data to assess outcomes

Questions?

References • Dorsey, S. , Mustillo, S. A. , Farmer, E. M. Z. & Elbogen, E. (2008) Caseworker assessments of risk for recurrent maltreatment: association with case- specific risk factors and re -reports. Child Abuse & Neglect 32 (3): 377 - 91. • Glendinning C. , Clarke S. , Hare P. , Maddison J. & Newbronner L. (2006) Outcomes Focused Services for Older People. Adult Services Knowledge Review 13, Social Care Institute for Excellence, London. • Hatry, H. , (1999) Performance measurement: getting results. The Urban Institute Press. Washington D. C. • Netten, A (2011) Overview of outcome measurement for adults using social care services and support. Methods review, 6. NIHR School for Social Care Research, London