REALIST EVALUATION IN MORAY SCOTLAND Using Realist Evaluation

  • Slides: 13
Download presentation
REALIST EVALUATION IN MORAY, SCOTLAND Using Realist Evaluation in practice Jeremy Akehurst Performance and

REALIST EVALUATION IN MORAY, SCOTLAND Using Realist Evaluation in practice Jeremy Akehurst Performance and Strategy Manager

A POTTED HISTORY � We started using Realist Evaluation in Moray to evaluate discrete

A POTTED HISTORY � We started using Realist Evaluation in Moray to evaluate discrete services with a narrow focus – e. g. Youth Justice, Housing Support � 2005 -6 we piloted using the method in the children and families social work service � We now use this method for a wide range of children’s social work services, including Safeguarding

REALIST EVALUATION � Aims to show whether targeted outcomes are showing improvement for service

REALIST EVALUATION � Aims to show whether targeted outcomes are showing improvement for service users � Provides a voice for the service user � Meets the requirement for evidencebased practice � Enables outcome-based service evaluation

REALIST EVALUATION How does it work? By using reliable measurement tools By measuring the

REALIST EVALUATION How does it work? By using reliable measurement tools By measuring the change over time By linking context and intervention with improvement

REALIST EVALUATION Realist can help by giving data on two levels � Single case

REALIST EVALUATION Realist can help by giving data on two levels � Single case analysis – by showing progress for a individual service user � Service level analysis - when data is aggregated, a higher level of analysis can be undertaken.

REALIST EVALUATION Measurement tools � Self assessment � Practitioner � Parent assessment / Carer

REALIST EVALUATION Measurement tools � Self assessment � Practitioner � Parent assessment / Carer assessment

Apr-08 Mar-09 Feb-09 Jan-09 Dec-08 Total Nov-08 Oct-08 Sep-08 Aug-08 My Wider World Jul-08

Apr-08 Mar-09 Feb-09 Jan-09 Dec-08 Total Nov-08 Oct-08 Sep-08 Aug-08 My Wider World Jul-08 Jun-08 May-08 Date of completed measures Mar-08 Feb-08 Jan-08 What I Need from Others Dec-07 Nov-07 Oct-07 Sep-07 Aug-07 Jul-07 Jun-07 May-07 Apr-07 How I Grow & Develop Mar-07 Feb-07 Jan-07 Dec-06 Nov-06 Oct-06 Sep-06 Aug-06 Jul-06 Jun-06 May-06 Score (Higher is Better) REALIST EVALUATION RL 13 - Client Holistic Measure Linear(Total) 90 80 70 60 50 40 30 20 10 0

REALIST EVALUATION Service User / worker benefits: � Service user is able to see

REALIST EVALUATION Service User / worker benefits: � Service user is able to see whether or not they are making progress � Social worker is able to judge the efficacy of interventions � In all cases, it can be determined whether or not improvement has been made � Encourages an open working relationship between workers and service user � Service users feel empowered through being part of the approach � Can be integrated with the practice cycle – assess, plan, act, review

REALIST EVALUATION Service benefits: � Conclusions can be drawn about strengths and weaknesses in

REALIST EVALUATION Service benefits: � Conclusions can be drawn about strengths and weaknesses in services � It supports learning about what makes a difference � Resources can be targeted appropriately � It supports a better understanding of the range of difficulties being faced by service users – and therefore by the service itself

REALIST EVALUATION The Child Holistic Measure: � Self-Report Completed by Young People � At

REALIST EVALUATION The Child Holistic Measure: � Self-Report Completed by Young People � At baseline 84 youth completed all 21 items � Internal consistency reliability alpha 0. 83

REALIST EVALUATION Child Holistic Measure Aggregate Score improved or not Valid not improved Missing

REALIST EVALUATION Child Holistic Measure Aggregate Score improved or not Valid not improved Missing Total Frequency 27 Percent 9. 1 Valid Percent 39. 1 improved 42 14. 1 60. 9 Total 69 23. 2 100. 0 228 76. 8 297 100. 0 System The chart above shows the percentage of cases (60. 9%) who have shown improvement measured by the Child Holistic Measure.

REALIST EVALUATION Analysis of the database came up with the following predictors: � �

REALIST EVALUATION Analysis of the database came up with the following predictors: � � � Families allocated a family support worker showed four times greater level of improvement than those who did not. From the age of 14 down, for every year younger, a young person was 1. 2 times more likely to improve then another young person who was older Those who had a case objective linked to better use of local resources were almost 7 times more likely not to improve than those who did not have this objective

SOME PRACTICE ISSUES � � There are practitioners who have embraced the use of

SOME PRACTICE ISSUES � � There are practitioners who have embraced the use of reliable measures – enthusiastic users They have found both measures and outcome charts support engagement on the part of parents and children and encourage positive change Many practitioners complain that they don’t have enough time to do this Some appear to have more difficulty in engaging service users in completing outcome measures