Anna Freud National Centre for Children and Families

Anna Freud National Centre for Children and Families 22/08/2018 Shared Learning Group for Assistants Accountability

Anna Freud National Centre for Children and Families Accountability What is Accountability? • Accountability is one of the five core principles of the CYP IAPT programme. Collecting data about outcomes is an important way to demonstrate the effectiveness of services and interventions, and can flag up areas for improvement. • Improves clinicians’ ability to detect worsening of symptoms (Lambert, 2010) • Reduces drop out (e. g. Miller et al. , 2006) • Achieves better outcomes, faster (Bickman et al. , 2011 & Lambert et al. , 2005) • helps to keep off-track cases (i. e. , patients not progressing well) on-track (Lambert et al. , 2003) • Helps with quality assurance • Helps with research 2

Anna Freud National Centre for Children and Families Accountability Routine Outcome Measures • Routine Outcome Monitoring describes the use of questionnaires and feedback tools to collect information about how people are feeling, treatment goals and experience of using services. The collaborative supports use of outcome monitoring through training and implementation groups. • The CYP IAPT programme embeds Routine Outcome Monitoring (ROM) into the day to day work in services in order to aid meaningful conversations and allow for greater collaboration with CYP, help with problem identification, enable both CYP and therapists to track progression and also see when things aren’t working. Use of ROMs have been shown to improve outcomes (Bickman et al 2011), reduce drop out (e. g. Miller et al. 2006) and reduce the length of interventions (Lambert et al. 2005). 3

Anna Freud National Centre for Children and Families Activity • What ROMs are used in your service? • What are the benefits of using these measures? • What are the challenges of using these measures? 4

Anna Freud National Centre for Children and Families Young Person’s Perspective ROMs can seem daunting but their purpose should be explained • Why are they beneficial to the process? • How they help both you and the YP monitor their progress 5

Some ways young people say ROMs can be unhelpful • It’s about HOW they’re used! They’re just another piece of paper if: • There is no clarity about why they are being used or how they can help • There are different expectations about what they are for • There isn’t a meaningful conversation about using them (unless this is pre agreed) • The tool doesn't feel the right fit for the young person • There is no choice which ones to use or how to use them “I find it hard to say how I feel… it takes me a while. I hate the scale 19 thing. I like the ones with words. ” “It’s like ticking boxes rather than, like, help. Like their criteria… numbers… like their number thing, it’s so rigid. ” 6

Activity Discuss the scenarios assigned to your group: A young person comes in with a goal to never feel anxious again. How do you deal with a YP expecting a goal unreachable within the 8 weeks you have with them? A YP is finding ROMs really boring and decides that they don’t want to do them anymore. How would you work around this situation/ what would you do? A YP comes to their first session not knowing what a CWP is. They have come with the idea that they are going to have a full assessment for CAMHS. How do you clarify the situation? A YP comes in, and once they have looked over their notes and read the YP has come in with mild depression, the clinician sets a reasonable goal to work towards within the 8 weeks. Is there anything unhelpful about the way the clinician has approached this session? 7

Key messages o It’s the spirit in which outcome measures are used that is most important o If it’s for service evaluation only – that’s fine. Just be honest about it o Young people don’t mind doing them if they’re relevant, meaningful and well-explained “This is to help us learn as a service. We won’t look at it together, but it could help the service to learn what we’re good at, what we need to get better at. ” 8

Anna Freud National Centre for Children and Families Break Question to think about: No Assistants- what would happen outcomes in your service? 9

Anna Freud National Centre for Children and Families Accountability How do assistants contribute to the implementation of outcomes? Case Example: Reading University • Thorough and in-depth understanding of the data being collected. • Background reading of the CORC website, visiting other sites and other professionals and learning best practice from their experience of data collection. • Ensure the data we collect is entered correctly and if we were unsure we spoke to WPs directly to clarify the information. • Created an MOU which was signed by all services prior to collecting data to ensure the purpose and the expectation of our role was clearly laid out. • Issues: WPs were unaware we would be looking at their data. Cause some anxiety that we were assessing the WPs themselves. Could overcome this by informing WPS that the data would be looked at the beginning of the course. 10

Anna Freud National Centre for Children and Families Accountability • POD is an outcome monitoring system used by the Children’s Wellbeing Practitioner to: • Monitor trainee learning and feedback throughout the course • Provide active feedback to clients in sessions after completing an outcome measure 11

Anna Freud National Centre for Children and Families 12 Accountability Outcomes for London and the South East Outcome Measure Completion Adolescent in secondary school RCADS (CYP) RCADS (subscale) Full Session 1 and 8 Session 2 -7 Parent of child in primary school SDQ (P 4 -17) SDQ IMPACT (4 -17) SDQ FOLLOW UP (417) Full Session 1 Session 2 -7 Full Session 8 Goals Goal Based Outcomes Session 1 -8 GBO + Helpful Ideas Tracker Experience Session Feedback Session 1 -8 Questionnaire Experience of Service Session 6, 7 or 8 Questionnaire

Anna Freud National Centre for Children and Families Accountability POD in cohort 1 • CWPs used on themselves to begin with for • Teaching Feedback Questionnaire (every lecture) • Skills and Knowledge Questionnaire (beginning of each term) • Learning Goals (beginning of each term) • Helped to embed the concept of outcome measures and active feedback loop • Feedback enables improvement to be made on the course 13

Anna Freud National Centre for Children and Families Accountability National Dataset Score Descriptives: Total, Mean (SD), 95% CI, Ttest, P value, Hedges G (effect size) Proportion Descriptives: Total, N above cut off T 1 vs T 2, Mean (SD), 95% CI, Mc. Nemer, OR Reliable Change: Total, N (%) Reliable Improvement, Reliable Detoriation, No Change 14

Anna Freud National Centre for Children and Families Accountability Executive Summary This report concerns cases seen by CWPs in Cohort 1 for London and the South East during the period 02/05/2017 to 24/07/2018. Primary outcome measures used for the CWP programme were: Goal Based Outcomes (GBOs), the Revised Anxiety and Depression Scale (RCADS), and the Strengths and Difficulties Questionnaire (SDQ). In this dataset there are 958 cases with time 1 outcome data. Of these, 590 had completed intervention (discharged). The remainder are mostly in treatment or have no recorded outcomes. The following analysis concerns the discharged cases only. For the purpose of this analysis, the outcomes data on GBOs (T 1=430, T 2= 385), SDQ (T 1=227, T 2=155) and RCADS (T 1=385, T 2=287) came from completed intervention cases. The percentage for completion of time 2 data was 89. 5% for GBOs, 68. 3% for SDQ and 74. 5% for RCADS. 15

Anna Freud National Centre for Children and Families Accountability For GBOs: The average score for goals at Time 1 was 2. 45 (including all completed goals) and this increased at Time 2 to 6. 97. This was a highly significant increase with an effect size of 2. 26. A substantial proportion of those who specified goals made a reliable improvement in relation to achieving these (79%). 16

Anna Freud National Centre for Children and Families Accountability For parent SDQs At time 1 the proportion of cases who were in the clinical range on any subscale was 0. 79 (SD=0. 40). Post treatment this was reduced to 0. 50 (0. 50). This was highly statistically significant (χ2(1) = 24. 20, p <. 001). Of the 89 cases above the cut off on any scale, 44% were no longer above cut off on any scale (recovered). The number of cases showing reliable improvement on SDQ Impact subscale was 38%. 17

Anna Freud National Centre for Children and Families Accountability For YP RCADS: · At time 1 the proportion of cases who were in the clinical range on any subscale was 0. 67 (SD=0. 47). Post treatment this was reduced to 0. 38 (0. 49). This was highly statistically significant (χ2(1) = 53. 49, p <. 001). The proportion recovered on all scales was 49. 7%. The number of cases showing reliable improvement is 33% for Total Anxiety and Depression. At post treatment the proportion of clinical cases was only 0. 18 (SD=0. 38) indicating a recovery rate of 45%. 18

Anna Freud National Centre for Children and Families Accountability All available data session feedback was included, including the Session Feedback Questionnaire (SFQ), completed a total of 2922 times by 709 clients, and the Session Rating Scale (SRS), completed a total of 345 times on POD by 86 clients. 96% of sessions were rated in the top 25% of the satisfaction scale for SFQ (16 or above) and 78% for the SRS (36 or above). 19

Anna Freud National Centre for Children and Families Thank you! Any questions please email cypiapt@annafreud. org Our Patron: Her Royal Highness The Duchess of Cambridge Anna Freud National Centre for Children and Families is a company limited by guarantee, company number 03819888, and a registered charity, number 1077106.