Our Path to Program Evaluation Catulpa Community Support

Our Path to Program Evaluation Catulpa Community Support Services

Beginnings. . �Use of an electronic data entry program to record demographic data and case notes to reflect service delivery �Specific data recorded and reported for each agency program to reflect funder requirements and our own internal standards for service delivery �Program driven annual parent satisfaction surveys

Steps along the way. . . � Explore evaluation processes for all agency programs �Develop agency standards and guidelines for evaluation �Develop program specific logic models � Determine program and agency wide evaluation questions � Utilize evidence based outcome measures for evaluation � Determine if there is an evidence based outcome measure that could apply to multiple agency programs

How we are getting there. . . �Executive Director and Board of Directors commitment to evaluation �Dedicated time for Quality Assurance lead �Support from the Ontario Centre of Excellence for Child and Youth Mental Health �Open, enthusiastic staff

Creating an Evaluation Process and Structure: Simcoe County Wrap. Around Program (SCW)

The Significance of Wrap. Around �To enhance the support networks and care -giving capacity of the family/youth to develop or maintain a safe and healthy environment

SCW Program Logic Model

Process and Outcome Evaluation �Resilience �Goal Attainment �Social Determinants of Health �Fidelity to Wrap Process �Participants experiences with Wrap Service and Wrap Evaluation

Collaborative Partnerships �Formation of the Wrap. Around Research Evaluation Network (WREN): • Catulpa Community Support Services • Wrap. Around Northumberland • Shalem Mental Health Network

WREN Priority �To develop a flexible, innovative evaluation based on the fundamental principles of Wrap. Around.

Exciting Opportunities

Exciting Opportinities �Resilience • Confirm sustainable change for families/clients • Ongoing consultation from Dr. Michael Ungar – Resilience Research Centre (Halifax) • Contributions to research �Child & Youth Resilience Measure (CYRM) �Pathways to Resilience Youth Measure (PRYM) �Resilience Scale for Adults (RSA)

Exciting Opportunities �Contributions �Informing to other Wrap programs a Catulpa Evaluation Framework

Accomplishments within WREN �Establish a shared evaluation framework across the 3 sites �Development of a strengths-based questioning for evaluation. �Development of a measure of family functioning based on the Social Determinants of Health

Evaluation Data

�WREN Evaluation 2012: �Across the 3 sites: • 13 adults completed evaluations at T 1 and T 2 for Resilience, Functional Indicators • 8 adults completed T 1 and T 2 for Goal Attainment Scaling • A total of 33 WFI interviews were completed: 7 caregivers, 14 facilitators, 5 youth and 7 team members

Results: Resilience

Results: Goal Attainment Scaling

Results: Social Determinants of Health �Based on responses to the Functional Indicators questionnaire (T 1 to T 2) �There was a significant increase in means for the following 5 questions: • I enjoy working in the home • My housing is secure and meets my personal/family needs • I have at least one person to confide in • I read adequately for my daily needs • I write adequately for my daily needs

Results: Fidelity

Feedback regarding the Evaluation and Wrap process �Participants indicated strong agreement with the following statements: • • • Overall I am satisfied with the services I received I felt I had someone to talk to when I was in trouble I had a say in how service was delivered to me I received services that were right for me I am now better able to cope when things go wrong

Comments from Parents and Team members “I knew nothing about Wrap previously. It is an amazing program and I would recommend it to anyone. It helped because people believed in me”. (Parent) � “Wrap. Around has improved a lot of aspects of my life by helping me find resources that are available, i. e. respite and other resources. ” (Parent) � “The youth wouldn’t have returned home if not for the planning that occurred within Wrap. Around” (Team member) � “…the program has opened other avenues for the family and helped with independence. ” (Team member) �

What did we learn? � We have meaningful results and now have an effective, robust program evaluation framework with which we can move forward: � There is evidence that the use of the evaluation tools within our framework have “added value” for the children, youth and families served and also assists the Facilitators to be more effective in their implementation of the Wraparound process.

What did we learn? � There is evidence that the use of the Wraparound process can help to build resilience and help children, youth and adults get their needs met with respect to the social determinants of health. � There is evidence that the children, youth and families served are building connections to and developing a community of support and a positive sense of their future.

What did we learn? � The WREN collaboration was effective in developing and implementing a comprehensive program evaluation framework and in developing new evaluation tools � The added time invested to ensure the effective communication and coordination of our collaborative evaluation team (WREN) has resulted in many significant benefits and outcomes that individually we would not have achieved.

Limitations �Our sample size was relatively small and thus further evaluation is necessary to confirm our current results. �Further efforts are required to engage youth in the evaluation process

WREN Next Steps � Continue to conduct our evaluation and report data results within WREN � Devise multiple strategies to engage youth � Eliminate � Devise duplication of questions more creative and time efficient ways to collect data

WREN Next Steps � Flexible strategies for administering the tools that are responsive to children, youth and adults with varying levels of literacy, learning difficulties, developmental delays (etc). � Developing a range of strategies to report data back to participants

Catulpa Next Steps � Develop an Agency Logic Model � Formalize a consistent process for gathering client feedback � Determine program and agency wide evaluation questions � Utilize evidence based outcome measures for evaluation � Determine if there is an evidence based outcome measure that could apply to multiple agency programs � Confirm our terms of reference for an Ethics Review Committee