Evaluating the Impact of a System Level Intervention

Evaluating the Impact of a System. Level Intervention Using a Developmental Evaluation Approach S. Elizabeth Mc. Gee, MA Evangeline Danseco, Ph. D Kyle Ferguson Purnima Sundar, Ph. D American Evaluation Association Annual Conference, October 23, 2012, Minneapolis, Minnesota

Presentation Overview 1. Context 2. The Task (challenges and opportunities) 3. Understanding our systems intervention through systems thinking and approaches 4. Why System Evaluations made sense 5. Our Approach 6. Limitations and Future Considerations

The Task

The Context Evaluation Framework Six-Year Strategic Plan Ontario Centre of Excellence for Child and Youth Mental Health sector Ontario, Canada

What is the Centre? Vision Optimal mental health and well-being for children and youth. Mission We bring people and knowledge together to strengthen the quality and effectiveness of mental health services for children, youth, and their families and caregivers. 5

Strategic Goals Six year strategic plan and it’s three key strategic priorities: Goal 1 Goal 2 Goal 3 • Foster a culture of organizational learning to support agencies in using evidence to improve client outcomes • Build and develop collaborative partnerships to sustain capacity within mental health services • Be a true learning organization and lead by example 6

Key Centre programs Training Support Services Information products and tools To improve the implementation, evaluation and sustainability of Evidence Informed Practices

The Challenge, the opportunity

Need for Evaluation Information Program Improvements Accountability Understanding our broader impact

The Centre’s work, as a System Intervention Using System Interventions • System interventions seek to change systemwide patterns of behavior among actors by changing underlying system dynamics, structures, and conditions (Wheatley 2001; Eoyang 2007).

The Challenges The Centre’s unique position in the sector • Capacity-building involves collective behavior, evaluation techniques are less widespread. • Changes in organizational behavior may not occur for several years. • Measuring changes in organizational process and decision-making are challenging.

Understanding our systems intervention through systems thinking and approaches

Systems Theory and Evaluation Planning Systems Thinking • Systems thinking provides a perspective that emphasizes the patterns of interrelationships between parts and the whole rather than the parts in isolation (Trochim et al. 2006; Parsons 2007; Hargreaves and Parsons 2009).

Simple, Complicated or Complex? Simple Systems Organization • fixed, static, as well as linear, direct cause and effect relationship between system parts. Complicated Systems Organization • Leaders plan and coordinate the activities of multiple parts • -Self-correcting feedback loops between parts help maintain system equilibrium by counterbalancing and neutralizing a change in the system Complex Systems Organization • Characterized by massively entangled webs of relationships, from which unpredicted outcomes emerge through the self-organizing interactions of many parts or actors within and across system levels • Adaptive, learn and coevolve as they interact with one another and respond to changes in their environment, inherently unstable

Components of our Complexity Components of Complexity system intervention Factors 1. Child and Youth Mental Health sector • funding changes • system transformation • intersections with other sectors (health, addictions, etc. ) • evolving and changing health trends 2. Centre of Excellence (i. e. our programs, services, resources) • collaborating with the Ministry and Regional Offices. • changing programs and resources to meet needs of sector/clients • new evidence/best practices 3. Clients (i. e. child and youth mental health agencies/programs) • 140 core agencies through the province • child welfare, hospitals, community health centres, faith-based centres • rural vs. urban, aboriginal and franaphone populations, etc.

Centre’s Theory of Change model Information Dissemination Education, Training and consultation Knowledge Exchange Stakeholder engagement and collaboration Relationship-building and coaching Improved EIP evaluation and implementation uptake Improved child and youth mental health outcomes

Why systems evaluations made sense

What we need to measure Evaluation Need Accountability Understanding our broader impact Program improvements Type of Systems Evaluation Level of Complexity • Program monitoring and Complicated accountability evaluation • Developmental Evaluation Complex • Process Evaluation Simple

Our Approach

Our Evaluation Context Our Strategic Plan (six-year) Strategic Goals Associated Outcomes

The context in which we are working in, Improved mental health outcomes for children and youth in Ontario 1. Foster a culture of organizational learning to support agencies in using evidence to improve client outcomes Improved evidence-informed practices and growth as learning organizations Enhanced stakeholder knowledge on evidence-informed practices, improved stakeholder skills and enhanced organizational capacity Improved relevance, effectiveness and usefulness of Centre programs and services 2. Build and develop collaborative partnerships to sustain capacity within mental health services Enhanced partnerships and collaborations among stakeholders Increase in and improved engagement with primary stakeholders Enhanced capacity of the Centre to reach key stakeholders and enhanced partnerships among stakeholders 3. Be a true learning organization and lead by example Enhanced capacity of the Centre’s as a ‘Learning Organization’

Our Approach to the Framework Ensuring all Centre programs fall within strategic goals.

Activities Linking Outcome Maps to Strategic Outcomes Consultations - Supporting activities via teleconferences, site visits, email communications Learning resources Webinars, measures database, toolkits, online learning modules, methods mini-kits Link to Strategic Goal Program Outcomes Enhanced awareness of the value of evaluation Increased knowledge of evaluation practices in evidence-informed programs Increased agency buy-in around evaluation Increased program evaluation skills Increased internal capacity to support evaluation systems and structures Strategic Core Outcome: Enhanced readiness among agencies to implement evidence-informed practices Increased agency buy-in around evidence informed programs Increased use of evaluation to inform practices/program improvements using an organizational learning approach Strategic Core Outcome: Enhanced understanding of what is meant by “evidence” among service providers

How do our programs align? Strategic Goal #1 Strategic Goal #2 • Implementation Support Service • Mental Health Literary • Youth and Family Engagement • Evaluation Support Service • Evidence-Insight • Dare to Dream Program • Regional Partners Initiative • e. Mental. Health. ca • Knowledge Exchange Initiative • Emerging Leaders Strategic Goal #3 • Staff Development • Improved Financial Processes • Evaluation and Performance Measurement • Improve Staff Retention and Job Satisfaction

Our Approach To Measuring Impact Developing high-level indicators (and subindicators) to measure the impact of our organization’s work.

Strategic Goals and Outcomes Improved mental health outcomes for children and youth in Ontario 1. Foster a culture of organizational learning to support agencies in using evidence to improve client outcomes Improved evidence-informed practices and growth as learning organizations Enhanced stakeholder knowledge on evidence-informed practices, improved stakeholder skills and enhanced organizational capacity Improved relevance, effectiveness and usefulness of Centre programs and services 2. Build and develop collaborative partnerships to sustain capacity within mental health services Enhanced partnerships and collaborations among stakeholders Increase in and improved engagement with primary stakeholders Enhanced capacity of the Centre to reach key stakeholders and enhanced partnerships among stakeholders 3. Be a true learning organization and lead by example Enhanced capacity of the Centre’s as a ‘Learning Organization’

Impact Areas and Indicators 3. Be a true learning organization and lead by example 2. Build and develop collaborative partnerships to sustain capacity within mental health services Sub. Indicators 1. Demonstrated leadership Sub. Indicators 2. Reach Sub. Indicators 4. Use and Impact of Knowledge Sub. Indicators 5. Relevance, Effectiveness and Usefulness 3. Engagement Sub. Indicators 1. Foster a culture of organizational learning to support agencies in using evidence to improve client outcomes Sub. Indicators 6. Sustainability Sub. Indicators

What we need to measure Evaluation Need Accountability Understanding our broader impact Program improvements Type of Systems Evaluation • Program monitoring and accountability evaluation • Developmental Evaluation • Process Evaluation Data Source Reporting Deliverable • output data • Manager’s activity reports (qual) • survey data (quant/qual) Operations Plan Progress Report • output data • Manager’s activity reports (qual) • survey data (quant/qual) Performance Measurement Scorecard • survey data (quant/qual) Data Feedback Loops Reports TBD

What we need to measure Evaluation Need Accountability Understanding our broader impact Program improvements Type of Systems Evaluation • Program monitoring and accountability evaluation • Developmental Evaluation • Process Evaluation Data Source Reporting Deliverable • output data • Manager’s activity reports (qual) • survey data (quant/qual) Operations Plan Progress Report • output data • Manager’s activity reports (qual) • survey data (quant/qual) Performance Measurement Scorecard • survey data (quant/qual) Data Feedback Loops Reports

Process Evaluation How do Data Feedback Loops work at the Centre? • Program-specific quantitative and qualitative data is aggregated multiple times a year (and annually). • Managers and teams analyze data, look for trends, and implement changes based on stakeholder feedback. • Larger trends across multiple programs are discussed with management and considered in Operations Plan/Strategic Planning (in development).

Limitations and Future Considerations

Limitations • One of the major limitations of our approach was the inability to make causal inferences regarding the influences of system level activities on system level outcomes

Moving forward, Evaluation Need Accountability Addressing Limitations Greater use of Operations Plans (and reflections) to ensure insights are incorporate in future operations and strategic planning. Stronger linkages between the Theory of Change and our Operations Planning Understanding our broader impact Incorporating evaluation directly into Strategic Planning. Better use of qualitative data to understand impact; unexpected outcomes. Exploring the use of a case study analysis Exploring other reporting mediums outside of a Dashboard. Program improvements Making stronger feedback loops at the program level to ensure qualitative data is being used

For more information S. Elizabeth Mc. Gee, MA Research Associate smcgee@cheo. on. ca 613 -737 -7600 ext 3313 Evangeline Danseco, Ph. D Director of Evaluation and Research edanseco@cheo. on. ca 613 -737 -7600 ext 3319