INTRODUCTION TO OUTCOMES PLANNING EVALUATION Contribution Analysis H

INTRODUCTION TO OUTCOMES PLANNING & EVALUATION Contribution Analysis H. Mc. Intosh & S. Wilson 20/10/15 Adapted in part from material prepared by John Connolly, UWS . W

AGENDA 10: 00 Welcome & Introductions 10: 10 Overview of contribution analysis 10: 40 Quiz 10. 55 Participative demonstration: evaluating the impact of applying knowledge to practice—Clinical Enquiry & Response Service (CLEAR) 11: 15 Coffee 11: 30 Group work 12: 30 Overview of performance reporting 13: 00 Lunch 13: 45 Performance Reporting 15: 40 Next steps 16: 00 Close

OBJECTIVES By the end of the session you will be able to: • List the key features of contribution analysis • Appreciate how an outcomes planning approach facilitates evaluation for learning and continuous improvement as well as accountability • Explain the outcomes chain to a colleague • Outline a basic evaluation framework (outcomes chain and monitoring & evaluation plan) for a project relevant to your own area of work

ATTRIBUTION and CONTRIBUTION • Questions about cause and effect are critical to evaluating impact • Attribution analysis – Did the intervention cause the observed results? • Contribution analysis – Has the intervention contributed to the observed results? – Has the intervention made a difference? • Outcomes Planning & Evaluation (OPE) is the process of undertaking contribution analysis

WHAT IS OPE? • A pragmatic approach to plan and evaluate the impact of programmes that operate in complex settings • Focuses on outcomes – does our programme make a difference? • Participative – involves stakeholders • Uses theory of change to map out the expected pathways to the intended outcomes within a time-based framework • Uses evidence to identify effective pathways • Facilitates evaluation for learning and continuous improvement, not just accountability • Supports performance reporting focussed on outcomes; the difference the programme is making

THEORY OF CHANGE • The theory of how and why an intervention works • Explains how a programme, as designed, will theoretically result in the intended outcomes • Shows the plausible links between the programme’s activities and subsequent outcomes • Identifies underlying assumptions • Takes account of the context in which interventions work–how external factors may affect results

OPE Process Theory of Change Current Situation Outcome Plan Gather evidence & develop performance story Gather more evidence Performance Story Outcome Planning Monitoring & Evaluation Performance reporting & sharing learning

TOOLS FOR OPE • Outcomes chain – Outcome planning – Theory of change • Monitoring & Evaluation Plan – Developed from the outcomes chain – Measuring success—identifying indicators • Outcomes Focussed Performance Reporting template

SPHERES OFCHAIN INFLUENCE OUTCOMES The difference you make Direct What do you do? Direct control Indirect influence Inputs Activities Outputs Reach Resources you need What you do What you produce Who you reach Existing practices and capacity Short term outcomes Medium term outcomes Changes in knowledge, skills, awareness Changes in behaviour, practice Long term outcomes Changes in health outcomes Political, technological, socio-economic, environmental, other factors External influences Organisational resources, skills, systems Expected impacts

OUTCOME TYPES • Short term—usually associated with changes in knowledge, skills, attitudes, awareness – Changed attitudes to alcohol and drinking – Librarian’s acquire skills in evidence synthesis to support Practice Based Small Group Learning (PBSGL) • Medium term—usually associated with changes in behaviour and practice at an individual or organisational level – Reduced alcohol-related violence and abuse – Librarian’s provide evidence search & synthesis service support to PBSGL • Long term—aspirational outcome, the overarching goal – Safer and happier families and communities – More widespread application of evidence-based practice

ASSUMPTIONS, RISKS AND COMPETING EXPLANATIONS • What assumptions do you make to get from one step in the outcomes chain to the next? • What are the risks that getting from one step to the next won’t happen as you imagine? – Any barriers you might anticipate? – How might you mitigate these risks? • What are the other influencing factors? – Other programmes, policies, services – Social, political factors – Enablers as well as barriers • What alternative explanations could there be for how the intervention works?

Partner contributions to shared outcomes Long-term Outcomes Medium-term outcomes National/Local Outcomes Culture Practice & Behaviours Environments Short-term Outcomes Reach Activities/ Outputs Inputs Outcomes chain Partner 1 Partner 2 Partner 3

Cross-sector partner contributions – Alcohol Long-term Outcomes Medium-term outcomes Improved mental wellbeing Reduced inequalities in healthy life expectancy Reduced inequalities in alcohol-related deaths and hospital admissions Behaviour Reduced alcohol consumption levels Less drunkenness; less drink-driving Environments Physical: Reduced exposure to alcohol-related hazards Economic: Reduced availability/affordability of alcohol Social: Drunkenness less attractive; sensible drinking the norm Short-term Outcomes Service uptake & engagement Increased sobriety & stability Understanding risks, attitudes to drinking Increased detection rate Compliance with laws Increased price Reduced incentives Reach Hazardous and harmful drinkers Adults with alcohol problems General public targeted Drivers Licensed trade Alcohol industry Preventive services Addiction services Sensible drinking messages Random breath testing Enforcement actions Taxation, displays, promotions, advertising Activities & Outputs Screening & Brief advice Detox, Intensive support Media campaigns NHS Vol orgs Scottish Govt Inputs Outcomes chain Enforcement of planning Enforcement of controls & drink driving laws licensing laws Police Local authorities Industry regulation SG, UK govts, EU

DEFINITIONS RECAP • Outputs are things produced as a result of programme activities • Outcomes are the benefits or results of a programme—the difference the programme makes – the changes that occur in the short, medium and long term as a result of the programme’s activities and outputs • Outputs are not outcomes

OPE PROCESS: MONITORING & EVALUATION Theory of Change Current Situation Outcome Plan Gather evidence & develop performance story Gather more evidence Performance Story Outcome Planning Monitoring & Evaluation Performance reporting & sharing learning

MONITORING & EVALUATION PLAN • Developed from the outcomes chain • Define indicators to monitor progress and measure success • Identify how and from where you will gather the evidence • Agree who has responsibility for data gathering/storage • Establish the analysis plan: how, when, who?

INDICATORS • Outcome indicators – Outcomes are the benefits or results of a programme—changes that occur in the short, medium and long term – Indicator is the measure of the extent to which the outcome is being achieved • Reach – Measures of who was engaged/involved • Reaction – Measures of people’s reaction to the engagement • Resource use, activities undertaken, outputs produced – Measures of the extent to which the programme was delivered as intended

DEFINING INDICATORS • What evidence do you need to be able to report meaningfully on the impact of your programme? • Involve key stakeholders • Consider what data are already collected! • Indicators need to be – Measurable—recorded analysed quantitatively or qualitatively – Precise—understood in the same way by all stakeholders – Sensitive and Consistent • What is ideal versus what is feasible How many indicators do I need? Is hard data better than soft data? Can I use proxy indicators? Do I need balancing measures? Do I have to evidence everything?

EXAMPLE: K 2 A SUPPORT FOR PBSGL Outcomes • Development of PBSGL modules routinely involves support from knowledge services • Relationship between librarians and practitioners is strengthened Indicators • Increase in proportion of modules produced that draw on knowledge services’ evidence search & synthesis service • Change in librarians’ perception of their working relationship with practitioners as a result of their involvement in module production • Qualitative evidence that practitioners value collaboration with knowledge services

MONITORING & EVALUATION PLAN OUTCOMES CHAIN PROGRESS MEASUREMENT Indicators Long-term outcomes Medium-term outcomes Short-term outcomes Reach Outputs Activities Inputs Data source Responsibility

QUIZ N O I T A R & T S N O M E D VE PA I T A IP C I T R

IMPACT OF A CLINICAL ENQUIRY SERVICE For each step in the chain, what will success look like? NHS staff in primary care or remote and rural areas Improved patient care Evidence informed clinical decisions Useful and relevant service which saves practitioner time Improved understanding of diagnosis, prognosis & therapies Web based evidence summaries answering clinical enquiries Develop service: processes, train staff, develop website Team of information professionals and researchers

IMPACT OF A CLINICAL ENQUIRY SERVICE For each step in the chain, what will success look like? Improved care Informed clinical decisions Improved understanding Quality service which saves practitioner time NHS staff in primary care or remote and rural areas Web based evidence summaries answering clinical enquiries Develop service: processes, train staff, develop website Team of information professionals and researchers

ASSUMPTIONS, RISKS, INDICATORS…. Number of requests received. Number of visits to website (reuse). Inaccurate information reported in summaries Practitioners value evidence based ways of working Staff indicate they received new information or resources. Practitioners will apply new learning to patient care Staff indicate their practice was informed by the information provided. External factors override ability to apply new knowledge Number of requests by professional group or health board Number of repeat users. Visits to website. Staff state they would recommend service and saves time.

ASSUMPTIONS, RISKS, INDICATORS…. Practitioners value evidence based ways of working Inaccurate information reported in summaries Number of requests received. Number of visits to website (reuse). Practitioners will apply new learning to patient care External factors override ability to apply new knowledge Number of requests by professional group or health board Number of repeat users. Visits to website. Staff state they would recommend service and saves time. Staff indicate they received new information or resources. Staff indicate their practice was informed by the information provided.

OPE PROCESS: PERFORMANCE REPORTING Theory of Change Current Situation Outcome Plan Gather evidence & develop performance story Gather more evidence Performance Story Outcome Planning Monitoring & Evaluation Performance reporting & sharing learning

OUTCOMES FOCUSSED PERFORMANCE REPORTING • Performance report focussed on what difference you made • Template to help set out your report 1. A headline about your work 2. What you were trying to do and why 3. Your activities/outputs 4. Your results How often should we prepare a performance report? 5. Other factors/challenges 6. Learning • Adaptable to brief and longer formats for different audiences

SUMMARY • OPE is a pragmatic, participative and outcomes focussed approach to plan and evaluate the impact of programmes that operate in complex settings • Enables credible assessment of the contribution a programme makes to observed results • Facilitates evaluation for learning and continuous improvement as well as accountability

OUTCOMES CHAIN The difference you make What do you do? Inputs Activities Outputs Reach Resources you need What you do What you produce Who you reach Expected impacts Short term outcomes Medium term outcomes Changes in knowledge, skills, awareness Changes in behaviour, practice Long term outcomes Changes in health outcomes

SPHERES OF INFLUENCE Direct influence Direct control Inputs Activities Reach Existing practices and capacity Short term outcomes Medium term outcomes Long term outcomes Political, technological, socio-economic, environmental, other factors External influences Organisational resources, skills, systems Outputs Indirect influence