Expost Evaluation Methodology and Theory of Change Farzana

Ex-post Evaluation Methodology and Theory of Change Farzana Ahmed Lead Evaluation Specialist, IED Simona Somma, Evaluation Specialist, IOE

Workshop Agenda Day 1 Ø Workshop Overview Evaluation Context, Theory of Change Ø Design and Monitoring Framework Ø Core Evaluation Criteria Day 2 Ø PCR Case Studies/Group Work Ø Group Presentations 2 Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR Day 3 Ø Case Studies/Group work (contd) Ø Evaluation at ADB and the Evaluation Information System Ø Panel Discussion Day 4 Ø Government Exchange Ø Field Visit Day 5 Ø ASEAN Exchange 2

Purpose of Evaluation 3 Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR

Why Evaluate? Accountability Are we doing the right things? Are we doing things right? Resources are properly allocated and used, and intended outcomes realized Learning lessons Identification of lessons for improving development impact of future policies, strategies, programs and projects Development effectiveness Maximizing development effectiveness of operations through evaluation feedback 4 Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR

Who is Involved? Independent Evaluation Self Evaluation Parliament (oversight) • • Line Management Independent Evaluators SAI IMED (independent evaluation, capacity development) (use of evaluations, acting on recommendations, influence on institution’s directions) (use of evaluations, evaluation capacity development) International Evaluation Community (harmonization and joint evaluations, ECG, OECD-DAC) 5 Citizens Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR Other Departments/ Ministries

Evaluation Context, Theory of Change, 6 Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR

Theory of Change, Managing Results Design Results and chain Monitoring Framework Impact Outcome Outputs Activities Inputs $ 7 Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR Allocate Implement Plan Evaluate Results Monitor resources

Results Chain Impact Outcomes Higher-level development effects partly attributable to the project Welfare effects/ behavioral changes of target group or performance changes of organizations, institutions, systems directly attributable to the project Sector reform outcomes Outputs Activities Inputs 8 Physical goods and services produced by the project Institutional and/or behavioral changes supported by the program Actions and tasks carried out to transform inputs into outputs Financial, human, material and other resources required Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR

Results-Based Management - Example Plan Outcome: Improved Literacy Outputs: Improved Access, Improved quality, etc. Activities: Build/Rehabilitate Schools, Train Teachers Education Sector Evaluate Outcome Has /Can literacy improved? Has employment increased? Monitor Activities Ouputs 9 Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR Budget Inputs: Materials Consultants Travel National Goal Increased Employment/ Improved Literacy Implement Build Schools Train Teachers

Role of Evaluation Design and Monitoring Framework Impact Relevance Plan Impact Sustainability Allocate Outcome Effectiveness Evaluate Results resources Outputs Efficiency Activities Inputs $ 10 Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR Monitor Implement

Definitions of Evaluation Criteria Ø Relevance: Consistency of a project's impact (goal) and outcome (objectives) with the government’s development strategy, and the adequacy of the design. Ø Effectiveness: The extent to which the outputs and outcomes, as specified in the project document or log frame, either as agreed at approval or as subsequently modified, were achieved. Ø Efficiency: How economically resources were converted to results, using indicators such as the economic internal rate of return, costeffectiveness, and process efficiency. Ø Sustainability: The likelihood that institutional, financial, and other resources are sufficient to maintain the outcome over its economic life using indicators such as the financial internal rate of return. • Impact: Contributions to higher level development outcomes, ADB corporate goals, longer-term development outcomes. Unintended positive and negative impacts 11 Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR

ADB Rating 12 Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR

Evaluation Criteria: Sovereign Projects Core Criteria • Relevance • Effectiveness • Efficiency • Likely Sustainability 13 13 Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR Non-core Criteria • Impacts • Socioeconomic • Environment • Institutional • ADB’s performance • Executing agencies performance

Rating Scores of Sovereign Projects Criterion Weight 14 1. Relevance 25% 2. Effectiveness 25% 3. Efficiency 25% 4. Sustainability Overall Assessment (weighted average of above criteria) 14 25% • • Rating Value Highly relevant 3 Relevant 2 Less than relevant 1 Irrelevant 0 Highly effective 3 Effective 2 Less than effective 1 Ineffective 0 Highly efficient 3 Efficient 2 Less than efficient 1 Inefficient 0 Most likely 3 Likely 2 Less than likely 1 Unlikely 0 Highly Successful: Overall weighted average is greater than or equal to 2. 5. Successful: Overall weighted average is greater than or equal to 1. 75 and less than 2. 5. Less than Successful: Overall weighted average is greater than or equal to 0. 75 and less than 1. 75. Unsuccessful: Overall weighted average is less than 0. 75. Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR

Overall Rating Calculations Relevance 3 3 3 2 1 1 2 0 Effectiveness 3 3 2 2 1 1 Efficiency 3 2 3 3 3 2 1 1 Sustainability 3 3 2 2. 5 Overall HS 15 Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR S S 3 3 2. 5 2. 25 2 1 1 1. 75 1. 25 0. 75 S S S LS U

Performance Ratings – Sovereign Operations Highly Successful • Overall weighted average is greater than or equal to 2. 5 • ≥ 2. 5 16 Less than Successful Unsuccessful • Overall weighted average is greater than 1. 5 and less than 2. 5 • Overall weighted average is greater than or equal to 0. 75 and less than or equal to 1. 5 • Overall weighted average is less than 0. 75 • ≥ 1. 75< 2. 5 • ≥ 0. 75< 1. 75 Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR • < 0. 75

IFAD specific evaluation criteria and rating system 17 Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR 17

IFAD’s specific evaluation criteria Rural Poverty Impact and 4 impact sub-domains (income and assets, human and social capital empowerment, food security and agricultural productivity, institutions and policies) Gender Equality and Women’s Empowerment Innovation and scaling-up Environment and natural resources management Climate change 18 Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR

IOE’s Rating system 19 Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR

Take Aways! Evaluation is • grounded in the Theory of Change • a critical component of Management - cannot be seen in isolation • as much about learning as it is about accountability 20 Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR

MONITORING vs. EVALUATION Monitoring Evaluation Timing Ongoing Episodic – at distinct points Nature Checking compliance with plans, targets and processes Identifying areas needing further investigation Investigating reasons for problems Trialling solutions and improvements Learning from success Reporting Regular reporting, dashboards Discrete reports 21 Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR 21

MONITORING Outputs (Products, Services, Deliverables, Reach) VS. EVALUATION Monitoring Question Examples Evaluation Question Examples Ø How many people or communities were reached or served? Ø Were the targeted numbers reached? Ø How adequate was program reach? Ø Did we reach enough people? Ø Did we reach the right people? Process Ø How was the program or policy (Design & implemented? Implementation) Ø Was implementation in accordance with design and specifications? Ø How well was the program implemented? Ø Fairly, ethically, legally, culturally appropriately, professionally, efficiently Outcomes/ impacts (things that happen to people or communities) Ø How substantial and valuable were the outcomes? Ø How well did they meet the most important needs? Were the outcomes worth achieving given the effort and investment put into obtaining them? 22 Ø What has changed since (and as a result of) program or policy implementation? Ø How much have outcomes changed relative to targets? Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR

Summary Planning Validity of the Results Chain DMF Evaluation Framework Allocation of Resources Align inputs to outputs Physical and Monetary targets Implementation Allocated expenditure = actual expenditure (delivery of services) Convert Inputs to Outputs Monitoring Data on inputs, Outcome Evaluation Efficiency, Effectiveness, Sustainability, Relevance – Inputs/Outcomes/Impacts 23 Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR

Theory of Change, Managing Results Design Results and chain Monitoring Framework Impact Outcome Outputs Activities Inputs $ 24 Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR Allocate Implement Plan Evaluate Results Monitor resources

Design and Monitoring Framework (DMF) Design Summary Alternatives Analysis Impact Objectives Analysis Outcome Problem Analysis Data Sources/ Reporting Mechanisms Assumptions/ Risks Outputs Stakeholder Analysis Training on Evaluation of Sovereign Projects 25 Training on Evaluation 20 -24 February 2017| Lao PDR of Performance Targets/ Indicators Activities with Milestones Inputs 25

Rating Scores of Sovereign Projects Criterion Weight 26 1. Relevance 25% 2. Effectiveness 25% 3. Efficiency 25% 4. Sustainability Overall Assessment (weighted average of above criteria) 26 25% • • Rating Value Highly relevant 3 Relevant 2 Less than relevant 1 Irrelevant 0 Highly effective 3 Effective 2 Less than effective 1 Ineffective 0 Highly efficient 3 Efficient 2 Less than efficient 1 Inefficient 0 Most likely 3 Likely 2 Less than likely 1 Unlikely 0 Highly Successful: Overall weighted average is greater than or equal to 2. 5. Successful: Overall weighted average is greater than or equal to 1. 75 and less than 2. 5. Less than Successful: Overall weighted average is greater than or equal to 0. 75 and less than 1. 75. Unsuccessful: Overall weighted average is less than 0. 75. Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR

Overall Rating Calculations 27 Training on Evaluation of Sovereign Projects 20 -24 February 2017| Lao PDR