Module 8 Guidelines for evaluating the SDGs through

Module 8 Guidelines for evaluating the SDGs through an equity focused and gender responsive lens: Overview Technical Assistance on Evaluating SDGs: Leave No One Behind Eval. Gender+ Network together with UN Women, UNEG and Eval. Partners

Presentation developed by Michael Bamberger and Asela Kalugampitiya based on Chapter 4 of ”Evaluating the Sustainable Development Goals within a “No-one left behind” lens through equity-focused and gender-responsive evaluations” 2

Outline 1. Overview of the proposed approach 2. The main stages of the EFGR country evaluation 3. Three Evaluation levels 4. Four main types of evaluation 5. Defining the priority evaluation questions 3

1. Overview of the EFGR evaluation approach The purpose of the Equity-Focused and Gender Responsive (EFGR) evaluation is to: • Assess the contribution of the SDGs towards the reduction of inequalities by: • Assessing progress towards the achievement of SDG-10 and each of its subgoals • Assessing the contribution of each of the other SDGs towards equity outcomes • Assessing the overall contribution of the country SDG program towards the achievement of equity outcomes, the identification of contributing factors and constraints 4

• Assess the contribution of the SDG to the achievement of gender equality and women’s empowerment outcomes • Assessing progress towards the achievement of SDG-5 and each of its subgoals • Assessing the contribution of each of the other SDGs towards gender equality and women’s empowerment • Assessing the overall contribution of the country SDG program towards the achievement of gender equality and women’s empowerment outcomes, the identification of contributing factors and constraints 5

The EFGR evaluation is an integral part of the SDG review process • All countries have committed to conduct SDG reviews • The focus and depth will vary according to country priorities and data collection and analysis capacity • The EFGR evaluations should form an integral part of the overall SDG review mechanisms. 6

Defining equity focused evaluation “ an assessment made of the relevance, effectiveness, efficiency, impact and sustainability of interventions on equitable development results. Equity focused evaluations look at structural bottlenecks and power relationships, and use an empowerment process. ” UNICEF 7

Defining gender-responsive evaluation “UN Women subscribes to the UNEG definition of evaluation but directly incorporates principles of gender equality, women’s rights and empowerment of women…. As a process itself, evaluation is also a means to enhance gender equality and the empowerment of women through the incorporation of gender and women’s rights dimensions into evaluation approaches, methods, process and use. ” UN Women 8

Defining gender evaluation [continued] “Not only does evaluation act as an important driver of positive change towards equality and the empowerment of women, but the way in which the evaluation process itself is undertaken empowers the stakeholders involved. ” UN Women 9

2. Main stages in the EFGR country evaluation • The design and implementation must be adapted to the priorities and data collection and analysis capacity of each country • All stages must be based on extensive consultation with al stakeholder countries at the national, regional and local levels • A participatory process is essential • For EFGR the process of design and implementation is as important as the outcomes 10

MAIN STAGES IN THE DESIGN, IMPLEMENTATION AND USE OF THE EFGR COUNTRY EVALUATION Participatory/ consultative approach at all stages 1. Broad-based consultations to select and develop the EFGR evaluation design Identifying the key evaluation questions 2. Selecting the best combination of evaluation designs 3. Incorporating complexity-responsive approaches [if required] 4. Evaluating resilience and sustainability 5. Conducting an evaluability assessment 6. Dissemination and use of the EFGR evaluations Implementing the evaluation 11

3. The levels at which EFGR evaluations can be conducted • The evaluation of the contribution of SDG country programs to equity and gender equality outcomes can be conducted at 3 levels: • National and sector policy level • Program level • Project level • A comprehensive evaluation will include evaluations at all 3 levels, but in many cases evaluations will only be conducted at 1 or 2 levels 12

The 3 levels at which EFGR country program evaluations can be conducted Country and sector policy evaluation EFGR country program evaluation Project evaluation 13

4. The main types of evaluation design • There are 4 main kinds of evaluation design that can be used • • Policy evaluations Formative evaluations Developmental evaluations Summative evaluations • Each design has a different purpose and addresses different evaluation questions (see section 4) • Many evaluations will combine different evaluation designs 14

The 4 kinds of evaluations that can be used to assess how SDG programs contribute to EFGR goals Evaluation Purpose Policy evaluations How well do the design, implementation and outcomes of national and sector policies contribute to EFGR outcomes? Formative evaluations • Providing regular feedback to management and stakeholders on how the design and implementation of SDG projects contribute to EFGR goals • Proposing corrective measures • Learning lessons Developmental evaluations • Similar to formative evaluation but focusing on innovative and emergent projects • Assessing how complexity dimensions affect implementation and EFGR outcomes Summative (impact) evaluations • Estimating quantitative (and sometimes qualitative) changes in EFGR indicators • Assessing the extent to which these changes can be attributed to the SDG program 15

Policy evaluations • Assessing how well the design, implementation and outcomes of national and sector policies contribute to the achievement of EFGR goals • These may be conducted • Throughout the policy implementation (less common) • Retrospectively (more common • A major challenge is that it may take 4 -6 years before the effects of policies can be fully assessed • So it is often necessary to assess likely outcomes based on progress indicators 16

Formative evaluation • Providing regular feedback to management and other stakeholders to help strengthen the implementation of EFGR-responsive programs and projects • Used through project/program cycle • Closely linked to agile monitoring • Evaluation as a learning tool • Uses wide range of data collection and analysis techniques with focus on mixed methods 17

Developmental evaluation • Similar to formative evaluation but focuses on innovative and emergent projects/programs • Used to evaluate complex programs where conventional evaluation methodologies are difficult to apply • Assumes that many programs evolve and adapt and do not have a defined end-point with a set of defined final outcomes • Requires close collaboration between managers and evaluators 18

Summative evaluation • Purpose is to assess the extent to which observed changes in outcome variables can be attributed to the effects of the intervention • Used for accountability and to assess the potential replicability of a pilot project • Where possible uses a randomized control trial design • Difficult to use for programs with multiple outcomes or operating in complex environments 19

5. The key EFGR evaluation questions • There is no standard set of evaluation questions • Questions depend on the issues of concern to different stakeholder groups • It is essential to define the evaluation questions BEFORE selecting the evaluation design 20

Define information required by different stakeholders Select the appropriate evaluation design DO NOT Select the “best” evaluation design Only provide information on questions this design can address 21

Examples of EFGR questions addressed by each kind of evaluation design [see following slides for more detail] Policy evaluation • How well did the evaluation contribute to EFGR in terms of relevance, efficiency, impact etc ? • Are there short-term indicators for assessing longterm EFGR policy outcomes? Formative evaluation • How likely is the project design to contribute to EFGR outcomes? • Are any sectors being excluded from access to program services? Developmental evaluation • Are there mechanisms to ensure all sectors have access to program services? • Does the design address complexity dimensions affecting EFGR outcomes? Summative evaluation • To what extent can specific EFGR outcomes be attributed to program interventions? • What are the key factors in contributing to success or failure of EFGR outcomes? 22

Typical questions for an EFGR policy evaluation • How well did the policy contribute to EFGR goals in terms of • • • Relevance Efficacy Effectiveness Impact Sustainability • To what extent can observed EFGR outcomes be attributed to the policy? • How influential was donor advice and support in the formulation and implementation of the policy? 23

Policy questions [continued] • For policies whose outcomes will not be achieved for a number of years, are there short-term indicators to show whether the implementation is likely to achieve the intended outcomes? • How do policies contribute to gender equality and women’s empowerment outcomes? • What were the main factors contributing to the success or failure to achieve gender outcomes • How do policies contribute to reducing inequalities? • What were the main factors contributing to the success or failure to reduce inequalities? 24

Typical questions for EFGR formative evaluations • How likely is project design to contribute to EFGR outcomes? • How effectively is the project being implemented? • Is the implementation process consistent with EFGR principles? • Are any sectors of the target population being excluded from access to benefits or services? • Are there any unintended outcomes likely to affect (positively or negatively) EFGR outcomes 25

Typical questions for EFGR developmental evaluations • Are there mechanisms to ensure all sectors of the population are consulted in program design? • Are services and benefits reaching all sectors? • Does the program design identify and address complexity dimensions • Does implementation have the flexibility to adapt to the changing context • Do the different program components and the overall program contribute to EFGR goals? 26

Typical questions for EFGR summative evaluations • To what extent can a specific EFGR outcome be attributed to the project? • Did the intervention make a difference? How? • Will the intervention produce similar EFGR outcomes in other contexts? • What are the key factors contributing to success or failure in achieving EFGR outcomes? • How are EFGR outcomes affected by issues of coordination among different stakeholders? 27