Common monitoring and evaluation framework for evaluation of





































- Slides: 37

Common monitoring and evaluation framework for evaluation of rural development program Jela Tvrdonova, 2016

Outline of the presentation Strategic approach to RD Common approach to the evaluation - the CMEF Monitoring and evaluation – explanatory notes Evaluation framework Intervention logic Evaluation questions Indicators CMEF structure and content Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009 2

Strategic approach to rural development Focus on limited number of objectives Competitiveness – Environment – Quality of life Axes connected with strategic objectives Strong and dynamic agri-food sector Agriculture and forestry with high added value Employment and growth in rural areas Improvement of the governance in rural areas and mobilisation of the endogenous potential

Why do we need then common approach to M&E? • Increase the accountability and transparency of interventions financed of EU funds, • Demonstrate achievements towards EU national and local objectives, • Assess effectiveness, efficiency, results and impacts, and • Learn how to design and implement policy better • Ensure the comparability of evaluation findings among EU Member states 4

Evaluation in policy cycle Programme design Will it work? Policy priorities Did it work? Ex post evaluation Ex ante evaluation Is it working? Mid-term Evaluation 5

What is the legal framework? Community strategic guidelines for rural development Council Regulation (EC) 1698/2005 Commission Regulation 1974/2006 and its Annex VIII Para 66 of 1698/2005 (revised): “The effectiveness and the impact of actions under the EAFRD also depend on improved evaluation on the basis of the common monitoring and evaluation framework. In particular, the programmes should be evaluated for their preparation, implementation and completion. ” CMEF as guidance to implement the legal framework 6

Common approach to monitoring and evaluation Based on: ü Exact definition of objectives in Regulation, Strategic Guidelines for RD and rural development programs ü EU Strategic monitoring a national strategies ü Definition of baseline indicators at the program start period ü Suitable combination of output, result and impact indicators, ü Common evaluation questions for all RD programs (94) ü Single framework for all program intervetion

Content of the CMEF – Common monitoring and evaluation framework – Handbook, Annexes, Guidance notes Common and additional/program specific indicators Common and additional/program specific evaluation questions Intervetion logic of rural development programs and individual measures Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009 8

Monitoring and evaluation Monitoring – measurement of inmediate outcomes and results at measure and axis level Evaluation - measurement of long-term and rural areas effects/impacts of intervention, within the program development context Comparissons and lessons learned for next interventions Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009 9

Monitoring On-going process which monitors the gradual implementation of the program at the level of financial inpiuts, physical outputs and axes results It subject of annual reporting on the program implementation Instruments – common and additional indicators of output and results,

Strategic monitoring Since 2010 and every other two years Focus of the EC on the progress of national strategic plans and its objectives implementation Assesment of the contribution of national plans to the EU Rural Development Strategy

Evaluation On-going evaluation is based on CMEF and country´s own methodology to carry on the evaluation during the program implementation Process based on annual reporting of result/impact indicators including periodical exercises: Ex – ante: evaluation of relevance of the planned intervention and optimizing of the RDP budget Mid-term and ex-post: for the assessing the progress related to the specific and overall/program objectives,

Monitoring and evaluation in program logic Good program is basic ground for the high quality of monitoring and evaluation The evaluation and the monitoring strats with the program intervention logic Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009 13

Intervention logic § What do we mean by intervention logic? § § § What are major challenges? § Complexity of composition: overall, axis and operational objectives, measures, activities, expected outputs, results and impacts, § Looking beyond the visible - various types of IL (economy, public goods/bads, quality of life), What shall be assessed? § § Relevance, coherence, unintended effects, efficiency, How shall the intervention logic be assessed? § § EU rural policy and RDP intervention logic and its evolution over the time (e. g. Health check), Two stages in assessment: desk scrutiny, facilitated workshops. Specificities with respect to TA and NRN 14

RDP intervention logic - hierarchy of objectives Overall RDP objectives – EU common and RDP specific Axis 1 objectives Axis 2 objectives Axis 3 objectives Competitiveness Environment Quality of life and diversification Measures Axis 4 objectives Leader Measures

RDP intervention logic - hierarchy of objectives Axis 1: Improving competitiveness of the agricultural sector Conditions for measures aimed at promoting knowledge and improving human potential Conditions for measures aimed at restructuring and developing physical potential and promoting innovation Conditions for measures aiming to improve the quality of agricultural production and products Axis 2: : Improving the environment and the countryside Conditions for transitional measures Conditions for measures targeting the sustainable use of agricultural land Axis 3: The quality of life in rural areas and diversification of the rural economy Conditions for measures targeting the sustainable use of forestry land Conditions governing the measures to diversify the rural economy Axis 4: Leader Conditions governing the measures to improve the quality of life in rural areas Training, skills acquisition and animation To implement the Leader approach in mainstream rural development programming (49)

EU policy objectives Intervention logic of RDP Context , its description SWOT and needs assessment Complementarity Relevance Overall objectives EU/MS Programme level Impacts Specific objectives EU/MS Axis level Results Operational objectives EU/MS Measure level Outputs Inputs Measures, projects and their management and implementation Efficiency Effectiveness Coherence RDP Intervention logic

Monitoring and evaluation system Focus of evaluation Basis of evaluation Attribution of impacts Evaluation questions Evaluation methods Intervention logic Measurement tools Data Collection of evidence Indicators

Evaluation questions § The role of evaluation questions in ex post evaluation § Common and programme specific evaluation questions § Evaluation question, judgment criteria, links to indicators: triangular consistency New!§ § § Programme specific evaluation questions § Role of programme specific evaluation questions § Development of programme specific evaluation questions and judgment criteria, links to indicators How to use evaluation questions in ex post evaluation of RDP? § § Revised set of common evaluation questions In structuring, observing, analysing, judging, reporting Specificities with respect to TA and NRN 19

New! Revised set of common evaluation questions Original set of 150 common evaluation questions of CMEF has been simplified and reduced to the essential demand for knowledge from the European perspective. 3 groups of CEQs Programme-related: Community strategic priorities Health Check objectives 7 impact indicators TA and NRN Efficiency of RDP resources Measure-related: Contribution of the measure to its axis objectives Other measure effects and contribution to other axes objectives Leader approach-related: Community strategic priorities: Employment, diversification and governance Leader approach LAG contribution to LDS and RDP 20

Hierarchy of objectives and indicators 5 types of indicators coresponding with the hierarchy of objectives Hierarchy of objectives: - instrument helping to show local activities contribute to the overall objectives, It is composed of: - operational objectives (measure level) - specific objectives (Axes/priority areas level) - overall – cross cutting/strategic/horizontal objectives (at the program level) Indicators copy objectives in the hieararchy

Role of indicators Indicators used as tools to assess how far the expected objectives have been achieved by measures or whole programmes should be specific, measurable, available / achievable in a cost effective way, relevant for the programme, and available in a timely manner (SMART) Indicators can not always be filled with quantitative statistical data; in some cases, indicators might also include qualitative assessments or logical assumptions Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009 22

Indicators § The CMEF indicators in ex post evaluation § Definitions, role and types of indicators (baseline, input, output, result and impact) § Values for indicators Programme-specific indicators § § § How to use indicators in evaluation of RDPs § Balancing importance and measurability § Coherence with intervention logic, links to evaluation questions, methods and data, § Interpretation of values, Specificities with respect to Leader § § Existing indicators and development of new indicators New! Examples of additional indicators linked to revised CEQ Specificities with respect to TA and NRN 23

The quality of indicators - SMART approach S – simple, specific M – measurable - data exist A – achievable – available – at low cost R – relevant – selected well for given program T - timely oriented

Types of indicators Baseline indicators: they relate to general socioeconomic context of the programme area (contextrelated baseline indicators) and to the state of the economic, social or environmental situation in direct relation with the wider objectives of the programme (objectives-related baseline indicators) Financial execution (input) indicators: they refer to the budget or other resources allocated to the programmes Output indicators: measure activities directly realized within programmes Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009 25

Types of indicators Result indicators: measure the direct and immediate effects of the intervention and provide information on changes that have taken place Impact indicators: refer to the benefits of the programme both at the level of the intervention but also more generally in the programme area. They are linked to the wider objectives of the programme Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009 26

The process Monitoring and Evaluation process Impact Result Ouputs Activities Inputs

Common indicators A common set of baseline, output, result, and impact indicators for the RDPs (Art. 62 Reg. 1974/2006) “shall form the Common Monitoring and Evaluation Framework (CMEF)’’ (Annex VIII lists the common indicators) Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009 28

Additional indicators Since common indicators may not fully capture all effects of programme activities, it is necessary to define additional indicators within the programmes (see Guidance notes A & K) Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009 29

Why additional indicators? Challenges of the CMEF & Ongoing Evaluation The CMEF provides the Common Framework to be applied throughout the EU, thus ensuring comparability The specificities of each single programme cannot be fully reflected by this Common Framework It is therefore important to complement the Common Framework by additional indicators in order to capture the full range of intended effects of a given programme As general rule, a thorough analysis of the programme intervention logic can drive the choice of relevant additional indicators Considering the limited number of common impact indicators and their broad scope, additional indicators are essential to overcome attribution gaps 4/5 June 2009 30

Methods Methodological challenges in assessing programme effects Distinction between programme results and impacts Identification of main programme effects – direct and indirect (leverage, deadweight, multiplier, displacement, substitution effects) Identifying programme net effects Distinction of positive and negative externalities, expected and unexpected programme effects Identifying factors distorting programme effects (confounding factors, selection bias, self-selection bias, homogeneous and heterogeneous treatment effects, etc. ) New structure and more guidance Addressing methodological challenges Using counterfactual in addressing methodological challenges Evaluation designs (randomised, quasiexperimental, non-experimental) Key approaches to evaluation (Theory based, quantitative, qualitative, mixed) – advantages, disadvantages, when to be used, various techniques to apply etc. How to select appropriate evaluation design (quality standards, considering limits and constrains, e. g. resources) Suggested methods in relation to the assessment of individual axes, TA, NRN and answering related evaluation questions 31

Data Major challenges in data management and collection during the evaluation of RDP Data needed for evaluation of RDP results at beneficiary level Data needed for evaluation of RDP results and impacts at RDP level § § Economic § Environmental § Social Specificities with respect TA and NRN 32

CMEF structure Handbook Annex 1: Guidance notes - A: Choice and use of indicators, B: On-going evaluation , C: Mid-term evaluation Annex 2: Guidance notes – D: Hierarchy of objectives, E – Measure Fiches Annex 3: Guidance notes – F – K, Common indicators Fiches Annex 4 Guidance J – O Other guidances Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009 33

CMEF guidance on indicators CMEF provides guidance for MAs in setting up indicators at each level of intervention and baseline indicators Annex 3 of the CMEF Handbook provides detailed description of all common indicators (Guidance notes F – K): Challenges of the CMEF & Ongoing Evaluation F: COMMON INDICATOR LIST (overview of all common indicators) G. BASELINE INDICATOR FICHES (detailed description of indicators) H. OUTPUT INDICATOR FICHES I. RESULT INDICATOR FICHES J. IMPACT INDICATOR FICHES 4/5 June 2009 34

Indicator Fiches Each indicator fiche contains the following elements: § § § Type of indicator Related measures Measure Codes Definition of the indicator Subdivision Unit of measurement Level of collection Responsible actor for collection Collection method/good practice Sources Registration frequency Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009 35

Measure Fiches Guidance on practical use of indicators in monitoring and evaluation of RDPs Guidance note E provides the following on use of indicators within each particular measure: Challenges of the CMEF & Ongoing Evaluation Measure Code Rationale of the measure Target group Target area Common indicators Link rationale of the measure and indicators Evaluation questions 4/5 June 2009 36

Thank you for your attention! jelatvrdonova@gmail. com