Lesson 2 Monitoring evaluation and indicators Macerata 4

Lesson 2: Monitoring, evaluation and indicators Macerata, 4 th October Andrea Gramillano, t 33 srl

Agenda v What do we need from the last lesson? v Accountability and monitoring q Definition and types of indicators q Quality of the indicator system v Evaluation q Criteria of evaluation q Timing of evaluation

WHAT DO WE NEED FROM THE LAST LESSON?

Some references for defining theoretical concepts • • • Policy: A deliberate plan of action to guide decisions and achieve rational outcome(s). The term may apply to government, private sector organizations and groups, and individuals. Presidential executive orders, corporate privacy policies, and parliamentary rules of order are all examples of policy. Policy differs from rules or law. While law can compel or prohibit behaviours (e. g. a law requiring the payment of taxes on income) policy merely guides actions toward those that are most likely to achieve a desired outcome. Programme (policy delivery tool): the donor defines criteria, priorities, objectives, call for projects and allocate a budget for a set of projects, without saying which projects Projects: have sharp costs, clearly identified beneficiaries, timing and activities, output and outcomes

Build project’s rationale: Theory of change CONCLUDING NOTES ON THE SOCIAL PHILOSOPHY TOWARDS WHICH THE GENERAL THEORY MIGHT LEAD (J. M. Keynes – General Theory) • The outstanding faults of the economic society in which we live are its failure to provide for full employment and its arbitrary and inequitable distribution of wealth and incomes • “The ideas of economists and political philosophers, both when they are right and when they are wrong are more powerful than is commonly understood. Indeed, the world is ruled by little else. Practical men, who believe themselves to be quite exempt from any intellectual influences, are usually slaves of some defunct economist. ”

Why do we need a project? To make a change… The project is …. . • Project vs. routine • Project is limited in time • Resources are defined • Beneficiary identified To… Solve a problem / social need / change a behaviour: - Economic growth - Accessibility - Social inclusion …. .

Project life cycle The different phases • Identification • Formulation • Implementation • Evaluation & Audit Approach: problem solving Example

KEY QUESTIONS ON STEPS • Identification: Is the project concept relevant (in line with the needs, challenges and consistent with the policy priorities)? • Formulation: Is the project feasible, coherent and producing sustainable benefits? • Implementation: Are results being achieved (efficiently and effectively)? • Evaluation & Audit: Have planned benefits been achieved? What are the lessons learned? Has there been compliance with applicable rules and laws?

Some references for defining theoretical concepts • The identification phase delivers a project idea (a pre-feasibility study) and identifies all the problems and possible solutions. • The formulation phase is after the identification phase: project applicants’ delivery is a sort of feasibility study, explaining why and how the proposed project is well-grounded and is likely to reach the objectives, outputs and outcomes and in particular to address the challenges and needs of beneficiaries and target groups. The main purposes are: confirm the relevance and feasibility of the project idea (see identification phase); provide more details about technical, operational, economic, financial aspects…; prepare a financial proposal and a financing decision.

Relevance and Coherence ( EX ANTE = before financing decision) Needs Resources (inputs) Relevance Output (implementation) Programme / policy Outcome External coherence Internal Coherence

FRAMEWORK FOR PROJECT DEFINITIONS • INPUT: Financial, human, material, organisational and regulatory means mobilised for the implementation of an intervention. Monitoring and evaluation focus primarily on the inputs allocated by public authorities and used by operators to obtain outputs. • OUTPUT: physical" product of spending resources through policy interventions. Examples are: the length, width or quality of the roads built; the number of hours of extra teaching hours provided by the intervention; the capital investment made by using subsidies. • Outcome: The specific dimension of the wellbeing of people that motivates policy action, i. e. , that is expected to be modified by the interventions designed and implemented by a policy. Examples are: the mobility in an area; the competence in a given sector of activity INPUT OUTCOME

Glossary • 1. Stakeholders: Individuals or institutions that may – directly or indirectly, positively or negatively – affect or be affected by a project or programme. • 2. Beneficiaries: Are those who benefit in whatever way from the implementation of the project. They may be: q (a) Target group(s): The group/entity who will be directly positively affected by the project at the Project Purpose level. This may include the staff from partner organisations; q (b) Final beneficiaries: Those who benefit from the project in the long term at the level of the society or sector at large. • 3. Project partners: These are financial recipients and those who implement the projects in-country (who are also stakeholders, and may be a ‘target group’).

Questions (for the exam) • • Explain the difference between policy, programme, project. What are the key steps / phases of the project life cycle? Provide the definition of the formulation phase in the project cycle. What are the main criteria of the project assessment before approval/implementation? • What is the possible use of the SWOT analysis? • Please distinguish between internal and external coherence. • What does the relevance criterion mean? When is it used?

Project development SELECTION Identification and Formulation Needs Resources (inputs) Implementation Realisation (output) Outcome END OF PROJECT At / after Conclusion

ACCOUNTABILITY, MONITORING & EVALUATION

Accountability in Governance – World Bank The concept of accountability involves two distinct stages: answerability and enforcement. • Answerability refers to the obligation of the government, its agencies and public officials to provide information about their decisions and actions and to justify them to the public and those institutions of accountability tasked with providing oversight (there is someone answering and explaining decisions) • Enforcement suggests that the public or the institution responsible for accountability can sanction the offending party or remedy the contravening behavior. As such, different institutions of accountability might be responsible for either or both of these stages. (there is someone who can sanction)

What about ACCOUNTABILITY • Project • Community • Donor • Beneficiary • Tax payer • State Evaluation Monitor Audit

Monitoring, Auditing and Evaluation Monitoring: observing and collecting data form the project. What time is it? 12, 34 At what time is the train ? 11, 50 Auditing: judgement against a rule/ standard The train is at 11, 50 and you missed it Evaluation: interpreting data to provide a subjective judgement for accountability and learning: • You did not catch the train because of…. . • An alternative can be……. .

Evaluation Based on monitoring, evaluation is the systematic collection and analysis of data needed to make decisions, a process in which most well-run programs engage from the outset ( American Evaluation Society) Evaluation tries to answer two distinctive questions: • Did the public intervention have an effect at all and if yes, how big – positive or negative – was this effect. The question is: Does it work? Is there a causal link? This is the counterfactual question • Why an intervention produces intended (and unintended) effects. The goal is to answer the “why and how it works? ” question. To answer this question is the aim of theory-based impact (European Commission – DG REGIO) An evaluation is an assessment, as systematic and objective as possible, of an on-going or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, developmental efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of both recipients and donors ( OECD/DAC)

Monitoring • To monitor means to observe. • Monitoring is the regular observation and recording of activities taking place in a project or program. It is a process of routinely gathering information on all aspects of the project. • To monitor is to check on how project activities are progressing. • Monitoring also involves giving feedbacks about the progress of the project to the donors, implementers and beneficiaries of the project. • Reporting enables gathered information to be used in making decisions for improving the project performance. Context indicators Identification Evaluation and review Formulation Implementation Input indicators Outcome indicators Impact indicators

Indicators: what is it needed? v. Definition v. Unit of measurement: square meters / km / Euro v Source of information: going from internal (project activities, project survey) to external (statistical institute and administrative registers) v. Baseline (i. e. the initial value against which an indicator is subsequently measured) ü Refer to local context; or ü Refer to activities in previous programmes v. Current status (milestones) v. Target (the expected final value) Not for all the indicators

Context indicators: ü reflect the socioeconomic conditions of a programme area ü they enable to assess local needs of the programme area, to assess how the general context of a programme is evolving

Output indicators: direct products of the programme ü Could require information from beneficiaries, or can be calculated directly at the programme level

Outcome indicators: relate to the effect brought about by a programme. They provide information on changes for the well being of the people to, for example, the behaviour, capacity or performance of beneficiaries.

ROAD CONSTRUCTION Indicator type Indicator Name Unit Output Length of rehabilitated/modernized county roads KM Outcome Passengers and freight traffic on the rehabilitated, constructed, modernized roads Number / Tons

SOCIAL INFRASTRUCTURE Indicator type Indicator Name Output Rehabilitated/equipped health care mobile units (total and by type) Outcome Average response time of mobile units Unit N Minutes

BUSINESS ENVIRONMENT Indicator type Indicator Name Unit Output Micro-enterprises created and supported N Outcome New jobs created in the supported microenterprises N

TOURISM PROMOTION Indicator type Indicator Name Unit Output Tourism infrastructure / accommodation projects implemented N Outcome Increase of overnights-staying %

Classification of the indicators according to project steps Context indicators Identification Evaluation and review Formulation Implementation Input, outcome, impact indicators

Indicators – Physical, financial, procedural v Physical indicators: describe the concrete ‘products’ of the programme; v Financial indicators: provide a basic picture for usage of the available resources (i. e. how fast? which priorities? ): ü Committed/spent; ü Year/Priority/Source of funding. v Procedural indicators: provide information on the current status of the operational level and forecasts for the next steps.

WORKOUT

Quality - individual indicators SMART Specific indicator, if it is appropriate to the phenomenon is measuring (e. g. the project is about transport accessibility and finances bridges, the indicators measure the number of passengers and the length of bridges) Measurable indicator, if it has a measurement unit and it is possible to measure the baseline and the target, there is a source Attainable (achievable) indicator, if it seems possible to achieve the expected target value Relevant indicator, if it measures the contribution of the project to addressing the needs Time – bound, if the indicator is available and updated in different periods

Quality - individual indicators v Clarity (in particular for result indicators) CLE (the definition and the direction of change are clear) A (available indicator or ad hoc) R (the methodology of construction is defined and sound)

Quality - individual indicators RACER criteria 1. Relevant (adequate thematic and financial coverage and policy responsiveness) 2. Accepted (understood by those in charge of data collection and its measurability is in line with capacity of the authorities to absorb data and information is limited) 3. Credible (unambiguous) 4. Easy to monitor (feasible in terms of costs for the authorities) 5. Robust (clearly defined and with a sound methodology)

WORK OUT Project BRIDGE invests EUR 2 billion to build two bridges in order to improve accessibility of the country, in particular of the rural villages located in the North of the country. How can you assess the following indicators? Type (input, outcome) Km of roads renovated in 2015 (source: MS) % increase in the accessibility index 20 % increase in the accessibility index in 2015 (knowing that similar target have never been achieved in the past) Number of project activities finalized each year (source: MS) Number of inhabitants improving their life conditions (source not identified) Increase of wellbeing in 2015 in the country (source: NAT STAT) Euro spent (source: MS) Number of bridges built in 2014 in rural villages located in the South of the country (source: MS) 10% increase in satisfaction of health services in 2016 (source: NAT STAT, knowing that similar targets have been achieved in the past) Legend: Y = Yes; N = No; ? = it depends or not enough information available; / =not pertinent question Physical, financial, procedural S M A R T

Impact Outcome CONTRIBUTION TO POLICY OR PROGRAMME OBJECTIVES BENEFITS FOR THE TARGET GROUPS Output TANGIBLE PRODUCTS / SERVICES Input ACTIVITIES and also financial values

Programme «International Development Programme» The International Development Programme for 2007 -2013 provides an excellent opportunity for the region “POOR” to further progress its economic development, through priorities determined by the region itself and defined by a strong regional partnership drawn from all parts of the region. In addition to International Development Programme, the Regional Economic Strategy (RES) represents the “POOR’S” response with the following targets: • To raise the region’s GVA per head from 80% to 90% of the national average • To get between 61, 000 and 73, 000 more people in the region into work • To create between 18, 500 and 22, 000 new businesses Programme Priorities The International Development Programme comprise three priorities: • Priority One: Enhancing and Exploiting Innovation (53%). • Priority Two: Business Growth and Enterprise (43%) • Priority three: Addressing Climate Change Accordingly, through Priority 1, the strategy will: • Increase the region's value-added by investing in ‘opportunity’ to exploit the region's science and technology strengths to take advantage of growing markets. I • Invest in integrated projects that, on one level, will strengthen capacity to apply science, technology and innovation in businesses. The programme will invest exclusively by using financial instruments as: • Guarantee schemes in combination with grants • Loans • Venture Capital

External coherence: - Is the project A (B) coherent with the programme? (vertical coherence) - Is the project A coherent with project B? (horizontal coherence) - In terms of objective, activities, and type of support? Internal coherence Has the project A (B) the appropriate internal coherence? Objective and activities? Activities and budget? the programme? (vertical coherence) Is the project A coherent with project B? (horizontal coherence) Project A Project B Objective Improve the capacity of social enterprises to develop and deliver sustainable and high impact innovations Objective Introduce technologies, methods and tools to organise this circular approach to sediment management. Activities Research models Activities Organisation of meetings and drafting of manual for circular economy (e. g. recycling) on effective business Pilot test on new business models, evaluation of business models Pilot tests Establishing network New investments Building a motorway Budget Estimated budget 3, 800, 000. 00 Euro Type of support Guarantee scheme without combination with grant any Budget Estimated budget 5, 800, 000. 00 Euro Type of support Grant

See you www. t 33. it a. gramillano@t 33. it