Lesson 3 Performance effectiveness efficiency Macerata 12 October

Lesson 3: Performance, effectiveness, efficiency Macerata, 12 October Andrea Gramillano, t 33 srl

Agenda 1. What do we need from the last lesson? 2. What is performance? 3. What does effectiveness mean? 4. What does efficiency mean?

1) What is needed from the last lesson?

Phase Criteria Tool Time Identification (of needs) Relevance SWOT - PEST Before the project starts Formulation (feasibility study) Coherence (at least) Matrix analysis and document analysis Before the project starts Appraisal and approval Implementation Performance Efficiency Effectiveness Evaluation and audit Impact Sustainability Monitoring / indicator Multicriteria analysis Cost effectiveness Comparison achieved value and target During the project implementation In an advanced phase and at the end

Relevance and Coherence ( EX ANTE) Needs Programme / policy Utility Resources (inputs) Internal Coherence Output (implementation) Result (Specific Objective) External coherence Relevance

WORK OUT – Solution of the exercice lesson 3 Priority axis: Improving accessibility of the country Specific objective: Increase the accessibility of the rural villages located in the North of the country Type (input, outcome) Physical, financial, procedural S M A R T Clear (only for outcome) Km of roads renovated in 2015 (source: MS) Output Physical Y Y ? Y Y / % increase in the accessibility index outcome / Y Y ? Y N Y 20 % increase in the accessibility index in 2015 (knowing that similar target have never been achieved in the past) outcome / Y Y N Y Y Y Number of project activities finalized each year (source: MS) Input Procedural Y N ? Y Y / Number of inhabitants improving their life conditions (source not identified) outcome / N N ? ? N Y Increase of wellbeing in 2015 in the country (source: NAT STAT) outcome / N Y ? ? Y Y Euro spent (source: MS) Input Financial Y Y ? Y N / Number of bridges built in 2014 in rural villages located in the South of the country (source: MS) Output Physical N Y ? Y Y / 10% increase in satisfaction of health services in 2016 (source: NAT STAT, knowing that similar targets have been achieved in the past) outcome / N Y Y ? Y Y Legend: Y = Yes; N = No; ? = it depends or not enough information available; / =not pertinent question

Indicators v. Definition v. Unit of measurement v Source of information v. Baseline (i. e. the initial value against which an indicator is subsequently measured) ü Refer to local context; or ü Refer to activities in previous programmes v. Target (combined with baseline information, provides information concerning the importance of the planned interventions)

Phase Criteria Tool Time Identification (of needs) Relevance SWOT - PEST Before the project starts Formulation (feasibility study) Coherence (at least) Matrix analysis and document analysis Before the project starts Appraisal and approval Implementation Performance Efficiency Effectiveness Evaluation and audit Impact Sustainability Monitoring / indicator Multicriteria analysis Cost effectiveness Comparison achieved value and target During the project implementation In an advanced phase and at the end

2) Performance

Performance: problematic defintion. OECD definition: The degree to which a development intervention or a development partner operates according to specific criteria/ standards/ guidelines or achieves results in accordance with stated goals or plans. EU Commission Defintion: The meaning of the word performance is not yet stable; it is therefore preferable to define it whenever it is used. Performance might mean that intended results were obtained at a reasonable cost, and/or that the beneficiaries are satisfied with them. Efficiency and performance are two similar notions, but the latter extends, more broadly, to include qualitative dimensions.

Performance: When? On going Evaluation Criteria the project activities are delivered on time, the outputs respect the targets, the resources are duly absorbed, the procedures are done according to the rules. Needs Resources (inputs) Output (implementation) Outcome (Specific Objective)

Performance questions: The project: - is able to spend all the financial resources; - meets the procedural deadlines; - achieved the target in terms of physical realization. And in the next period: - What is needed to increase the project performance? (human resources? political support? administrative enforcement? - Will the project be successfully completed? - Which might be future challenges Performance evaluation is based on monitoring indicators (procedural, financial, physical, output indicators)

Procedural monitoring Most public activities have to follow a more or less rigid schedule in which the different steps are mandated and the deadlines fixed (i. e. ). Procedural monitoring usually provides information about how project pipelines are progressing (where and when calls for tenders have been published, contracts have been awarded, …). Final payment …… Contract awarded Call published Specifications ready …… …… Actual …… …… …… Expected Actual …… 19 -08 -10 Expected Actual 08 -08 -10 Expected 14 -08 -10 Actual Status Open Expected 08 -08 -10 Procedure P 1

Physical monitoring Example of physical monitoring: Indicator (number of enterprises) Unit of Measurement Target Achievement Micro N 142 91 Small N 133 111 Medium N 39 21 Owner (women) N 50 40 Owner (<30 y) N 26 3 Start-up N 54 3

Financial monitoring Example of financial monitoring: Priority P 1 Expected expenditures Resources committed Expenditures Amount % (a) (b/a) (c/a) 133. 4 100. 4 75. 2 71. 8 53. 9

Multicriteria analysis

Multicriteria (assessment tool for performance) Tool used to compare several interventions in relation to several criteria. Multicriteria analysis is used also in the ex ante evaluation for comparing proposals. It can also be used in the ex post evaluation of an intervention, to compare the relative success of the different components of the intervention. Finally, it can be used to compare separate but similar interventions, for classification purposes. Multicriteria analysis may involve weighting, reflecting the relative importance attributed to each of the criteria. It may result in the formulation of a single judgement or synthetic classification, or in different classifications reflecting the stakeholders' diverse points of view. In the latter case, it is called multicriteria-multijudgement analysis. (from EVALSED)

Process Step 1 criteria Step 2 scoring or ranking Step 4 aggregating Step 3 weighting

STEP 1: setting criterion Criterion 1: Financial Performance Project 1 Project 2 Project 3 Project 4 Project n… Criterion 2: Procedural Criterion 3: Physical realisation

STEP 2: Score (judgment) - It is needed to find a way to appraise the project according to different aspects since we used different measuring units for different aspects of different process - We can opt for: - A) Scoring: by assigning a numeric value to different “interval” of performance. For example 3 for “above average” – 1 for “on line with average” – 3 for “below average” - B) Ranking: we simply order the different projects according to their performance from the first to the last

STEP 2: Scoring Criterion 1: Criterion 2: Criterion 3: Project 1 1 Project 2 0 2 1 Project 3 3 1 1 Project 4 3 3 0 Project n…

STEP 3: Establishment of weight If some Criteria is more important than others it shall be given more importance. To do it we simple apply a multiplication factor > 1 (e. g. 1, 5). Some criteria may have such importance that they have to be singled out. This is the case for criteria determined by a veto threshold (For example “Physical” if some project has 0 performance, it is excluded by the analysis).

STEP 3: Apply the weight Criterion 1: ( * 1, 5) Criterion 2: Criterion 3: Project 1 1, 5 1 1 Project 2 0 2 1 Project 3 4, 5 1 1 Project 4 4, 5 3 out Project n…

STEP 4: Aggregate the score Criterion 1: Criterion 2: Criterion ( * 1, 5) 3: Total (with weight) Total (without weight) Project 1 1(1, 5) 1 1 3. 5 3 Project 2 0 2 1 3 3 Project 3 3(4, 5) 1 1 6, 5 5 Project 4 3(4, 5) 3 out 6 Project n…

WORK OUT: SME INCUBATOR Criterion 1: Economic (average increase of turnover) Project 1 Project 2 Project 3 Project 4 Criterion 2: New Jobs Criterion 3: Satisfaction for quality service Total

WORK OUT DATAS Project 1 2 3 4 Economic 80% 70% 30% 70% Physical 50 10 30 5 Quality High Low High Medium Quality levels Low Medium High

Apply weight A Economic * 1, 5 Physical Veto = N. of job < 10 B Physical * 2 Quality Veto for “Low”

3) Effectiveness

Effectiveness (definition 1) The extent to which the development intervention’s objectives were achieved, or are expected to be achieved, taking into account their relative importance. Note: Also used as an aggregate measure of (or judgement about) the merit or worth of an activity, i. e. the extent to which an intervention has attained, or is expected to attain, its major relevant objectives efficiently in a sustainable fashion and with a positive institutional developmental impact. (OECD)

Effectiveness (definition 2) The term effectiveness has many possible meanings. The most common definition identifies effectiveness with “achievement of objectives”. This leaves open the definition to the different meanings of “objectives”. Objectives can be expressed quantitatively in terms of expected output or results. The effectiveness is evaluated simply by comparing what has been obtained with what had been planned: outputs and results indicators are all is needed. (European Commission – DG REGIO EVALSED GUIDE)

EFFECTIVENESS: WHAT TO EVALUATE ? EUROPEAN COMMISSION Quality: effectiveness is evaluated by comparing results with quality standards. OECD Effectiveness assesses whether the results outlined in the logframe are delivered and if they are likely to produce the expected objective. Ability of a given action to produce Evaluating effectiveness should a desired change: comparing include assessment of how what is observed after the action people (women and men) benefit has taken place with what would from the results brought by the have happened without the action. project One needs data that allow recovery of the counterfactual situation.

Evaluation Questions • To what extent were the originally defined objectives of the development intervention realistic? • To what extent have the (direct) objectives of the development intervention been achieved in accordance with the (adjusted, if applicable) target populations? • What are the (concrete) contributions of interventions for achieving the objectives of the development intervention? • What factors were crucial for the achievement or failure to achieve the project objectives so far (indication of strengths and weaknesses, e. g. the monitoring and evaluation system)? • What is the quality of development-policy, technical planning and coordination ?

4) Efficiency

Efficiency Definition Type of Questions • OECD: the project results 1. Was the budget adequate? Was the have been achieved at spending commensurate with the reasonable cost with delivery of activities and achieving minimum waste of effort, time, money and skills. results? • European Commission: 2. Were the human, financial, material obtaining a given output resources adequate in terms of at the minimum cost or, quality and quantity to achieve the equivalently, with project results? maximizing output for a given level of resources. 3. To what extent were costs of the project justified by the benefits in comparison with similar projects or known alternative approaches?

Information and Data on OUTCOME Indicators Methods Quantitative - Financial - Physical - Procedural Qualitative - Opinions on the level of achievement - Perceptions on satisfaction • • Literature review Interview Community interview Project visit Focus group Case study Survey

Different way for collecting information

Cost-effectiveness assessment

Cost - effectiveness Project A Project B Public contribution to each firm / project 100, 000 Euro 155, 000 Euro Number of jobs created in each firm 2 FTE 3 FTE Ratio 50000 51666

Cost – effectiveness Project A Project B Public contribution to each firm / project 100, 000 Euro 155, 000 Euro Number of jobs created in each firm 2 FTE 3 FTE Ratio 50000 51666

Cost - effectiveness Project A Project B Public contribution to each firm / project 100, 000 Euro 155, 000 Euro Number of jobs created in each firm 2 FTE 3 FTE Ratio 50000 51666

4) Exercise on effectiveness

Question n. 5 “Project benchmarking” (score of the exercise 5 points) Three projects have been just finalized. Their objective was to create new jobs. The following table provides figures on the performance of the projects. • Performance indicator: -New jobs created (in full time equivalent employees) – TARGET value -New jobs created (in full time equivalent employees) – ACHIEVED value • Total financial resources actually spent to reach the achieved value in dollars New jobs created – Pro TARGET ject value A B C 2 25 8 New jobs created – ACHIEVED value 2 5 4 Total financial resources spent to reach the achieved value 200 100 40

Effectiveness / Efficiency - exercise PROJECT: ONE WAY TO THE CENTER OBJECTIVE: The project aimed at increasing accessibility by building three bridges in the period 2012 -2015 investing 3 millions. At the end 4 millions were spent and two bridges built. EXPECTED OUTCOME: At the project start, the idea was to increase the accessibility by reducing the time to go from the city center to the periphery (from 50 minutes to 25 minutes). In 2016, going from the city center to the periphery takes 20 minutes (thus less than in the expected outcome). The overall circulation in all the city improved a lot, with a general reduction by 5% of time to move in the city (go from any area in the city to another one). TASKS • In which phase of the project cycle are we ? • Fill in the following table • Can you give any explanations of what happened in terms of increased accessibility (at project area and city level)? • Was the project effective? Was the project efficient? Type of indicator Indicator Built bridges … … Baseline value Target value Achieved value

Definitions • Relevance : the project outcomes and Impact CAN produce a change • External Coherence: the project is strategically aligned with the concerning policy (vertical) – the program works in synergy and complementarily with other contextual intervention (horizontal) • Internal coherence: the project objectives, activities, output, outcomes, impacts are logically connected • Performance: the project activities are delivered on time, the outputs respect the targets, the resources are duly absorbed, the procedures are done according to the rules. • Effectiveness: the project achieves outcomes / Impact accordingly with the targets • Efficiency: the project achieves outcomes / Impact accordingly with the targets with the minor costs and in the shorter time. • Impact: the project is capable to contribute significantly to the change • Sustainability: the project changes can last after the conclusion • Utility: Impacts obtained by the intervention correspond to society needs