Lesson 7 Effectiveness and efficiency Macerata 1 st

  • Slides: 74
Download presentation
Lesson 7: Effectiveness and efficiency Macerata, 1 st April Andrea Gramillano, t 33 srl

Lesson 7: Effectiveness and efficiency Macerata, 1 st April Andrea Gramillano, t 33 srl

Agenda v. What does Effectiveness mean? v. What does Efficiency mean? v Cost /

Agenda v. What does Effectiveness mean? v. What does Efficiency mean? v Cost / effectiveness analysis v. CBA analysis

Efficiency, Impact and sustainability in the evalaution Process Project Result 1 Result 2 Result

Efficiency, Impact and sustainability in the evalaution Process Project Result 1 Result 2 Result 3 Efficiency At what cost? Effective ness: which is the change?

Example Effectiveness: Does the new project (road) produce the positive result (objective) of a

Example Effectiveness: Does the new project (road) produce the positive result (objective) of a better accessibility? Yes, now 10. 000 people more can get the island saving 2 hours and 3 millions of euros Effectiveness: Are the two regions directly connected? Yes Efficiency: was it the cheapest solution? No a railway would have been cheaper

Effectiveness (definition 1) The extent to which the development intervention’s objectives were achieved, or

Effectiveness (definition 1) The extent to which the development intervention’s objectives were achieved, or are expected to be achieved, taking into account their relative importance. Note: Also used as an aggregate measure of (or judgement about) the merit or worth of an activity, i. e. the extent to which an intervention has attained, or is expected to attain, its major relevant objectives efficiently in a sustainable fashion and with a positive institutional developmental impact. (OECD)

Effectiveness (definition 2) The term effectiveness has many possible meanings. The most common definition

Effectiveness (definition 2) The term effectiveness has many possible meanings. The most common definition identifies effectiveness with “achievement of objectives”. This leaves open the definition to the different meanings of “objectives”. Objectives can be expressed quantitatively in terms of expected output or results. The effectiveness is evaluated simply by comparing what has been obtained with what had been planned: outputs and results indicators are all is needed. (European Commission – DG REGIO EVALSED GUIDE)

EFFECTIVENESS: WHAT TO EVALUATE ? EUROPEAN COMMISSION Quality: effectiveness is evaluated by comparing results

EFFECTIVENESS: WHAT TO EVALUATE ? EUROPEAN COMMISSION Quality: effectiveness is evaluated by comparing results with quality standards. OECD Effectiveness assesses whether the results outlined in the logframe are delivered and if they are likely to produce the expected objective. Ability of a given action to produce Evaluating effectiveness should a desired change: comparing include assessment of how what is observed after the action people (women and men) benefit has taken place with what would from the results brought by the have happened without the action. project One needs data that allow recovery of the counterfactual situation.

Evaluation Questions • To what extent were the originally defined objectives of the development

Evaluation Questions • To what extent were the originally defined objectives of the development intervention realistic? • To what extent have the (direct) objectives of the development intervention been achieved in accordance with the (adjusted, if applicable) target populations? • What are the (concrete) contributions of interventions for achieving the objectives of the development intervention? • What factors were crucial for the achievement or failure to achieve the project objectives so far (indication of strengths and weaknesses, e. g. the monitoring and evaluation system)? • What is the quality of development-policy, technical planning and coordination ?

Efficiency Definition Type of Questions • OECD: the project results 1. Was the budget

Efficiency Definition Type of Questions • OECD: the project results 1. Was the budget adequate? Was the have been achieved at spending commensurate with the reasonable cost with delivery of activities and achieving minimum waste of effort, time, money and skills. results? • European Commission: 2. Were the human, financial, material obtaining a given output resources adequate in terms of at the minimum cost or, quality and quantity to achieve the equivalently, with project results? maximizing output for a given level of resources. 3. To what extent were costs of the project justified by the benefits in comparison with similar projects or known alternative approaches?

Information and Data on OUTCOME Indicators Methods Quantitative - Financial - Physical - Procedural

Information and Data on OUTCOME Indicators Methods Quantitative - Financial - Physical - Procedural Qualitative - Opinions on the level of achievement - Perceptions on satisfaction • • Literature review Interview Community interview Project visit Focus group Case study Survey

Different way for collecting information

Different way for collecting information

THE ANALYSIS Comparing cost and outcome • Cost - effectiveness Comparing perception • Customer

THE ANALYSIS Comparing cost and outcome • Cost - effectiveness Comparing perception • Customer satisfaction Comparing Cost and Benefit • Cost and Benefit

Cost - effectiveness Project A Project B Public contribution to each firm / project

Cost - effectiveness Project A Project B Public contribution to each firm / project 100, 000 Euro 155, 000 Euro Number of jobs created in each firm 2 FTE 3 FTE Ratio 50000 51666

Cost - effectiveness Project A Project B Public contribution to each firm / project

Cost - effectiveness Project A Project B Public contribution to each firm / project 100, 000 Euro 155, 000 Euro Number of jobs created in each firm 2 FTE 3 FTE Ratio 50000 51666

Cost - effectiveness Project A Project B Public contribution to each firm / project

Cost - effectiveness Project A Project B Public contribution to each firm / project 100, 000 Euro 155, 000 Euro Number of jobs created in each firm 2 FTE 3 FTE Ratio 50000 51666

The Customer Satisfaction Customer satisfaction is a technique coming from Marketing. It aims to

The Customer Satisfaction Customer satisfaction is a technique coming from Marketing. It aims to understand how customer expectations are met by products and services. More in detail, CS is based on the comparison of customers’ expectations and their perceived performance. It is positive when a product performs over the expectations. CS is measured at the individual level, but it is almost always reported at an aggregate level and it is often measured along various dimensions. There are different methods adopted in business. One of the most famous is the SERVQUAL by Parasuraman, Zeithaml and Berry.

IDENTIFY GAPS • GAP: between service quality specification and service delivery This gap may

IDENTIFY GAPS • GAP: between service quality specification and service delivery This gap may arise through service personnel being poorly trained, incapable or unwilling to meet the set service standard. • GAP: between service delivery and external communication Consumer expectations are highly influenced by statements made by company representatives and advertisements. • GAP: between expected service and experienced service This gap arises when the consumer misinterprets the service quality.

STEPS TO DEVELOP CUSTOMER SATISFACTION Establish Dimensions Analyse results Define Questions Deliver Survey Fix

STEPS TO DEVELOP CUSTOMER SATISFACTION Establish Dimensions Analyse results Define Questions Deliver Survey Fix Scales Set questionnaire

DIMENSION AND QUESTIONS DIMENSIONS • Dimensions represent different aspects of the services and products

DIMENSION AND QUESTIONS DIMENSIONS • Dimensions represent different aspects of the services and products SERVQUAL model, refers to five dimensions of quality: q Reliability q Responsiveness q Assurance q Empathy q Tangibles QUESTIONS They represent the concrete translation of the dimensions in items to be asked to the customers. ES. A project related to vocational training defines the content of training, the methods and the materials. It might ask to students “how are you satisfied with the on line available documents? ”

FACTORS of the RATER model • Reliability: the ability to perform the promised service

FACTORS of the RATER model • Reliability: the ability to perform the promised service dependably and accurately • Assurance: the knowledge and courtesy of employees and their ability to convey trust and confidence • Tangibles: the appearance of physical facilities, equipment, personnel and communication materials • Empathy: the provision of caring, individualized attention to customers • Responsiveness: the willingness to help customers and to provide prompt service

Demographic information Responses (based on a Likert scale) Age / Year (Master degree –

Demographic information Responses (based on a Likert scale) Age / Year (Master degree – Bachelor degree) Employment situation (yes / no) City and County (living in Macerata or not) Frequency of the service (Attendance…)

Demographic information Responses (based on a Likert scale) Age / Year (Master degree –

Demographic information Responses (based on a Likert scale) Age / Year (Master degree – Bachelor degree) Employment situation (yes / no) City and County (living in Macerata or not) Frequency of the service (Attendance…) FACTORS of the RATER model • Reliability: the ability to perform the promised service dependably and accurately QUALITY OF THE SERVICE (LESSONS/CONTENTS) • Assurance: the knowledge and courtesy of employees and their ability to convey trust and confidence QUALITY OF THE SERVICE (LESSONS/PROFESSOR-QUALITY OF EXPLANATION) • Tangibles: the appearance of physical facilities, equipment, personnel and communication materials QUALITY OF THE ROOM / TIMING / ACCESSIBILITY • Empathy: the provision of caring, individualized attention to customers AVAILABILITY FOR INDIVIDUALISED SUPPORT • Responsiveness: the willingness to help customers and to provide prompt service AVAILABILITY FOR “HELP DESK” • …………. OPEN QUESTION FOR SUGGESTIONS

SCALE (LIKERT) and QUESTIONAIRE

SCALE (LIKERT) and QUESTIONAIRE

SURVEY and ANALYSIS Survey • DEFINE YOUR TARGET • SAMPLE ? • tools: q

SURVEY and ANALYSIS Survey • DEFINE YOUR TARGET • SAMPLE ? • tools: q PHONE q WEB q PERSONAL INTERVIEW ANALYSIS (descriptive statistics) Questions 1 … TOT. Resp. score Mode Exp.

WORKOUT: HOW DO YOU LIKE THIS LESSON? From dimensions to questions STEPS Establish Dimensions

WORKOUT: HOW DO YOU LIKE THIS LESSON? From dimensions to questions STEPS Establish Dimensions Analyse results Define Questions Deliver Survey Fix Scales Set questionnaire Dimensions Questions

Questions Very bad (1) QUESTIONAIRE 1) Are the materials / contents relevant / useful

Questions Very bad (1) QUESTIONAIRE 1) Are the materials / contents relevant / useful for your professional preparation? 2) Do you think that materials and contents provided during the lessons are adequate to prepare for the exam? 3) Did the course respect the predefined programme? 4) Was the professor available for providing additional information, when needed? 5) Are contents exposed in a clear way during the lessons ? 6)Is the classroom (furniture, room, computers …. ) adequate for the course? 7) Is the lesson schedule adequate compared to the entire students’ week timetable? 8) Is the professor answering the questions in an effective way? 9) Is the distribution of the hours dedicated to the subjects adequate? 10) Does the course fit your degree? 11) Was the professor able to attract the students’ attention? 12) Was the professor ready to provide supports and guidance after the class? 13) Did the professor respect the timetable (he was there for the lessons)? 14) Did the course meet your expectations? IF not please provide any suggestion… More bad than good (2) More good than bad (3) Very good (4)

Questions 1 2 3 4

Questions 1 2 3 4

ANALYSIS 1 2 3 4

ANALYSIS 1 2 3 4

Lesson 8: Efficiency, Impact and sustainibility Macerata, 8 th April Andrea Gramillano, t 33

Lesson 8: Efficiency, Impact and sustainibility Macerata, 8 th April Andrea Gramillano, t 33 srl

Agenda v. What do Impact and sustainability mean? v. What do you mean by

Agenda v. What do Impact and sustainability mean? v. What do you mean by counterfactual analysis? v. CBA analysis: § Financial Aanalysis §Economic Analysis §Risk Analysis

Efficiency, Impact and sustainability in the evaluation Process Effectiveness: which result? Project Result 1

Efficiency, Impact and sustainability in the evaluation Process Effectiveness: which result? Project Result 1 Result 2 Result 3 Efficiency At what cost? Sustainibility: Will they last? Impact: Are they directly related?

Sustainability Definition Type of Questions • OECD: Will the • 1. Are the results

Sustainability Definition Type of Questions • OECD: Will the • 1. Are the results likely to be political, institutional, sustained in the future? managerial, and Did national/regional/local technical, change last • 2. authorities, civil society take beyond the initiative? ownership of the results of the project? • 3. Are changes in attitudes, behaviour patterns and institutional arrangements likely to last once the project is closed?

Impact Definition Type of Questions • OECD: Did the 1. To what extent was

Impact Definition Type of Questions • OECD: Did the 1. To what extent was the project bring real objective achieved? How did the change ? results influence the indirect beneficiaries? • European Commission: The 2. Did the project produce unintended change that can be effects? Why did these unintended credibly attributed to results (positive or negative) an intervention. happen? 3. What was the influence of external factors (negative, positive, little/no influence) such as changes in policy legislation, general economic and financial conditions?

COUNTERFACTUAL APPROACH

COUNTERFACTUAL APPROACH

Example Effectiveness: Does the new project (road) produce the positive result (objective) of a

Example Effectiveness: Does the new project (road) produce the positive result (objective) of a better accessibility? Yes, now 10. 000 people more can get the island saving 2 hours and 3 millions of euros Efficiency: was it the cheapest solution? No a ferry boat would have been cheaper Sustainability: will the result last? Yes Impact: Are the effect directly connected? Yes

Outputs and results and impact: principle of the pie / apple (pie) External Factors

Outputs and results and impact: principle of the pie / apple (pie) External Factors Needs CHANGE (results) cto a f l a xtern E Outputs rs ion t u b i r Cont cts je of pro

COST AND BENEFIT ANALYSIS

COST AND BENEFIT ANALYSIS

FRAMEWORK Key concepts STEPS Sustainability Analysis Financial Analysis Economic Analysis Risk Analysis • •

FRAMEWORK Key concepts STEPS Sustainability Analysis Financial Analysis Economic Analysis Risk Analysis • • Economic vs. Financial Discounting Externalities Shadow prices ( conversion factor) • Conditionality • Time Horizon

DISCOUNTING Discount rate: The rate at which future values are discounted to the present.

DISCOUNTING Discount rate: The rate at which future values are discounted to the present. Usually considering roughly equal to the opportunity cost of capital.

DISCOUNTING Example: 1 Euro invested at 5% yearly rate, will become (1+5%)=1, 05 after

DISCOUNTING Example: 1 Euro invested at 5% yearly rate, will become (1+5%)=1, 05 after one year; (1, 05)x(1, 05)=1, 1025 after two years; (1, 05)x(1, 05)=1, 157625 after three years. The economic present value of 1 Euro that will be spent or gained two years later is 1/1, 1025=0, 907029; three years later is 1/1, 157625=0, 863838.

DISCOUNTING /Most Member States have their own social discount rates for public sector projects.

DISCOUNTING /Most Member States have their own social discount rates for public sector projects. /These discount rates comprise between a maximunm 10% rate and a minimum 3% real rate. /In this context a real 5% discount rate may be an appropriate benchmark value: low enough to focus attention on project not passing the test, but not too far from the average official discounting rate.

5 FINANCIAL ANALYSIS • The financial analysis is made up of a series of

5 FINANCIAL ANALYSIS • The financial analysis is made up of a series of tables that collect the financial flows of the investment: - total investment, - operating costs and revenue, - sources of financing, • The financial analysis should finally result in two tables summarising the cash flows to verify: – SUSTAINABILITY – RETURN OF INVESTIMENT

FINANCIAL SUSTAINABILITY TABLE Financial sustainability is verified if this row is more than or

FINANCIAL SUSTAINABILITY TABLE Financial sustainability is verified if this row is more than or equal to zero for all the years considered.

FINANCIAL INTERNAL RATE OF RETURN OF INVESTMENT (FRR/C) This value is calculated with a

FINANCIAL INTERNAL RATE OF RETURN OF INVESTMENT (FRR/C) This value is calculated with a financial discount rate of 6% This rate of return measures the capacity of operating net revenues to sustain the investment costs regardless the way they are financed.

Net present value Rate of return

Net present value Rate of return

ECONOMIC ANALYSIS “The economic analysis appraises the project contribution to the economic welfare of

ECONOMIC ANALYSIS “The economic analysis appraises the project contribution to the economic welfare of the region or country. It is made on behalf of the whole society (region or country) instead of just the owner of the infrastructure like in the financial analysis. ”

The three steps for economic corrections STEP 1 - Fiscal corrections STEP 2 Externalities

The three steps for economic corrections STEP 1 - Fiscal corrections STEP 2 Externalities corrections STEP 3 - From market to accounting price

(1). Phase 1. Fiscal correction. It is necessary to deduct from the flows of

(1). Phase 1. Fiscal correction. It is necessary to deduct from the flows of financial analysis, payments that have no real resources counterpart, as for subsidies and indirect taxes on input and output. For direct public tranfers they are already not included in the starting table for financial analysis which considers investment costs and not financial resources.

(2). Phase 2. Externalities correction. It is necessary to include among outflows and inflows

(2). Phase 2. Externalities correction. It is necessary to include among outflows and inflows also external costs and benefits for which there is no cash flow. Some examples could be costs for health services or losses in fisheries due to incresed pollution, time saved for investment in transports, specific infrastructures supplied by public sector for the project (a road built specifically for the project…) increased touristic flows, increased accessibility of the region…

(3). Phase 3. From market to accounting prices. It is necessary to determine a

(3). Phase 3. From market to accounting prices. It is necessary to determine a vector of conversion factors including distortion of the market into prices. Such distortions affect input, outputs and wages.

STEP 1: fiscal corrections This value is calculated with a social discount rate of

STEP 1: fiscal corrections This value is calculated with a social discount rate of 5% STEP 3: From market to accounting prices STEP 2: Externalities corrections

8 SENSITIVITY AND RISK 4 The project examiner should check if risks associated with

8 SENSITIVITY AND RISK 4 The project examiner should check if risks associated with the project have been assessed by the applicant. 4 This check can be done by sensitivity analysis alone, and implies a certain amount of reasoning, if not of calculations, in terms of probabilities of the main variables affecting the expected outcome of the project.

8 SENSITIVITY AND RISK 4 Sensitivity analysis: A study of the impact that changes

8 SENSITIVITY AND RISK 4 Sensitivity analysis: A study of the impact that changes in crucial estimates concerning costs and benefits would have on the profitability or present value of a project 4 Probability distribution: One is often interested to know what the probability is of a project having a economic rate of return less than the minimum required rate.

8 SENSITIVITY AND RISK Sensitivity analysis is a deterministic procedure, aimed at understanding which

8 SENSITIVITY AND RISK Sensitivity analysis is a deterministic procedure, aimed at understanding which variables influence the economic net present value or ERR of the project. Any independent variable for which 1% change results in 1% change or more of ENPV or ERR is certainly a critical one, and needs to be estimated as reliably as possible.

8 SENSITIVITY AND RISK Risk assessment implies assigning probabilities to these critical variables and

8 SENSITIVITY AND RISK Risk assessment implies assigning probabilities to these critical variables and calculating the probability distribution of ERR.

SUM UP

SUM UP

PROGRAM AND PROJECT The Project / Program cycle The Chain of Public Interventions Identification

PROGRAM AND PROJECT The Project / Program cycle The Chain of Public Interventions Identification Evaluation and review Policy Formulation Implementation Program Project (1) Other delivery policy tools: law, tax, agency, campaign. Project (2)

Why do we need a project? To make a change… The project is ….

Why do we need a project? To make a change… The project is …. . • Project vs. routine • Project is limited in time • Resources are defined • Beneficiary identified To… Solve a problem / social need / change a behaviour: - Economic growth - Accessibility - Social inclusion …. .

Monitoring • To monitor means to observe. • Monitoring is the regular observation and

Monitoring • To monitor means to observe. • Monitoring is the regular observation and recording of activities taking place in a project or program. It is a process of routinely gathering information on all aspects of the project. • To monitor is to check on how project activities are progressing. It is observation; ─ systematic and purposeful observation. • Monitoring also involves giving feedback about the progress of the project to the donors, implementers and beneficiaries of the project. • Reporting enables gathered information to be used in making decisions for improving the project performance.

Evaluation Based on monitoring, evaluation is the systematic collection and analysis of data needed

Evaluation Based on monitoring, evaluation is the systematic collection and analysis of data needed to make decisions, a process in which most well-run programs engage from the outset ( American Evaluation Society) Evaluation tries to answer two distinctive questions: • Did the public intervention have an effect at all and if yes, how big – positive or negative – was this effect. The question is: Does it work? Is there a causal link? This is the counterfactual question • Why an intervention produces intended (and unintended) effects. The goal is to answer the “why and how it works? ” question. To answer this question is the aim of theory-based impact (European Commission – DG REGIO) An evaluation is an assessment, as systematic and objective as possible, of an on -going or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, developmental efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of both recipients and donors ( OECD/DAC)

Monitoring, Auditing and Evalaution Monitoring: observing and collecting data form the project. What time

Monitoring, Auditing and Evalaution Monitoring: observing and collecting data form the project. What time is it? 12, 34 At what time is the train ? 11, 50 Evaluation: interpeting data to provide a subjuctive judgement for accountability and learning: • You did not catch the train because of…. . • An alternative can be……. .

Project development SELECTION Implementation Needs Identification and Formualtion Resources (inputs) END OF Realisation (output)

Project development SELECTION Implementation Needs Identification and Formualtion Resources (inputs) END OF Realisation (output) PROJECT Result / outcome After Conlcusion

Project development and Evaluation Needs On going Evaluation Ex ante Evaluation Resources (inputs) Realisation

Project development and Evaluation Needs On going Evaluation Ex ante Evaluation Resources (inputs) Realisation (output) Result / outcome Monitoring Ex Post Evaluation

Relevance and Coherence ( EX ANTE) Relevance ce Program / policy Needs Resources (inputs)

Relevance and Coherence ( EX ANTE) Relevance ce Program / policy Needs Resources (inputs) Output (implementation) Result (Specific Objective) External coherence Internal Coherence

Performance, Efficiency, Sustainibility, Effectiveness (on going / ex post) Needs Performance ce Resources (inputs)

Performance, Efficiency, Sustainibility, Effectiveness (on going / ex post) Needs Performance ce Resources (inputs) Output (implementation) Effectiveness ce Result (Specific Objective) Efficiencyce Sustainibility

Definitions • Relevance : the project results and Impact CAN produce a change •

Definitions • Relevance : the project results and Impact CAN produce a change • External Coherence: the project is strategically aligned with the concerning policy (vertical) – the program works in synergy and complementarily with other contextual intervention (horizontal) • Internal coherence: the project objectives, activities, output, results, impacts are logically connected • Performance: the project activities are delivered on time, the outputs respect the targets, the resources are duly absorbed, the procedures are done according to the rules. • Effectiveness: the project achieves results / Impact accordingly with the targets • Efficiency: the project achieves results / Impact accordingly with the targets with the minor costs and in the shorter time. • Impact: the project is capable to contribute significantly to the change • Sustainability: the project changes can last after the conclusion • Utility: Impacts obtained by the intervention correspond to society needs

TOOL : FOCUS GROUP

TOOL : FOCUS GROUP

Focus group Definiton A focus group is a form of qualitative research in which

Focus group Definiton A focus group is a form of qualitative research in which a group of people are asked about their perceptions, opinions, beliefs, and attitudes towards a product, service, concept, advertisement, idea, or packaging. Questions are asked in an interactive group setting where participants are free to talk with other group members. Facilitation tools • Concept mapping • Metaplan ® • Dual Group • Moderator

Performing a Focus group Cabaret setting presentation Brain storming / break question Wrap up

Performing a Focus group Cabaret setting presentation Brain storming / break question Wrap up Structured discussion

RISK AND CHALLENGES ONE PREDOMINANT VOICE POLARIZATION AND AMBIVALENCE BIZARRE TOOLS AND OBVIOUSE OUTCOME

RISK AND CHALLENGES ONE PREDOMINANT VOICE POLARIZATION AND AMBIVALENCE BIZARRE TOOLS AND OBVIOUSE OUTCOME

SWOT Relevance Internal coherence External Coherence Performance Efficiency Effectiveness Impact and Utility Sustainibility ASSESSMENT

SWOT Relevance Internal coherence External Coherence Performance Efficiency Effectiveness Impact and Utility Sustainibility ASSESSMENT MATRIX CUSTUMER SATISFACTION MUTICRITERIA ACB

QUESTIONS FOR FOCUS GROUP RELEVANCE EFFICIENCY EFFECTIVENESS SUSTAINIBILITY BEST TOPIC

QUESTIONS FOR FOCUS GROUP RELEVANCE EFFICIENCY EFFECTIVENESS SUSTAINIBILITY BEST TOPIC

See you www. t 33. it a. gramillano@t 33. it

See you www. t 33. it a. gramillano@t 33. it