Lesson 7 Effectiveness and efficiency Macerata 1 st
- Slides: 74
Lesson 7: Effectiveness and efficiency Macerata, 1 st April Andrea Gramillano, t 33 srl
Agenda v. What does Effectiveness mean? v. What does Efficiency mean? v Cost / effectiveness analysis v. CBA analysis
Efficiency, Impact and sustainability in the evalaution Process Project Result 1 Result 2 Result 3 Efficiency At what cost? Effective ness: which is the change?
Example Effectiveness: Does the new project (road) produce the positive result (objective) of a better accessibility? Yes, now 10. 000 people more can get the island saving 2 hours and 3 millions of euros Effectiveness: Are the two regions directly connected? Yes Efficiency: was it the cheapest solution? No a railway would have been cheaper
Effectiveness (definition 1) The extent to which the development intervention’s objectives were achieved, or are expected to be achieved, taking into account their relative importance. Note: Also used as an aggregate measure of (or judgement about) the merit or worth of an activity, i. e. the extent to which an intervention has attained, or is expected to attain, its major relevant objectives efficiently in a sustainable fashion and with a positive institutional developmental impact. (OECD)
Effectiveness (definition 2) The term effectiveness has many possible meanings. The most common definition identifies effectiveness with “achievement of objectives”. This leaves open the definition to the different meanings of “objectives”. Objectives can be expressed quantitatively in terms of expected output or results. The effectiveness is evaluated simply by comparing what has been obtained with what had been planned: outputs and results indicators are all is needed. (European Commission – DG REGIO EVALSED GUIDE)
EFFECTIVENESS: WHAT TO EVALUATE ? EUROPEAN COMMISSION Quality: effectiveness is evaluated by comparing results with quality standards. OECD Effectiveness assesses whether the results outlined in the logframe are delivered and if they are likely to produce the expected objective. Ability of a given action to produce Evaluating effectiveness should a desired change: comparing include assessment of how what is observed after the action people (women and men) benefit has taken place with what would from the results brought by the have happened without the action. project One needs data that allow recovery of the counterfactual situation.
Evaluation Questions • To what extent were the originally defined objectives of the development intervention realistic? • To what extent have the (direct) objectives of the development intervention been achieved in accordance with the (adjusted, if applicable) target populations? • What are the (concrete) contributions of interventions for achieving the objectives of the development intervention? • What factors were crucial for the achievement or failure to achieve the project objectives so far (indication of strengths and weaknesses, e. g. the monitoring and evaluation system)? • What is the quality of development-policy, technical planning and coordination ?
Efficiency Definition Type of Questions • OECD: the project results 1. Was the budget adequate? Was the have been achieved at spending commensurate with the reasonable cost with delivery of activities and achieving minimum waste of effort, time, money and skills. results? • European Commission: 2. Were the human, financial, material obtaining a given output resources adequate in terms of at the minimum cost or, quality and quantity to achieve the equivalently, with project results? maximizing output for a given level of resources. 3. To what extent were costs of the project justified by the benefits in comparison with similar projects or known alternative approaches?
Information and Data on OUTCOME Indicators Methods Quantitative - Financial - Physical - Procedural Qualitative - Opinions on the level of achievement - Perceptions on satisfaction • • Literature review Interview Community interview Project visit Focus group Case study Survey
Different way for collecting information
THE ANALYSIS Comparing cost and outcome • Cost - effectiveness Comparing perception • Customer satisfaction Comparing Cost and Benefit • Cost and Benefit
Cost - effectiveness Project A Project B Public contribution to each firm / project 100, 000 Euro 155, 000 Euro Number of jobs created in each firm 2 FTE 3 FTE Ratio 50000 51666
Cost - effectiveness Project A Project B Public contribution to each firm / project 100, 000 Euro 155, 000 Euro Number of jobs created in each firm 2 FTE 3 FTE Ratio 50000 51666
Cost - effectiveness Project A Project B Public contribution to each firm / project 100, 000 Euro 155, 000 Euro Number of jobs created in each firm 2 FTE 3 FTE Ratio 50000 51666
The Customer Satisfaction Customer satisfaction is a technique coming from Marketing. It aims to understand how customer expectations are met by products and services. More in detail, CS is based on the comparison of customers’ expectations and their perceived performance. It is positive when a product performs over the expectations. CS is measured at the individual level, but it is almost always reported at an aggregate level and it is often measured along various dimensions. There are different methods adopted in business. One of the most famous is the SERVQUAL by Parasuraman, Zeithaml and Berry.
IDENTIFY GAPS • GAP: between service quality specification and service delivery This gap may arise through service personnel being poorly trained, incapable or unwilling to meet the set service standard. • GAP: between service delivery and external communication Consumer expectations are highly influenced by statements made by company representatives and advertisements. • GAP: between expected service and experienced service This gap arises when the consumer misinterprets the service quality.
STEPS TO DEVELOP CUSTOMER SATISFACTION Establish Dimensions Analyse results Define Questions Deliver Survey Fix Scales Set questionnaire
DIMENSION AND QUESTIONS DIMENSIONS • Dimensions represent different aspects of the services and products SERVQUAL model, refers to five dimensions of quality: q Reliability q Responsiveness q Assurance q Empathy q Tangibles QUESTIONS They represent the concrete translation of the dimensions in items to be asked to the customers. ES. A project related to vocational training defines the content of training, the methods and the materials. It might ask to students “how are you satisfied with the on line available documents? ”
FACTORS of the RATER model • Reliability: the ability to perform the promised service dependably and accurately • Assurance: the knowledge and courtesy of employees and their ability to convey trust and confidence • Tangibles: the appearance of physical facilities, equipment, personnel and communication materials • Empathy: the provision of caring, individualized attention to customers • Responsiveness: the willingness to help customers and to provide prompt service
Demographic information Responses (based on a Likert scale) Age / Year (Master degree – Bachelor degree) Employment situation (yes / no) City and County (living in Macerata or not) Frequency of the service (Attendance…)
Demographic information Responses (based on a Likert scale) Age / Year (Master degree – Bachelor degree) Employment situation (yes / no) City and County (living in Macerata or not) Frequency of the service (Attendance…) FACTORS of the RATER model • Reliability: the ability to perform the promised service dependably and accurately QUALITY OF THE SERVICE (LESSONS/CONTENTS) • Assurance: the knowledge and courtesy of employees and their ability to convey trust and confidence QUALITY OF THE SERVICE (LESSONS/PROFESSOR-QUALITY OF EXPLANATION) • Tangibles: the appearance of physical facilities, equipment, personnel and communication materials QUALITY OF THE ROOM / TIMING / ACCESSIBILITY • Empathy: the provision of caring, individualized attention to customers AVAILABILITY FOR INDIVIDUALISED SUPPORT • Responsiveness: the willingness to help customers and to provide prompt service AVAILABILITY FOR “HELP DESK” • …………. OPEN QUESTION FOR SUGGESTIONS
SCALE (LIKERT) and QUESTIONAIRE
SURVEY and ANALYSIS Survey • DEFINE YOUR TARGET • SAMPLE ? • tools: q PHONE q WEB q PERSONAL INTERVIEW ANALYSIS (descriptive statistics) Questions 1 … TOT. Resp. score Mode Exp.
WORKOUT: HOW DO YOU LIKE THIS LESSON? From dimensions to questions STEPS Establish Dimensions Analyse results Define Questions Deliver Survey Fix Scales Set questionnaire Dimensions Questions
Questions Very bad (1) QUESTIONAIRE 1) Are the materials / contents relevant / useful for your professional preparation? 2) Do you think that materials and contents provided during the lessons are adequate to prepare for the exam? 3) Did the course respect the predefined programme? 4) Was the professor available for providing additional information, when needed? 5) Are contents exposed in a clear way during the lessons ? 6)Is the classroom (furniture, room, computers …. ) adequate for the course? 7) Is the lesson schedule adequate compared to the entire students’ week timetable? 8) Is the professor answering the questions in an effective way? 9) Is the distribution of the hours dedicated to the subjects adequate? 10) Does the course fit your degree? 11) Was the professor able to attract the students’ attention? 12) Was the professor ready to provide supports and guidance after the class? 13) Did the professor respect the timetable (he was there for the lessons)? 14) Did the course meet your expectations? IF not please provide any suggestion… More bad than good (2) More good than bad (3) Very good (4)
Questions 1 2 3 4
ANALYSIS 1 2 3 4
Lesson 8: Efficiency, Impact and sustainibility Macerata, 8 th April Andrea Gramillano, t 33 srl
Agenda v. What do Impact and sustainability mean? v. What do you mean by counterfactual analysis? v. CBA analysis: § Financial Aanalysis §Economic Analysis §Risk Analysis
Efficiency, Impact and sustainability in the evaluation Process Effectiveness: which result? Project Result 1 Result 2 Result 3 Efficiency At what cost? Sustainibility: Will they last? Impact: Are they directly related?
Sustainability Definition Type of Questions • OECD: Will the • 1. Are the results likely to be political, institutional, sustained in the future? managerial, and Did national/regional/local technical, change last • 2. authorities, civil society take beyond the initiative? ownership of the results of the project? • 3. Are changes in attitudes, behaviour patterns and institutional arrangements likely to last once the project is closed?
Impact Definition Type of Questions • OECD: Did the 1. To what extent was the project bring real objective achieved? How did the change ? results influence the indirect beneficiaries? • European Commission: The 2. Did the project produce unintended change that can be effects? Why did these unintended credibly attributed to results (positive or negative) an intervention. happen? 3. What was the influence of external factors (negative, positive, little/no influence) such as changes in policy legislation, general economic and financial conditions?
COUNTERFACTUAL APPROACH
Example Effectiveness: Does the new project (road) produce the positive result (objective) of a better accessibility? Yes, now 10. 000 people more can get the island saving 2 hours and 3 millions of euros Efficiency: was it the cheapest solution? No a ferry boat would have been cheaper Sustainability: will the result last? Yes Impact: Are the effect directly connected? Yes
Outputs and results and impact: principle of the pie / apple (pie) External Factors Needs CHANGE (results) cto a f l a xtern E Outputs rs ion t u b i r Cont cts je of pro
COST AND BENEFIT ANALYSIS
FRAMEWORK Key concepts STEPS Sustainability Analysis Financial Analysis Economic Analysis Risk Analysis • • Economic vs. Financial Discounting Externalities Shadow prices ( conversion factor) • Conditionality • Time Horizon
DISCOUNTING Discount rate: The rate at which future values are discounted to the present. Usually considering roughly equal to the opportunity cost of capital.
DISCOUNTING Example: 1 Euro invested at 5% yearly rate, will become (1+5%)=1, 05 after one year; (1, 05)x(1, 05)=1, 1025 after two years; (1, 05)x(1, 05)=1, 157625 after three years. The economic present value of 1 Euro that will be spent or gained two years later is 1/1, 1025=0, 907029; three years later is 1/1, 157625=0, 863838.
DISCOUNTING /Most Member States have their own social discount rates for public sector projects. /These discount rates comprise between a maximunm 10% rate and a minimum 3% real rate. /In this context a real 5% discount rate may be an appropriate benchmark value: low enough to focus attention on project not passing the test, but not too far from the average official discounting rate.
5 FINANCIAL ANALYSIS • The financial analysis is made up of a series of tables that collect the financial flows of the investment: - total investment, - operating costs and revenue, - sources of financing, • The financial analysis should finally result in two tables summarising the cash flows to verify: – SUSTAINABILITY – RETURN OF INVESTIMENT
FINANCIAL SUSTAINABILITY TABLE Financial sustainability is verified if this row is more than or equal to zero for all the years considered.
FINANCIAL INTERNAL RATE OF RETURN OF INVESTMENT (FRR/C) This value is calculated with a financial discount rate of 6% This rate of return measures the capacity of operating net revenues to sustain the investment costs regardless the way they are financed.
Net present value Rate of return
ECONOMIC ANALYSIS “The economic analysis appraises the project contribution to the economic welfare of the region or country. It is made on behalf of the whole society (region or country) instead of just the owner of the infrastructure like in the financial analysis. ”
The three steps for economic corrections STEP 1 - Fiscal corrections STEP 2 Externalities corrections STEP 3 - From market to accounting price
(1). Phase 1. Fiscal correction. It is necessary to deduct from the flows of financial analysis, payments that have no real resources counterpart, as for subsidies and indirect taxes on input and output. For direct public tranfers they are already not included in the starting table for financial analysis which considers investment costs and not financial resources.
(2). Phase 2. Externalities correction. It is necessary to include among outflows and inflows also external costs and benefits for which there is no cash flow. Some examples could be costs for health services or losses in fisheries due to incresed pollution, time saved for investment in transports, specific infrastructures supplied by public sector for the project (a road built specifically for the project…) increased touristic flows, increased accessibility of the region…
(3). Phase 3. From market to accounting prices. It is necessary to determine a vector of conversion factors including distortion of the market into prices. Such distortions affect input, outputs and wages.
STEP 1: fiscal corrections This value is calculated with a social discount rate of 5% STEP 3: From market to accounting prices STEP 2: Externalities corrections
8 SENSITIVITY AND RISK 4 The project examiner should check if risks associated with the project have been assessed by the applicant. 4 This check can be done by sensitivity analysis alone, and implies a certain amount of reasoning, if not of calculations, in terms of probabilities of the main variables affecting the expected outcome of the project.
8 SENSITIVITY AND RISK 4 Sensitivity analysis: A study of the impact that changes in crucial estimates concerning costs and benefits would have on the profitability or present value of a project 4 Probability distribution: One is often interested to know what the probability is of a project having a economic rate of return less than the minimum required rate.
8 SENSITIVITY AND RISK Sensitivity analysis is a deterministic procedure, aimed at understanding which variables influence the economic net present value or ERR of the project. Any independent variable for which 1% change results in 1% change or more of ENPV or ERR is certainly a critical one, and needs to be estimated as reliably as possible.
8 SENSITIVITY AND RISK Risk assessment implies assigning probabilities to these critical variables and calculating the probability distribution of ERR.
SUM UP
PROGRAM AND PROJECT The Project / Program cycle The Chain of Public Interventions Identification Evaluation and review Policy Formulation Implementation Program Project (1) Other delivery policy tools: law, tax, agency, campaign. Project (2)
Why do we need a project? To make a change… The project is …. . • Project vs. routine • Project is limited in time • Resources are defined • Beneficiary identified To… Solve a problem / social need / change a behaviour: - Economic growth - Accessibility - Social inclusion …. .
Monitoring • To monitor means to observe. • Monitoring is the regular observation and recording of activities taking place in a project or program. It is a process of routinely gathering information on all aspects of the project. • To monitor is to check on how project activities are progressing. It is observation; ─ systematic and purposeful observation. • Monitoring also involves giving feedback about the progress of the project to the donors, implementers and beneficiaries of the project. • Reporting enables gathered information to be used in making decisions for improving the project performance.
Evaluation Based on monitoring, evaluation is the systematic collection and analysis of data needed to make decisions, a process in which most well-run programs engage from the outset ( American Evaluation Society) Evaluation tries to answer two distinctive questions: • Did the public intervention have an effect at all and if yes, how big – positive or negative – was this effect. The question is: Does it work? Is there a causal link? This is the counterfactual question • Why an intervention produces intended (and unintended) effects. The goal is to answer the “why and how it works? ” question. To answer this question is the aim of theory-based impact (European Commission – DG REGIO) An evaluation is an assessment, as systematic and objective as possible, of an on -going or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, developmental efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of both recipients and donors ( OECD/DAC)
Monitoring, Auditing and Evalaution Monitoring: observing and collecting data form the project. What time is it? 12, 34 At what time is the train ? 11, 50 Evaluation: interpeting data to provide a subjuctive judgement for accountability and learning: • You did not catch the train because of…. . • An alternative can be……. .
Project development SELECTION Implementation Needs Identification and Formualtion Resources (inputs) END OF Realisation (output) PROJECT Result / outcome After Conlcusion
Project development and Evaluation Needs On going Evaluation Ex ante Evaluation Resources (inputs) Realisation (output) Result / outcome Monitoring Ex Post Evaluation
Relevance and Coherence ( EX ANTE) Relevance ce Program / policy Needs Resources (inputs) Output (implementation) Result (Specific Objective) External coherence Internal Coherence
Performance, Efficiency, Sustainibility, Effectiveness (on going / ex post) Needs Performance ce Resources (inputs) Output (implementation) Effectiveness ce Result (Specific Objective) Efficiencyce Sustainibility
Definitions • Relevance : the project results and Impact CAN produce a change • External Coherence: the project is strategically aligned with the concerning policy (vertical) – the program works in synergy and complementarily with other contextual intervention (horizontal) • Internal coherence: the project objectives, activities, output, results, impacts are logically connected • Performance: the project activities are delivered on time, the outputs respect the targets, the resources are duly absorbed, the procedures are done according to the rules. • Effectiveness: the project achieves results / Impact accordingly with the targets • Efficiency: the project achieves results / Impact accordingly with the targets with the minor costs and in the shorter time. • Impact: the project is capable to contribute significantly to the change • Sustainability: the project changes can last after the conclusion • Utility: Impacts obtained by the intervention correspond to society needs
TOOL : FOCUS GROUP
Focus group Definiton A focus group is a form of qualitative research in which a group of people are asked about their perceptions, opinions, beliefs, and attitudes towards a product, service, concept, advertisement, idea, or packaging. Questions are asked in an interactive group setting where participants are free to talk with other group members. Facilitation tools • Concept mapping • Metaplan ® • Dual Group • Moderator
Performing a Focus group Cabaret setting presentation Brain storming / break question Wrap up Structured discussion
RISK AND CHALLENGES ONE PREDOMINANT VOICE POLARIZATION AND AMBIVALENCE BIZARRE TOOLS AND OBVIOUSE OUTCOME
SWOT Relevance Internal coherence External Coherence Performance Efficiency Effectiveness Impact and Utility Sustainibility ASSESSMENT MATRIX CUSTUMER SATISFACTION MUTICRITERIA ACB
QUESTIONS FOR FOCUS GROUP RELEVANCE EFFICIENCY EFFECTIVENESS SUSTAINIBILITY BEST TOPIC
See you www. t 33. it a. gramillano@t 33. it
- It is arrangement of people in an organization
- Efficiency and effectiveness examples
- Efficiency and effectiveness examples
- Efficiency and effectiveness in higher education
- Fin efficiency vs effectiveness
- Productive inefficiency and allocative inefficiency
- Allocative efficiency vs productive efficiency
- Allocative efficiency vs productive efficiency
- Prove scritte tfa sostegno già svolte con soluzioni
- Maculopatia laser macerata
- Scuola media dante alighieri macerata
- Itas matteo ricci
- Unimc olat
- The effectiveness of online and blended learning
- Managing productivity and marketing effectiveness
- Managerial behavior and effectiveness
- Unit 4 crime and punishment
- Steiners model of group effectiveness
- Definition of personal resources
- Steam condenser heat exchanger
- Cath lab cost effectiveness
- Liberal reforms essay
- Cmin cmax heat exchanger
- Enhancing personal effectiveness
- Slip slop slap effectiveness
- Robbins team effectiveness model
- Definition of personal effectiveness
- Certificate of personal effectiveness
- Pear audit
- Goals of interpersonal effectiveness
- Disadvantages of cognitively guided instruction
- Effectiveness algoritma
- Measuring the effectiveness of the promotional program
- Advancing education effectiveness
- Look for me
- Accounting constraints
- Ppg examples for educator effectiveness
- Sales performance management framework
- Overall equipment effectiveness
- Personal effectiveness framework
- Enthalpy wheel
- It is a graphic language in engineering application
- Measuring coaching effectiveness
- Principles of effectiveness
- Nebido effectiveness
- Happy mid term break
- Sales force effectiveness ppt
- Pear process effectiveness assessment report
- Main idea
- Issues in curriculum development in india
- Steiner model of group effectiveness
- Stronge leader effectiveness performance evaluation model
- Debriifing
- Loss of effectiveness
- Ai sop
- Levonorgeste
- Casey fink survey
- Sales operations organizational structure
- Relative effectiveness
- Air force uei
- Unit effectiveness inspection
- Nc educator effectiveness system
- Measuring workforce effectiveness
- Bed exit alarm effectiveness
- Micro lesson planning
- Sat vocabulary lesson 4
- Lesson 1 thermal energy and the behavior of matter
- Lesson 4 gravity and motion lesson review
- Magnetism lesson outline answer key
- Lesson outline lesson 3 eclipses and tides answer key
- Work power and efficiency
- Environmental and resource efficiency issues
- Ima wheel and axle
- Flexibility and efficiency of use
- Consumers, producers, and the efficiency of markets