Lesson 4 Sustainability impact and customer satisfaction Macerata

Lesson 4: Sustainability, impact and customer satisfaction Macerata, 19 October Andrea Gramillano, t 33 srl

Agenda v 1) A few words on the next lesson v 2) What we need from the previous lesson (performance, efficiency, effectiveness) v 3) What do impact and sustainability mean? v 4) Put a table on ex-ante / on going / ex-post v 5) Customer satisfaction

Next week Morning 11 -13 (Prof. Moroni – UNHCR) Comprehensive refugee response framework: a global solution for a global problem • definitions (refugees, internal displace people, asylum seekers, etc) • global data on displacement, key countries of origins of refugees and displaced people, and key hosting countries • need for comprehensive response and key elements of the approach Link with the course 1) International cooperation = humanitarian cooperation + development cooperation 2) Type of funding = NGOs+PCM+proposals

Previous lessons Phase Criteria Tool Time Identification (of needs) Relevance SWOT - PEST Before the project starts Formulation (feasibility study) Coherence (at least) Matrix analysis and document analysis Before the project starts Appraisal and approval Implementation Performance Efficiency Effectiveness Evaluation and audit Performance Efficiency Effectiveness Impact Sustainability Monitoring / indicator Multicriteria analysis Cost effectiveness Comparison achieved value and target During the project implementation See above In an advanced phase and at the end

Logframe and impact Outcome Output Input

Logframe and impact Impact Outcome Output Input

What are the stakeholders behind each step? (e. g. construction of a bridge, new school, support to enterprises) Impact Outcome Output Input

Impact – OECD approach Definition • OECD: Did the project bring real change? Type of Questions 1. To what extent was the project objective achieved? How did the outcomes influence the indirect beneficiaries? 2. Did the project produce unintended effects? Why did these unintended outcomes (positive or negative) happen? 3. What was the influence of external factors (negative, positive, little/no influence) such as changes in policy legislation, general economic and financial conditions?

Impact indicators Outcome indicators Output indicators Procedural indicators Input indicators

Sustainability Definition Type of Questions • OECD: Will the • 1. Are the outcome likely to be political, institutional, sustained in the future? managerial, and Did national/regional/local technical, change last • 2. authorities, civil society take beyond the initiative? ownership of the outcomes of the project? • 3. Are changes in attitudes, behaviour patterns and institutional arrangements likely to last once the project is closed?

Efficiency, Impact and sustainability in the evaluation Process Impact as a contribution to the overall objective What does it mean? ?

Efficiency, Impact and sustainability in the evaluation Process Effectiveness: Have targets been achieved? Project Output / outcome 1 Outcome / output 2 Outcome / output 3 Efficiency At what cost? Sustainability: Will they last? Impact: Are they directly related? Impact as a net effect (counterfactual approach) – similar to added value

Definitions • Relevance : the project outcomes and Impact CAN produce a change • External Coherence: the project is strategically aligned with the concerning policy (vertical) – the program works in synergy and complementarily with other contextual intervention (horizontal) • Internal coherence: the project objectives, activities, output, outcomes, impacts are logically connected • Performance: the project activities are delivered on time, the outputs respect the targets, the resources are duly absorbed, the procedures are done according to the rules. • Effectiveness: the project achieves outcomes / Impact accordingly with the targets • Efficiency: the project achieves outcomes / Impact accordingly with the targets with the minor costs and in the shorter time. • Impact: the project is capable to contribute significantly to the change • Sustainability: the project changes can last after the conclusion • Utility: Impacts obtained by the intervention correspond to society needs

Project development SELECTION Implementation Identification and Formualtion Needs Resources (inputs) Realisation (output) Result / outcome END OF PROJECT After Conlcusion

Project development and Evaluation Needs On going Evaluation Ex ante Evaluation Resources (inputs) Realisation (output) Result / outcome Monitoring Ex Post Evaluation

Relevance and Coherence ( EX ANTE) Relevance ce Program / policy Needs Resources (inputs) Output (implementation) Result (Specific Objective) External coherence Internal Coherence

Performance, Efficiency, Sustainability, Effectiveness (on going / ex post) Needs Performance ce Resources (inputs) Output (implementation) Effectiveness ce Outcome Efficiencyce Sustainability

Definitions • Relevance : the project results and Impact CAN produce a change • External Coherence: the project is strategically aligned with the concerning policy (vertical) – the program works in synergy and complementarily with other contextual intervention (horizontal) • Internal coherence: the project objectives, activities, output, results, impacts are logically connected • Performance: the project activities are delivered on time, the outputs respect the targets, the resources are duly absorbed, the procedures are done according to the rules. • Effectiveness: the project achieves results / Impact accordingly with the targets • Efficiency: the project achieves results / Impact accordingly with the targets with the minor costs and in the shorter time. • Impact: the project is capable to contribute significantly to the change • Sustainability: the project changes can last after the conclusion • Utility: Impacts obtained by the intervention correspond to society needs

Phases of evaluation and Criteria EX ANTE Relevance Internal coherence External Coherence Performance Efficiency Effectiveness Sustainability / Impact / Utility Only in some project proposals ON GOING EX POST

Evaluation from different perspectives Prospective and Summative: the evaluation provide a judgement approach Formative: evaluation is for learning and changing the project Key difference Status Evaluator: - summative, shall be independent - formative, he can be from the organisation implementing the project (auto - evaluation) Qualitative: analysis of intangible effect (opinions, perceptions, facts) and narrative approach Quantitative: analysis of tangible effects by sharp measurement. Methods: Qualitative: case studies, interviews, focus group Quantitative: statistical analysis, cost and benefit analysis, counterfactual Counterfactual: identify what is Theory based: identify the Type of projects: the impact (net effect) casual effects linked the Counterfactual: the impact shall be quantifiable and impact/ effect to the project easy to be identified Participative: involvement of Top down: technically the stakeholders during the driven and detached from whole evaluation process ( data the project actors collection, analysis, assessment, dissemination) Involvement of project actors

SUPPORT= 1 MILLION DOLLARS ENTERPRISES SUPPORTED=1000 OUTPUTS= MACHINERY, EQUIPMENTS OUTCOME=1000 JOBS CREATED (1 FOR EACH ENTERPRISE) SUSTAINABILITY (1000 JOBS ARE STILL THERE AFTER A FEW YEARS) IMPACT = ADDITIONAL JOBS IN THE SAME ENTERPRISES AND OTHER ENTERPRISES FOLLOW THE SAME BEHAVIOUR OF SUPPORTED ONES.

WHAT IS THE IMPACT (CONTRIBUTION TO THE CHANGE IN THE CONTEXT) OF 1000 JOBS IN SUPPORTED ENTERPRISES IF ON AVERAGE IN THE CONTEXT NON-SUPPORTED ENTERPRISES INCREASE 1 JOB EACH SUPPORTED ENTERPRISES = OUTCOME 1000 JOBS ( 1 EACH) NON SUPPORTED ENTERPRISES 1 JOB INCREASE ON AVERAGE?

5) Customer satisfaction (effectiveness = achieving a certain level of student’s satisfaction)

The Customer Satisfaction Customer satisfaction is a technique coming from Marketing. It aims to understand how customer expectations are met by products and services. More in detail, CS is based on the comparison of customers’ expectations and their perceived performance. It is positive when a product performs over the expectations. CS is measured at the individual level, but it is almost always reported at an aggregate level and it is often measured along various dimensions. There are different methods adopted in business. One of the most famous is the SERVQUAL by Parasuraman, Zeithaml and Berry.

IDENTIFY GAPS • GAP: between service quality specification and service delivery This gap may arise through service personnel being poorly trained, incapable or unwilling to meet the set service standard. • GAP: between service delivery and external communication Consumer expectations are highly influenced by statements made by company representatives and advertisements. • GAP: between expected service and experienced service This gap arises when the consumer misinterprets the service quality.

STEPS TO DEVELOP CUSTOMER SATISFACTION Establish Dimensions Define Questions Fix Scales Analyse results Deliver Survey Set questionnaire

DIMENSION AND QUESTIONS DIMENSIONS • Dimensions represent different aspects of the services and products SERVQUAL model, refers to five dimensions of quality: q Reliability q Responsiveness q Assurance q Empathy q Tangibles QUESTIONS They represent the concrete translation of the dimensions in items to be asked to the customers. ES. A project related to vocational training defines the content of training, the methods and the materials. It might ask to students “how are you satisfied with the on line available documents? ”

FACTORS of the RATER model • Reliability: the ability to perform the promised service dependably and accurately • Assurance: the knowledge and courtesy of employees and their ability to convey trust and confidence • Tangibles: the appearance of physical facilities, equipment, personnel and communication materials • Empathy: the provision of caring, individualized attention to customers • Responsiveness: the willingness to help customers and to provide prompt service

Demographic information Responses (based on a Likert scale) Age / Year (Master degree – Bachelor degree) Employment situation (yes / no) City and County (living in Macerata or not) Frequency of the service (Attendance…)

Demographic information Responses (based on a Likert scale) Age / Year (Master degree – Bachelor degree) Employment situation (yes / no) City and County (living in Macerata or not) Frequency of the service (Attendance…) FACTORS of the RATER model • Reliability: the ability to perform the promised service dependably and accurately QUALITY OF EXPLANATION (LESSONS/CONTENTS) • Assurance: the knowledge and courtesy of employees and their ability to convey trust and confidence APPROACH TO STUDENTS IN CLASS Y • Tangibles: the appearance of physical facilities, equipment, personnel and communication materials QUALITY OF THE ROOM / TIMING / ACCESSIBILITY • Empathy: the provision of caring, individualized attention to customers AVAILABILITY FOR INDIVIDUALISED SUPPORT • Responsiveness: the willingness to help customers and to provide prompt service Time to answer • …………. OPEN QUESTION FOR SUGGESTIONS

SCALE (LIKERT) and QUESTIONAIRE

SURVEY and ANALYSIS Survey • DEFINE YOUR TARGET • SAMPLE ? • tools: q PHONE q WEB q PERSONAL INTERVIEW ANALYSIS (descriptive statistics) Questions 1 … TOT. Resp. score Mode Exp.

WORKOUT: DO YOU LIKE THIS COURSE? From dimensions to questions STEPS Establish Dimensions Analyse results Define Questions Deliver Survey Fix Scales Set questionnaire Dimensions Questions

Questions Very bad (1) QUESTIONAIRE 1) Are the materials / contents relevant / useful for your professional preparation? 2) Do you think that materials and contents provided during the lessons are adequate to prepare for the exam? 3) Did the course respect the predefined programme? 4) Was the professor available for providing additional information, when needed? 5) Are contents exposed in a clear way during the lessons ? 6)Is the classroom (furniture, room, computers …. ) adequate for the course? 7) Is the lesson schedule adequate compared to the entire students’ week timetable? 8) Is the professor answering the questions in an effective way? 9) Is the distribution of the hours dedicated to the subjects adequate? 10) Does the course fit your degree? 11) Was the professor able to attract the students’ attention? 12) Was the professor ready to provide supports and guidance after the class? 13) Did the professor respect the timetable (he was there for the lessons)? 14) Did the course meet your expectations? IF not please provide any suggestion… More bad than good (2) More good than bad (3) Very good (4)

1) Are the materials / contents relevant / useful for your professional preparation? 2) Do you think that materials and contents provided during the lessons are adequate to prepare for the exam? 3) Did the course respect the predefined programme? 4) Was the professor available for providing additional information, when needed? 5) Are contents exposed in a clear way during the lessons ? 6)Is the classroom (furniture, room, computers …. ) adequate for the course? 7) Is the lesson schedule adequate compared to the entire students’ week timetable? 8) Is the professor answering the questions in an effective way? 9) Is the distribution of the hours dedicated to the subjects adequate? 10) Does the course fit your degree? 11) Was the professor able to attract the students’ attention? 12) Was the professor ready to provide supports and guidance after the class? 13) Did the professor respect the timetable (he was there for the lessons)?

Questions 1 2 3 4

ANALYSIS 1 2 3 4

6) Performance

COST AND BENEFIT ANALYSIS

FRAMEWORK STEPS Sustainability Analysis Financial Analysis Economic Analysis Risk Analysis Key concepts • • Economic vs. Financial Discounting Externalities Shadow prices ( conversion factor) • Conditionality • Time Horizon

EXERCISE

Project – Renewal of a museum • Investment costs Machinery and equipment = 20, 000 euro (first year) Work = 20, 000 euro (second year) Materials = 10, 000 euro (second year) Other costs = 6, 000 euro (second year) Extraordinary maintenance = 6, 000 euro (8° year) • Operational costs (from 3 th year) General costs = 1, 000 euro (from 3 th year) Ordinary maintenance = 5, 000 euro (from 3 th year) Work = 2, 000 euro (from 3 th year) • Cash flows (from 3 th year) • Biglietti = 10, 000 euro (from 3 th year) • Financial resources • Own resources = 25, 000 euro (second year) • Public contribution = 35, 000 euro (first year)

Project – New wind power plant • Investment costs Machinery and equipment = 15, 000 euro a year (for three years) Work = 6, 000 euro a year (for three years) Materials, land= 1, 000 euro (for three years) Extraordinary maintenance = 6, 000 euro (8° year) Residual value = 5, 000 euro • Operational costs (from 4 th year) General costs = 500 euro Ordinary maintenance = 1, 000 euro Work = 500 euro • Cash flows (from 4 th year) • Tarif = 24, 000 euro (from 3 th year) • Financial resources • ? ? ?

Year 1 Cash flows Financi al resourc es Own resources Investm ent costs Machinery and equipments Public resources Work Land other costs Residual value Extraordinary maintenance Operati onal costs General costs Ordinary maintenance Work Rev and costs Revenues – costs cumulated Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year 10

ANNEX

DISCOUNTING Discount rate: The rate at which future values are discounted to the present. Usually considering roughly equal to the opportunity cost of capital.

DISCOUNTING Example: 1 Euro invested at 5% yearly rate, will become (1+5%)=1, 05 after one year; (1, 05)x(1, 05)=1, 1025 after two years; (1, 05)x(1, 05)=1, 157625 after three years. The economic present value of 1 Euro that will be spent or gained two years later is 1/1, 1025=0, 907029; three years later is 1/1, 157625=0, 863838.

DISCOUNTING /Most Member States have their own social discount rates for public sector projects. /These discount rates comprise between a maximunm 10% rate and a minimum 3% real rate. /In this context a real 5% discount rate may be an appropriate benchmark value: low enough to focus attention on project not passing the test, but not too far from the average official discounting rate.

5 FINANCIAL ANALYSIS • The financial analysis is made up of a series of tables that collect the financial flows of the investment: - total investment, - operating costs and revenue, - sources of financing, • The financial analysis should finally result in two tables summarising the cash flows to verify: – SUSTAINABILITY – RETURN OF INVESTIMENT

FINANCIAL SUSTAINABILITY TABLE Financial sustainability is verified if this row is more than or equal to zero for all the years considered.

FINANCIAL INTERNAL RATE OF RETURN OF INVESTMENT (FRR/C) This value is calculated with a financial discount rate of 6% This rate of return measures the capacity of operating net revenues to sustain the investment costs regardless the way they are financed.

Net present value Rate of return

ECONOMIC ANALYSIS “The economic analysis appraises the project contribution to the economic welfare of the region or country. It is made on behalf of the whole society (region or country) instead of just the owner of the infrastructure like in the financial analysis. ”

The three steps for economic corrections STEP 1 - Fiscal corrections STEP 2 - Externalities corrections STEP 3 - From market to accounting price

(1). Phase 1. Fiscal correction. It is necessary to deduct from the flows of financial analysis, payments that have no real resources counterpart, as for subsidies and indirect taxes on input and output. For direct public tranfers they are already not included in the starting table for financial analysis which considers investment costs and not financial resources.

(2). Phase 2. Externalities correction. It is necessary to include among outflows and inflows also external costs and benefits for which there is no cash flow. Some examples could be costs for health services or losses in fisheries due to incresed pollution, time saved for investment in transports, specific infrastructures supplied by public sector for the project (a road built specifically for the project…) increased touristic flows, increased accessibility of the region…

(3). Phase 3. From market to accounting prices. It is necessary to determine a vector of conversion factors including distortion of the market into prices. Such distortions affect input, outputs and wages.

STEP 1: fiscal corrections This value is calculated with a social discount rate of 5% STEP 3: From market to accounting prices STEP 2: Externalities corrections

8 SENSITIVITY AND RISK 4 The project examiner should check if risks associated with the project have been assessed by the applicant. 4 This check can be done by sensitivity analysis alone, and implies a certain amount of reasoning, if not of calculations, in terms of probabilities of the main variables affecting the expected outcome of the project.

8 SENSITIVITY AND RISK 4 Sensitivity analysis: A study of the impact that changes in crucial estimates concerning costs and benefits would have on the profitability or present value of a project 4 Probability distribution: One is often interested to know what the probability is of a project having a economic rate of return less than the minimum required rate.

8 SENSITIVITY AND RISK Sensitivity analysis is a deterministic procedure, aimed at understanding which variables influence the economic net present value or ERR of the project. Any independent variable for which 1% change results in 1% change or more of ENPV or ERR is certainly a critical one, and needs to be estimated as reliably as possible.

8 SENSITIVITY AND RISK Risk assessment implies assigning probabilities to these critical variables and calculating the probability distribution of ERR.