International Development Cooperation Theory Practice SNU GSIS Evaluation

International Development Cooperation: Theory & Practice SNU GSIS Evaluation of Development Aid

Evolution of Evaluation v Evaluation began to be implemented in 1950 s n n To measure project/program outputs Logical Framework Approach in 1970 s n As a tool for planning, implementing, monitoring and evaluating projects v Expansion of interest in evaluation during 1980 s n n Evaluation institutionalized, evaluation units set up Focus on long-term impact of aid assistance Active evaluation for individual aid projects To satisfy public opinion and government need to know how public aid funds were used 2 site here Your

Evaluation of Development Assistance v Since 1990 s n In early 1990 OECD DAC introduced monitoring and evaluation on development policy and program n n Evaluation as a strategic tool for knowledge acquisition and construction with the aim of facilitating decision making and organizational learning n Ø DAC Principles for Evaluation of Development Assistance (1991) 5 Criteria for Evaluating Development Assistance Emphasis on aid effectiveness and Result Based Management (RBM) the focus of evaluation is shifting from small projects to national programs and policies, requiring systemic approach to evaluation Based on DAC evaluation principles, many donor countries and agencies developed evaluation methodologies and criteria for their development policy and programs 3 Your site here

Towards Results-based Evaluation v Objectives-based evaluation n n Whether the stated objectives were met Depending on how well the objectives are formulated v Results-based evaluation n ‘results’ defined as ‘the output, outcome or impact of a development intervention’ Focus on not only results but also process and utilization in order to provide lessons for the future Key steps n n To select the right indicators at project design phases To have sound baseline data To set a theory of change, how the intervention is expected to bring changes To look beyond stated objectives to assess unintended, indirect consequences, whether positive or negative 4 site here Your

Types of Evaluation v What is being evaluated n n Project evaluation Country/program evaluation Sector evaluation Thematic evaluation v Project stages n n Ex-ante evaluation Interim evaluation End-of-project evaluation Ex-post evaluation v Goals n n Process evaluation Impact evaluation 5 site here Your

Evaluation Practice: OECD DAC v DAC Network on Development Evaluation (2003) n Members’ Evaluation Guidelines: http: //www. oecd. org/document/11/0, 3343, en_35038640_35039563_35126667_ 1_1_1_1, 00. html n n n DAC Guidance for Managing Joint Evaluations (2006) Evaluation Systems and Use, a Working Tool for Peer Reviews and Assessments (2006) DAC Quality Standards for Development Evaluation (http: //www. oecd. org/dataoecd/55/0/44798177. pdf) v DAC Evaluation Resource Centre (DERe. C) n Evaluation Reports available at: www. oecd. org/dac/evaluationnetwork/derec n Reports search by sectors or countries 6 site here Your

Evaluation Practice: World Bank (1) v World Bank: Independent Evaluation Group (M&E) (http: //ieg. worldbankgroup. org ) n n In charge of evaluating the activities of IBRD and IDA (the World Bank), the work of IFC in private sector development, and MIGA's guarantee projects and services IEG > Publication search v Aid Effectiveness Review n n http: //go. worldbank. org/E 561 OLG 8 Z 0 Project database Project maps Country strategy 7 site here Your

Evaluation Practice: World Bank (2) v Ten-Step to a Results-Based Monitoring and Evaluation System: A Handbook for Development Practitioners (2004) (pdf) 8 site here Your

World Bank: IEG-Evaluation Tools v Project Reviews n n From ICR (Implementation Completion Report) to PPAs (Project Performances Assessment Report) Rate projects in terms of outcome (relevance, efficacy and efficiency), sustainability of results, and institutional development impact v Country Assistance Evaluations (CAEs) v Sector and Thematic Reviews n n By sector (agriculture, transport, etc. ) By thematic area (poverty, gender, etc. ) v Process Reviews n On ongoing activities (aid coordination, grant making, etc. ) v Impact Evaluation: systematic identification of effects input outcome impact experts equipment resources reports labor skill roads income employment productivity econ growth better health life expect. 9 Your site here

Evaluation of Development Aid v Why is evaluation important? n n Past performance is no indication of future return: As we can recognize these failures, we shouldn’t be forced to repeat them. Evaluation of development programs provides information about what works, what does not and why. n This learning contributes to improving the development effectiveness of aid and helps hold donors and partner country governments accountable for results. v Agenda for international development cooperation and development evaluation n More aid Increasing the demands on evaluation Better aid Quality of evaluation to be ensured Aid coordination Quality of aid to be assessed 10 site here Your

Evaluation of Development Aid v OECD DAC Definition n An evaluation is an assessment, as systematic and objective as possible, of an on-going or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfillment of objectives, developmental efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of both recipients and donors. v The main purposes of evaluation are: n n to improve future aid policy, programmes and projects through feedback of lessons learned; to provide a basis for accountability, including the provision of information to the public. 11 site here Your

Monitoring and Evaluation v Complementarity between Monitoring and Evaluation Monitoring Continuous or periodic Improve progress in implementation, efficiency, adjust work plan Short term Evaluation Frequency Basic purpose horizon Episodic Improve relevance, effectiveness, impact, future programming Long term, beyond scope of specific program Inputs, process, outputs Focus in program cycle Also outcomes and impact Work plans, performance targets, and reference indicators References for comparison Broader objectives and strategy, as well as external benchmarks/standards Objectives taken as given Objectives Also assess how well the objectives are formulated Pre-defined indicators assumed to be appropriate Indicators Validity and relevance of re-defined indicators open to question Focus on intended results, without answering causal questions Results and causal chains Usually an internal management function Undertaken by Identifies both intended and unintended results, providing answers to causal questions Often done by external evaluators 12 site here Your

DAC 5 Criteria for Evaluating ODA v OECD “Quality Standards for Development Evaluation” (2010) n n The evaluation applies the agreed DAC criteria for evaluating development assistance: relevance, efficiency, effectiveness, impact and sustainability. The application of these and any additional criteria depends on the evaluation questions and the objectives of the evaluation. If a particular criterion is not applied and/or any additional criteria added, this is explained in the evaluation report. All criteria applied are defined in unambiguous terms. Your site here

DAC 5 Criteria: 1. Relevance v Definition: n The extent to which the aid activity is suited to the priorities and policies of the target group, recipient and donor. v Key questions: n To what extent are the objectives of the project still valid? n n Are the activities and outputs of the project consistent with the overall goal and the attainment of its objectives? n n Are they consistent with partner country’s development priorities as well as the donor’s strategy? were the project design and problem solving methods appropriate? Are the activities and outputs of the project consistent with the intended impacts and effects? n Were the outcomes and impacts set appropriately? Focuses primarily on the ‘goals and priorities’ of governments’ strategies, not on meeting the ‘needs’ of targeted population. 14 Your site here

DAC 5 Criteria: 2. Efficiency v Definition: n Efficiency measures the outputs (qualitative and quantitative) in relation to the inputs. n n It is an economic term: whether the aid uses the least costly resources possible in order to achieve the desired results. Generally requires comparing alternative approaches to achieving the same outputs, to see whether the most efficient process has been adopted. v Key questions: n n n Were activities cost-efficient? Were objectives achieved on time? Was the program or project implemented in the most efficient way compared to alternatives? Falls short on the coverage of (non-monetary) costs and comparisons. 15 Your site here

DAC 5 Criteria: 3. Effectiveness v Definition n The extent to which an aid activity attains its objectives. v Key questions: n To what extent were the objectives achieved? n n Were the planned output achieved? Were the planned outcome achieved? Are the planned and actual beneficiaries identical? What were the major factors influencing the achievement or non-achievement of the objectives? ▶ Goals (objectives) reflect only the expectations, and can be set too low or too high and thus not provide a good parameter for evaluation. ▶ Possibly to be subsumed under ‘impact’ criterion 16 Your site here

DAC 5 Criteria: 4. Impact v Definition n The positive and negative changes produced by a development intervention, directly or indirectly, intended or unintended. n the main impacts and effects resulting from the activity on the local social, economic, environmental and other development indicators. v Key questions: n What has happened as a result of the program or project? n n Impacts on comprehensive socio-economic development including poverty reduction, job creation and economic growth, human resource development, gender equality, etc. Impacts on systemic changes of the partner country through improvement in governance, practices, motivations, capacity, etc. What real difference has the activity made to the beneficiaries? How many people have been affected? Possibly on superior position in terms of importance but with same weight as the other criteria 17 Your site here

DAC 5 Criteria: 5. Sustainability v Definition: n Sustainability is concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. n Projects need to be environmentally as well as financially sustainable. v Key questions: n To what extent did the benefits of a program or project continue after donor funding ceased? n n Is there an effective demand for the results of the project? Is it possible to secure financial and human resources for the appropriate operation and maintenance of the project? Does the partner government have ownership of the project? What were the major factors which influenced the achievement or non-achievement of sustainability? Misses other important elements, e. g. , political support, sociocultural adequacy, technological appropriateness, institutional capacity, etc. 18 Your site here

19 site here Your

20 site here Your

DAC Criteria: Assessment (Chianca, 2008) v Ideas for improvement n Revisions of definitions n n n Addition of key missing criteria n n n Refocus ‘relevance’ and ‘effectiveness’ to address people’s need and not agencies’ goal Subsume ‘effectiveness’ under ‘impact’ Improve coverage of ‘cost’ and ‘comparison’ under ‘efficiency’ Include ‘cultural appropriateness’ under ‘sustainability’ Quality of process Exportability Discussion on level of importance of the criteria n n Minimum acceptable level (bar) Weighing (relative importance) 21 Your site here

Evaluation standards in other agencies v ALNAP (Active Learning Network for Accountability and Performance in Humanitarian Action) n three maintained: effectiveness (including timeliness & coordination), efficiency, impact (including breadth of impact) n ‘relevance’ replaced by ‘appropriateness’ n ‘connectedness’ instead of ‘sustainability’ n two new: coverage (need), coherence (in policies) v Europe. Aid n based on DAC 5 criteria n two new: coherence/complementarity & community value added v World Bank (Independent Evaluation Group) n DAC 5 criteria (slightly different interpretation) n new: bank and borrower performance 22 Your site here

Evaluation standards in other agencies 23 Your site here

Evaluation Standards: Japan and Korea v JICA’s new rating system for ex-post evaluation (2008) n n Common to all three schemes: tech-coop, loans & grants 4 Sub-rating (in three grades of a/b/c) n (1) Relevance, (2) Effectiveness (impacts), (3) Efficiency, and (4) sustainability Overall rating (in four grades of A/B/C/D) (New JICA Guidelines for Project Evaluation, 1 st ed. , 2010) n v EDCF Evaluation Manual (2011) n n 4 level rating (1~4 points) for each DAC 5 criteria Overall rating of evaluation results: with weight of 20% for the score of each criterion n n >3. 7: highly successful 2. 6~3. 6: successful 1. 8~2. 5: partly successful 1. 8<: unsuccessful 24 Your site here

Evaluation of Aid Effectiveness and Quality v Survey on Monitoring the Paris Declaration, 2011 n n Evaluation on progress in implementing the PD For both donor and partner countries v Commitment to Development Index (CDI) n n n http: //www. cgdev. org/section/initiatives/_active/cdi/ in seven policy areas: aid (both quantity as a share of income and quality), trade, investment, migration, environment, security, and technology For individual donor countries’ policy v Quality of ODA (Qu. ODA) n http: //www. cgdev. org/section/topics/aid_effectiveness/quoda n n n Four dimensions: maximizing efficiency, fostering institutions, reducing burdens, transparency and learning 30 indicators For individual donor countries and development agencies 25 Your site here

5 Principles of Paris Declaration on Aid Effectiveness (2005) • 5 Partnership Commitments ü ü ü 26 Ownership Alignment Harmonization Managing for results Mutual accountability Your site here

Paris Declaration Indicators Ownership (partner countries) 1 　 2 a 　 2 b 　 3 　 4 　 5 a Alignment (donors-partners) 　 5 b 　 6 　 7 　 8 　 9 　 Harmonisation 10 a (donors-donors) 　 10 b 　 11 Results 　 12 Mutual accountability 　 Operational Development Strategies % of countries having a national development strategy rated “A” or “B” on a five-point scale Reliable public financial management (PFM) systems % of countries moving up at least one measure on the PFM/CPIA scale since 2005 Reliable procurement systems % of countries moving up at least one measure on the four-point scale since 2005 Aid flows are aligned on national priorities % of aid for the government sector reported on the government’s budget Strengthen capacity by co-ordinated support % of technical co-operation implemented through co-ordinated programmes consistent with national development strategies Use of country PFM systems % of aid for the government sector using partner countries’ Use of country procurement systems % of aid for the government sector using partnercountries’ procurement systems Strengthen capacity by avoiding parallel PIUs Total number of parallel project implementation units (PIUs) Aid is more predictable % of aid for the government sector disbursed within the fiscal year for which it was scheduled and recorded in government accounting systems Aid is untied % of aid that is fully untied Use of common arrangements or procedures % of aid provided in the context of programme-based approaches Joint missions % of donor missions to the field undertaken jointly Joint country analytic work % of country analytic work undertaken jointly Results-oriented frameworks % of countries with transparent and monitorable performance assessment frameworks Mutual accountability % of countries with mutual assessment reviews in place 27 Your site here

Qu. ODA - Indicators of Aid Quality Maximizing Efficiency Fostering Institutions Transparency and Learning Reducing Burden Share of allocation to poor Share of aid to recipients’ countries top development priorities Significance of aid relationships Member of IATI Share of allocation to well- Avoidance of PIUs (Project governed countries Implementation Units) Fragmentation among agencies within a given donor country. Recording of project title and descriptions Low administrative unit costs Share of aid recorded in recipient budgets Median project size Detail of project descriptions High country programmable share Share of aid to partners with Contribution to good operational strategies multilaterals Reporting of aid delivery channel Focus/specialization by recipient country Use of recipient country systems Coordinated missions Share of projects reporting disbursements Focus/specialization by sector Coordination of technical cooperation Coordinated analytical work Completeness of projectlevel commitment data Support of select global public good facilities Share of scheduled aid recorded as received by recipients Use of programmatic aid Aid to partners with good M&E frameworks Share of untied aid Coverage of forward spending plans/Aid predictability 　 　 Source: Birdsall and Kharas (2010) 28 site here Your

Maximizing Efficiency v If donors optimized aid practices and policies, they could potentially save billions of dollars. v Collier and Dollar (2002) suggested the effectiveness of aid on poverty could double if selectivity in terms of governance and pro-poor orientation were optimized. v Tying aid is believed to add anywhere from 10 to 30% of costs. More than 10 percent of all DAC donor aid is still tied, or $12 billion, so the costs of tying are in the range of $1. 2~$3. 6 billion. And non-DAC donors tie considerably higher portions of their aid. n Aid. Watch reports that “buying food locally in sub-Saharan Africa… costs 34 percent less than shipping it from the US, and gets there on average more than 100 days more quickly”. v Volatility also has significant costs, estimated at $6. 8 billion. n One-fifth of the explained volatility is associated with donor behavior (as opposed to such recipient issues as conflict or elections), or $1. 3 billion. 29 site here Your

Maximizing Efficiency 30 site here Your

Fostering Institutions v Carefully planned development assistance can be harmonized with recipient systems, while go-italone or parallel approaches to aid delivery risk substituting for local institutions and stunting their growth. v Higher aid levels tend to reduce institutional quality significantly. This effect is larger for grants —the dominant form of aid—than for loans, perhaps because grants and loans are often administered by different donor agencies. v The key point is that the way in which aid is provided can affect the impact of aid depending on whether domestic institutions are strengthened or weakened in the process. 31 Your site here

Fostering Institutions 32 site here Your

Reducing the administrative burden v The fragmentation of official development assistance has created a huge burden on recipient country administration. With many bilateral and multilateral relationships to manage, recipient administrators are spending an increasing share of their time and resources receiving donor missions. v The OECD estimates that the deadweight losses associated with an average of 263 missions annually to aid-recipient countries could be as high as $5 billion. v Some countries must prepare more than 800 new foreign-assisted aid projects annually and present 2, 400 quarterly reports on progress. 33 site here Your

Reducing the administrative burden 34 site here Your

Transparency and learning v The benefits of transparency come from the improvements in development effectiveness that can result from better information within partner countries, which empirical studies indicate is associated with reduced diversion of aid-funded government expenditures to other purposes. v The Aidinfo report includes an estimate of the benefits of this greater transparency within countries on the order of $1 billion a year. v Accessible, standardized, timely information has the potential to dramatically enhance effectiveness. 35 site here Your

Transparency and learning 36 site here Your

Aid Quality by Country 37 Your site here

Aid Quality by Country 38 site here Your

Q&A