IFADs Approach to Project Evaluation Presentation at SHIPDET

IFAD’s Approach to Project Evaluation Presentation at SHIPDET, 5 May 2015 国际农业发展基金的项目评价方法 上海国际发展评价培训春季班, 2015年 5月5日 Independent Office of Evaluation 独立评价办公室

Contents 纲要 • Background on IFAD’s Independent Office of Evaluation (IOE) • Some thoughts on why independent evaluation is useful • Experiences in planning and conducting evaluations • Use of project evaluations • IOE’s evaluation products 2 • 国际农发基金独立评价办 公室（IOE）背景介绍 • 对独立评价作用的认识 • 评价的计划与实施经验 • 项目评价的用途 • IOE的评价产品 Independent Office of Evaluation 独立评价办公室

Introduction 背景介绍 INDEPENDENT OFFICE OF EVALUATION OF IFAD (IOE) 国际农发基金的独立评价办公室 3 Independent Office of Evaluation 独立评价办公室

Background on IFAD 国际农发基金概况 • IFAD = International Fund for • IFAD = 国际农业发展基金 Agricultural Development • Specialised UN Agency, operating as • 专门化的联合国机构，按照国际金 an international financial institution 融机构的方式运作 Ø loans to governments for agricultural investment projects Ø 贷款—各国政府的农业投资项目 Ø 小额赠款--致力于研究、创新和能力 建设的各类机构 Ø Smaller grant programme to a wider variety of actors for research, innovation and capacity building • IFAD started operations in 1978. • IFAD从1978年开始运营。截至 2014 Since then, approved operations for 年 11月，批准的项目金额达 160亿 US$ 16 billion (November 2014) 美元。 4 Independent Office of Evaluation 独立评价办公室

Independent Office of Evaluation of IFAD 国际农发基金独立评价办公室 • Responsibility of the Independent Office of Evaluation • 独立评价办公室（IOE）的职责 (IOE) • Special division of IFAD; does not • IFAD的特设部门，不向IFAD的 管理层汇报 report to IFAD’s Management • Reports to the Executive Board of IFAD (representative of Member States) • 向IFAD的执行董事会报告（执 董会由推选的成员国代表组成） • Evaluation Policy 2003 and 2011 • 评价政策 2003与2011版 • Two main principles: accountability and learning • 两条主要原则：问责与学习 5 Independent Office of Evaluation 独立评价办公室

Types of Evaluations 评价类别 • Corporate-level Evaluations • 机构层面的评价 • Country Programme Evaluations • 国别项目评价 • Project Evaluations • 项目评价 • Evaluation Synthesis Reports • 评价综合报告 • Impact Evaluations • 影响评价 • Annual Report on Results and Impact of IFAD Operations (ARRI) • 年度结果与影响报告 6 Independent Office of Evaluation 独立评价办公室

Some general considerations 一般性问题 WHY INDEPENDENT EVALUATION? 为什么要做独立评价？ 7 Independent Office of Evaluation 独立评价办公室

Accountability and Learning in IFAD的问责与学习 Independent Evaluation 独立评价 Selfevaluation Monitoring 监测 自评价 8 Independent Office of Evaluation 独立评价办公室

Independent Evaluation 独立评价 Independent Evaluation • Judges performance in an objective and transparent way • Contributes to operational knowledge creation or dissemination • Delivers operations evaluation findings and lessons early enough to inform decision making • Focuses on topics and issues that have substantial relevance to development effectiveness. • Engages with diverse stakeholders while maintaining its integrity • Adheres to professional standards • Uses feedback from stakeholders • Provides an external perspectives for innovation and learning • Shifts management focus from inputs to results 独立评价 • 对绩效进行客观透明的评判 • 有助于营运知识的创造与传播 • 能尽早获得关于营运情况的评价结论与 经验教训，有助于改进决策 • 重点关注与发展效果有实质性关联的话 题和问题 • 与各种利益相关者密切合作的同时，保 持IOE的独立公正性 • 遵循职业标准 • 使用来自于利益相关者的反馈 • 为创新和学习提供外部视角 (Based on Bob Picciotto, 2013. Evaluation Independence in Organizations; Journal of Multi Disciplinary Evaluation Volume 9, Issue 20) 9 • 将管理的重心从关注投入转移到关注结 果 Independent Office of Evaluation 独立评价办公室

Exercise 练习 • What are the differences • “独立评价”与“自评 between “independent 价”的区别是什么？ evaluation” and “self evaluation”? • What are the advantages • 两种评价各自的优缺点 and disadvantages for 是什么？ each of the two types of evaluations? 10 Independent Office of Evaluation 独立评价办公室

Independent & Self-Evaluation 独立评价与自评价 Self-evaluation Independent evaluation 自评价 独立评价 • • • Challenges assumptions on the design of policies & programs • Makes self-evaluators think about organizational achievements, systematic stakeholder consultation, reaching consensus about program goals and focusing resources program goals • Enhances effectiveness of self-evaluation • Validates information • • 挑战政策与项目设计时 的假设 • 促使自评价人员对组织 取得的成绩、对利益相 关者的系统性咨询、对 项目目标达成共识、围 绕项目目标实现来进行 资源配置等进行思考 • 提高自评价的有效性 • 验证信息 • • • Closer to the action Lacks distance, less objective Exposed to vested interests Often lacks evaluation skills Interests and concerns may differ from beneficiaries. Likely to support risky management decisions Likely to select evidence that supports assumptions • • • 11 与评价的活动关系更 近 因缺乏距离，不够客 观 易受既得利益的影响 通常缺乏评价技能 可能有着与受益人不 尽相同的利益与关注 点 很可能会支持风险较 大的管理决定 很可能会选择那些支 持假设的证据 Independent Office of Evaluation 独立评价办公室

Some experiences and tips 经验与技巧 PLANNING AND CONDUCTING EVALUATIONS 评价的设计与实施 12 Independent Office of Evaluation 独立评价办公室

IOE’s Evaluation Process 独立评价办公室的评价流程 • Selection of cases & themes • Approach paper • Desk review • Data collection in the field • Debriefing • Draft report & peer review • Comments by Government & IFAD Management • Feedback to Stakeholders • Management response • Agreement at completion point • Final report • Communication products • 选择案例和主题 • 方法文件 • 案头审查 • 实地数据采集 • 任务报告 • 起草报告和同行评议 • 政府和IFAD管理层进行评论 • 反馈给利益相关者 • 管理层的反馈 • 评价完成时达成协定 • 评价报告终稿 • 宣传材料 13 Independent Office of Evaluation 独立评价办公室

Planning an Evaluation 评价的计划 Clarify purpose, users & use 明确评价的目的、使用者和用途 Define evaluation subject 定义评价主题 Evaluation questions 评价问题 Analytical framework 分析框架 Define methodology 确定评价方法 Develop tools 开发评价 具 14 Independent Office of Evaluation 独立评价办公室

Evaluation Criteria • • • 评价标准 Relevance Effectiveness Efficiency Impact Sustainability Gender equality and women’s empowerment Natural resources and climate change Innovation and Scaling-up Borrower performance (Government) Lender performance (IFAD) • 相关性 • 效果 • 效率 • 影响 • 可持续性 • 性别平等和妇女赋权 • 自然资源和气候变化 • 创新与规模扩大 • 借款人的绩效（政府） • 贷款人的绩效（IFAD） 15 Independent Office of Evaluation 独立评价办公室

Exercise 练习 • Which evaluation criteria • 你对哪种评价标准比较 are you familiar with? 熟悉？ • Please identify a few • 请列出几个评价问题来 evaluation questions to 对那些评价标准进行评 assess those criteria. 估？ 16 Independent Office of Evaluation 独立评价办公室

Evaluation Framework 评价框架体系 Evaluation questions Detailed questions/ indicators Data sources Methods used 评价问题 Relevance 相关性 Q 1 Q 2 Q 3 Effectiveness 效果 Efficiency 效率 Impact 影响 Sustainability 可持续性 17 详细问题/ 指标 数据来源 使用的方法 Independent Office of Evaluation 独立评价办公室

Data Collection Methods 数据收集方法 Qualitative and semi-structured 定性数据与半结构化的数据 Direct field observations 直接的实地观察 Key informant interviews 关键知情人访谈 Focus group discussions 焦点小组讨论 Case studies 案例研究 Participatory techniques (e. g. wealth ranking, community mapping, historical transects, most significant changes) 参与式技术（如财富排名、社区地图 绘制、历史横切面、最显著变化） 定量数据 Quantitative Web-based surveys 基于网络的调查 Large sample household surveys 大样本的入户调查 18 Independent Office of Evaluation 独立评价办公室

Exercise 练习 • Which data collection methods are you familiar with? List as many as you can! • Which methods are best suited for the following evaluations: ØEvaluating the impact of drinking water systems in 50 villages ØEvaluating the efficiency of small scale wind power projects in two provinces ØEvaluating the effectiveness of training for accountants • 你对哪种数据收集方法比较熟 悉？请尽可能多地列举。 • 下列评价最适用哪种数据收集 方法： Ø对 50个村庄的饮用水供水系 统的影响进行评价 Ø对两个省的小型风电项目的 效能进行评价 Ø对会计从业人员培训的效果 进行评价 19 Independent Office of Evaluation 独立评价办公室

Rating System 评级体系 Rating Assessment Category �� �价 6 Highly satisfactory 6 非常满意 5 Satisfactory 5 满意 SATISFACTORY �� 满意 4 Moderately satisfactory 4 比较满意 3 Moderately unsatisfactory 3 比较不满意 2 Unsatisfactory 2 不满意 1 Highly unsatisfactory 1 非常不满意 UNSATISFACTOR Y 20 不满意 Independent Office of Evaluation 独立评价办公室

Report Outline 评价报告的提纲 • Executive summary • Management response • Evaluation objectives, methodology and process • The project • 摘要 • 管理层的反馈意见 • 评价的目的、方法与流程 • 项目 - 项目背景 - 项目实施 - Project context - Project implementation • 评价的主要发现 • Main evaluation findings - - Rural poverty impact Core criteria Other performance criteria Overall project achievement Performance of partners • Conclusions • Recommendations 对农村贫困率的影响 核心标准 其它绩效标准 项目的总体成绩 合作伙伴的绩效 • 结论 • 建议 21 Independent Office of Evaluation 独立评价办公室

After the evaluation 评价完成之后 COMMUNICATION AND USE OF EVALUATIONS IN IFAD 评价在IFAD的交流与使用 22 Independent Office of Evaluation 独立评价办公室

Use of Project Evaluations 项目评价的用途 Project evaluations 项目评价 Country programme evaluations 国别项目评价 Corporatelevel evaluations 机构层面的评价 Annual Report on Results & Impact of IFAD Operations 国际农发基金年 度结果与影响报 告 23 Independent Office of Evaluation 独立评价办公室

Main Evaluation Products 主要的评价产品 Other publications 其它出版物 Evaluation reports 评价报告 ARRI Corporate-level Impact A partnership for better results (IOE -SDC) Ten yea rs indepen dent evaluati on booklet book w e i v Over rporateo of a c valuation e l leve gramme Country pro Synthesis Project IOE brochure 24 Newsletter Independent Office of Evaluation 独立评价办公室

Innovative Evaluation Products 创新型的评价产品 Videos 视频 2014 ARRI highlights Profiles & insights 摘要和灼见 Visual sheets 视觉表 Bolivia Country Programme Evaluation Perspectives of beneficiaries on the contributions of the projects supported by IFAD and their vision for achieving more in the future WEAVING A BETTER FUTURE 25 Independent Office of Evaluation 独立评价办公室