Importance of Monitoring and Evaluation Lecture Overview Monitoring

Importance of Monitoring and Evaluation

Lecture Overview § Monitoring and Evaluation § How to Build an M&E System

What is M, what is E, why and how to monitor MONITORING & EVALUATION

What is Monitoring § Ongoing process that generates information to inform decision about the program while it is being implemented. § Routine collection and analysis of information to track progress against set plans and check compliance to established standards § Helps identify trends & patterns, adapts strategies, and inform decisions § Key words: • Continuous – ongoing, frequent in nature • Collecting and analyzing information – to measure progress towards goals • Comparing results – assessing the performance of a program/project

Why is Monitoring Important? § Evidence of how much has been or has NOT been achieved • Quantitative: numbers, percentage • Qualitative: narrative or observation § Examination of trends § Highlight problems § Early warning signs § Corrective actions § Evaluate effectiveness of management action § Determine achievement of results

What is Evaluation § Evaluation is an assessment of an intervention to determine its relevance, efficiency, effectiveness, impact, and sustainability. The intention is to provide information that is credible and useful, enabling incorporation of lessons learned into decision making processes. § Key Words: • Assessment – of the value of an event of action • Relevance • Efficiency • Effectiveness • Impact • Sustainability • Lessons learned

What is Evaluation? Evaluation Program Evaluation Impact Evaluation

What is M and what is E? Monitoring Evaluation Measures progress towards goals, but doesn’t tell us the extent to which results achieved or the impact Measures whether progress towards goal is caused by the intervention - causality Continuous, frequent Infrequent, time bound Has to take place during the intervention Can evaluate ongoing or completed intervention

Monitoring and Evaluation Program Evaluation Monitoring Impact Evaluation

Components of Program Evaluation § What are the characteristics of the target population? What are the risks and opportunities? What programs are most suitable? § What is the logical chain connecting our program to the desired results? § Is the program being rolled out as planned? Is their high uptake among clients? What do they think of it? § What was the impact and the magnitude of the program? Needs assessment Program theory assessment Monitoring and process evaluation Impact evaluation Cost effectiveness § Given the magnitude of impact and cost, how efficient is the program? Are your questions connected to decision-making?

Program Evaluation Programme Evaluation Impact Evaluation

Who is this Evaluation For? § Academics § Donors • Their Constituents § Politicians / policymakers § Technocrats § Implementers § Proponents, Skeptics § Beneficiaries

How can Impact Evaluation Help Us? § Answers the following questions • What works best, why and when? • How can we scale up what works? § Surprisingly little hard evidence on what works § Can do more with given budget with better evidence § If people knew money was going to programmes that worked, could help increase pot for anti-poverty programmes

Programs and their Evaluations: Where do we Start? Intervention Program Evaluation § Start with a problem § Start with a question § Verify that the problem actually exists § Verify the question hasn’t been answered § Generate a theory of why the problem exists § State a hypothesis § Design the program § Design the evaluation § Think about whether the solution is cost effective § Determine whether the value of the answer is worth the cost of the evaluation

Life Cycle of a Program Endline Evaluation Baseline Evaluation Theory of Change/ Needs Assessment Designing the program to implement Background preparation, logistics, roll out of program Monitoring implementati on • Process evaluation • Progress towards target Planning for continuous improveme nt Reporting findings impact, process evaluation findings Change or improvement Distributing reading materials and training volunteers • Reading materials delivered • Volunteers trained • Target children are reached • Classes are run, volunteers show up • Attendance in classes • Entire district is covered • Refresher training of teachers • Tracking the target children, convincing parents to send their child • Incentives to volunteer to run classes daily and efficiently (motivation) • Efforts made for children to attend regularly • Improve coverage Using the findings to improve program model and delivery

Program Theory – a Snap Shot Impacts Results Outcomes Outputs Implementation Activities Inputs

With a Focus on measuring both implementation and results? HOW TO BUILD AN M&E SYSTEM

Methods of Monitoring § First hand information § Citizens reporting § Surveys § Formal reports by project/programme staff • Project status report • Project schedule chart • Project financial status report • Informal Reports • Graphic presentations

Monitoring: Questions § Is the intervention implemented as designed? Does the program perform? Plans and targets Inputs Outcomes Implementation § Is intervention money, staff and other inputs available and put to use as planned? Are inputs used effectively? § Are the services being delivered as planned? § Is the intervention reaching the right population and target numbers? § Is the target population satisfied with services? Are they utilizing the services? § What is the intensity of the treatment?

Implementing Monitoring § Develop a monitoring plan • How should implementation be carried out? What is going to be changed? • Are the staff’s incentives aligned with project? Can they be incentivized to follow the implementation protocol? • How will you train staff? How will they interact with beneficiaries or other stakeholders? • What supplies or tools can you give your staff to make following the implementation design easier? • What can you do to monitor? (Field visits, tracking forms, administrative data, etc. ) • Intensity of monitoring (frequency, resources required, …)?

Ten Steps to a Results-based Monitoring and Evaluation System Selecting key indicators to monitor outcomes Conducting a readiness and needs assessment 1 2 Agreeing on outcomes to monitor and evaluate 3 Planning for improvement selecting realistic targets 4 Gathering baseline data on indicators 5 Using evaluation information 6 Monitoring for results 7 Using findings 8 Reporting findings 9 10 Sustaining the M&E system within the organization

Conducting a needs and readiness assessment 1 § § § § § 2 3 4 5 6 7 8 9 10 What are the current systems that exist? What is the need for the monitoring and evaluation? Who will benefit from this system? At what levels will the data be used? Do we have organization willingness and capacity to establish the M&E system? Who has the skills to design and build the M&E system? Who will manage? What are the barriers to implementing M&E system on the ground (resource-crunch)? How will you fight these barriers? Will there be pilot programs that can be evaluated within the M&E system? - DO WE GO AHEAD?

Agreeing on outcomes (to monitor and evaluate) 1 2 3 4 5 6 7 8 9 10 § What are we trying to achieve? What is the vision that our M&E system will help us achieve? § Are there national or sectoral goals (commitment to achieving the MDGs)? § Political/donor driven interest in goals? § In other words, what are our Outcomes: Improving coverage, learning outcomes… broader than focusing on merely inputs and activities

Selecting key indicators to monitor outcomes 1 § 2 3 4 5 6 7 8 9 Identify WHAT needs to get measured so that we know we have achieved our results? § Avoid broad based results, but assess based on feasibility, time, cost, relevance § Indicator development is a core activity in building an M&E system and drives all subsequent data collection, analysis, and reporting § Arriving at indicators will take come time § Identify plans for data collection, analysis, reporting PILOT! 10

Gathering baseline data on indicators 1 2 3 4 5 6 7 8 § Where are we today? § What is the performance of indicators today? § Sources of baseline information: Primary or Secondary data § Date types: Qualitative or Quantitative § Data collection instruments 9 10

Planning for improvement selecting realistic targets 1 2 3 4 5 6 7 8 9 10 § Targets – quantifiable levels of the indicators § Sequential, feasible and measurable targets § If we reach our sequential set of targets, then we will reach our outcomes! § Time bound – Universal enrolment by 2015 (outcome – better economic opportunities), Every child immunized by 2013 (outcome - reduction in infant mortality) etc. § Funding and resources available to be taken into account Target 1 Target 3 Target 2 Outcome s

Monitoring for implementation and results 1 2 3 4 5 Impacts Results monitoring Results Outcomes 6 7 8 9 Change in percentage children who cannot read; Change in teacher attendance Outputs Implementation Activities Inputs Implementation monitoring Provision of materials; training of volunteers; usage of material; number of volunteers teaching 10

Evaluation(? ), Using Evaluation Information 1 2 3 4 5 6 7 8 9 10 Monitoring does not information on attribution and causality. Information through Evaluation can be useful to § Helps determine are the right things being done § Helps select competing strategies by comparing results – are there better ways of doing things? § Helps build consensus on scale-up § Investigate why something did not work – scope for in-depth analysis § Evaluate the costs relative to benefits and help allocate limited resources

Reporting findings Sustaining M&E System 1 2 3 4 5 6 7 8 9 10 Using Results § Reporting Findings: What findings are reported to whom, in what format, and at what intervals. A good M&E system should provide an early warning system to detect problems or inconsistencies, as well as being a vehicle for demonstrating the value of an intervention – so do not hide poor results. § Using Results: recognize both internal and external uses of your results § Sustaining the M&E System: Some ways of doing this are generating demand, assigning responsibilities, increasing capacity, gather ing trustworthy data.

THANK YOU