Integrating Planning Monitoring and Evaluation to Increase Effectiveness

Integrating Planning, Monitoring and Evaluation to Increase Effectiveness of your Interventions Training organized on 12 December 2011 for HIDA (Ethiopia based NGO) by Laetitia Lienart , IAS Planning, Monitoring & Evaluation Expert

PROGRAMME Introduction Project planning M&E scoping & framework Data management, analysis & use Proposal writing

INTRODUCTION

M&E DEFINITION & ROLE M&E is a systematic process of collecting and analyzing information to accompany the implementation of an action, project or programme and assess its process, outcomes and impacts. LEARNING / CONTINUOUS IMPROVEMENT ACCOUNTABILITY Institutional Memory & Evidence* M&E is not an audit nor does it seek to place blame.

WHAT MAKES M&E SUCCESSFUL? E Ethical, Effective, Efficient V Valued A Accurate, Achievable, Accessible L Learning-oriented U Usable, Used, Useful A See above A T Timely, Transparent, Technically sound I Inclusive O Objective N Non-biaised

Another Success Key for M&E: Good Planning Reflecting and making changes Project planning (proposal) M&E scoping M&E framework Reporting Information management & analysis Information collection

PROJECT PLANNING

DEFINITION § Planning refers to the phase when you design your project. In other words, when you define its goal, objective, expected results, activities you will have to implement to deliver expected results and inputs/resources needed to implement such activities. § The way you design your project should follow a certain logic that shows in a clear way the links between all these elements. Such logic model is essential for results-based management and for a successful M&E which needs to have a clear understanding of what the project is supposed to achieve and how. Failing to plan is Planning to fail!

LOGIC MODEL § Also called ‘’theory of change’’ or ‘’results chain” § Various types of logic model exist and all have pros & cons (just choose the one which is the best adapted to your needs and available resources): ü Logframe ü Outcome mapping ü Outcome chain ü Social framework ü Etc.

THE RESULTS CHAIN The logframe (project framework) is an expression of the “Results Chain” – the results you expect the project to achieve. The box below provides an example of a Results Chain and how it aligns with the LF.

WHAT IS A LOGFRAME? A table giving a clear and synthetic picture of the project impact, HERE outcome, outputs and inputs, their respective indicators and sources of verification as well as assumptions. THERE The LF process helps guide the planning of a journey from where we are now, HERE, to where we want to go, THERE. A - Where do we want to be? What are our expected impact and outcome? B - How will we get there? What are the outputs do we have to deliver? C - What may stop us getting there? What are the risks and how can we manage them? What assumptions are we making? D - How will we know if we’ve got there? What are our indicators, baseline, milestones and targets? What source of information (evidence) do we need? E – What do we need to get there? What detailed resources and related budget are needed?

STEP 1 - DEFINE THE IMPACT (GOAL) § This is the higher order objective, the longer term impact, that the project will contribute to. § It defines the overall “big picture” need or problem being addressed; it expresses the justification of what is planned. TIPS ü Have only one Impact/Goal statement. ü Example: Decreased AIDS mortality rate in the Nbiya region.

STEP 2 - DEFINE THE OUTCOME (PURPOSE/OBJECTIVE) § This describes the specific and immediate results (outcomes) of the project. § It should not be entirely deliverable, i. e. fully within the project manager’s control. If it is deliverable, then it should be an Output*. TIPS ü Have only one Outcome. If you think you have more, then you may need more than one logframe; or your multiple outcomes are in fact indicators of a single outcome or lower outputs. ü Example: Improved access to HIV/AIDS treatment in the Nbiya region.

STEP 3 - DEFINE THE ‘RESULTS/PRODUCTS’ OR OUTPUTS § The Outputs describe what the project will deliver in order to achieve the Outcome/Purpose. They are the results that the project must deliver in the control of the project manager. § Examples: a) Medical infrastructures rehabilitated/reinforced in the Nbiya region; b) HIV/AIDS awareness of Nbiya region’s inhabitants raised; c) Health care workers of the Nbiya region adequatly trained on HIV/AIDS; d) Partnerships between Nbiya’s government and drug supplier(s) initiated; etc. § Typically there are between 2 – 6 Outputs; any more than that and the logframe will become over-complicated.

STEP 4 – DEFINE THE ACTIVITIES § The Activities describe what actions will be undertaken to achieve each output. § Examples: needs assessment, recruitment of experts/consultants, design of materials, development of training programme, selection of participants/trainers/suppliers, organization of coordination meetings, distribution of awareness materials, implementation of works, procurement of equipment & supplies, etc. § Unlike the previous LF template, activities do not have to be captured in the LF. You can report them in a separate table and depending on the donor requirements, specify for each of them the milestones, risks and staff responsible (see example next slide).

ACTIVITIES LOG TEMPLATE

STEP 5 – TEST THE LOGIC FROM THE BOTTOM TO THE TOP § Use the IF/THEN test to check cause and effect, reading from the bottom up: Ø If we do these activities, then this output will be delivered. Ø If we deliver these outputs, then this outcome will be achieved. Ø If the outcome is achieved, then this will contribute to the impact.

STEP 6 – UNDERTAKE A RISK ANALYSIS IDENTIFY THE RISKS § Taking all the activities needed for Output 1, ask the question: ‘if we complete these Activities successfully, then what can stop us delivering Output 1? ’. Repeat for all the other Outputs taking each Output and its associated activities in turn. § Taking all the Outputs together, ask the question: ‘if we deliver all these Outputs successfully, then what can stop us achieving our Outcome? ’ § Now ask the question: ‘if we are achieve our Outcome successfully, then what can stop us contributing to the Impact? ’

STEP 6 – UNDERTAKE A RISK ANALYSIS (cont. ) ANALYSE AND RATE EACH RISK § What is its likely impact? high, medium or low. § What is its likely probability? high, medium or low. You may at this point decide to hereafter disregard insignificant risks. § Discuss & agree possible mitigation measures; transfer them into the Activities log template (i. e. extra activities). § Example: Ø risk: decrease in the # of health care workers affected to the Nbiya region Ø mitigation measure/new activity: advocacy/lobbying to the relevant authorities & stakeholders involved in health care workers management

STEP 7 – FORMULATE THE ASSUMPTIONS § Assumptions are what remains after the mitigation measures have been put in place (even if mitigation measures are successful, it is unlikely you can remove the risk completely). § Assumptions are external factors which could affect the success of the project but over which the project manager has no direct control. § Example: ‘’Strong commitment and continuous support of the Nbiya’s authorities towards combatting HIV/AIDS‘’ § Logic test: once Activities have been carried out, and if the Assumptions at this level hold true, Outputs will be delivered. Test the same logic for the upper levels.

STEP 8 – FORMULATE INDICATORS § Once your logic model is established, it is important to think about which information you will need to collect to measure your project performance. § Indicators are performance measures, which tell us what we are going to measure not what is to be achieved. § Indicators should be specific, usable and easy to measure. The basic principle is that “if you can measure it, you can manage it’’. TIPS Do not start from scratch and use indicators that have been used & tested in your field, such as: http: //www. unaids. org/en/resources/presscentre/featurestories/20 11/november/20111118 reportingguidelines/

Indicators at impact level § Indicators should be “impact” measures. They should only state what will be measured – i. e. they should not include elements of the baseline or target. Under-five AIDS mortality rate falls by at least 5% points by 2012 (to be avoided) Under-five AIDS mortality rate (recommended)

Indicators at outcome level § Outcome indicators demonstrate the changes which take place as a result of the organization’s work, and show progress towards meeting the outcome. § As with the Impact, indicators should not include elements of the baseline or target. The rule is that each Indicator you choose to measure your objectives must be verifiable by some means. If not, you must find another indicator. 50% increase in ARV uptake rate in targeted areas by 2012 ARV uptake rate in targeted areas

Indicators at output level § Output indicators demonstrate the work the organization does and show progress towards meeting objectives. § Best Practice suggests a maximum of three Indicators per Output indicators can be set for: ü Quantity: Number of HIV testing services ü Take-up: Number of people using HIV testing services ü Accessibility: Type of people using HIV testing services ü Quality: Level of user satisfaction with HIV testing services ü Cost

STEP 9 – DEVELOP BASELINE All projects SHOULD have baseline data at Impact, Outcome & Output level before being sent for approval § To make sure the information– in particular baseline data - is robust, you might need to commission new analysis. However, this should be avoided wherever possible: you are encouraged to draw data and analyses from existing sources (for example, from UNAIDS, a partner government statistical office, etc. ). § What is important is that the data and analysis are current, consistent and as accurate as can be reasonably achieved, and disaggregated by sex where appropriate.

STEP 10 – MILESTONES & TARGETS § Indicators are a means by which change will be measured while milestones & targets are respectively mid-term and end “goals”. § Milestones are intended to help you track progress, and therefore you should consider the specific trajectory of your project, taking into account all relevant factors, including the sequencing of activities and the release of data from the source of monitoring information*. § Targets must be Specific, Measurable, Achievable, Relevant and Time bound (i. e. SMART), thereby indicating the desired result at the end of the project. § In line with the guidance for Indicators and baselines, targets must be included wherever baseline data is available and should be disaggregated by sex where appropriate.

STEP 11 – IDENTIFY THE SOURCE § When you choose an indicator, it is important to decide how you will measure. It is what we usually refer to as « Evidence » or « Verification » . § This is a vital stage of the initial planning that is often overlooked. § It should be considered when you formulate your indicators.

Some typical sources of verification § National and international statistics § Project records, reviews and reports § Minutes of meetings and attendance lists § Stakeholder feedback & focus groups § Surveys and reports § Newspapers, radio and TV recordings, photographs, satellite imagery § External evaluation reports, training evaluation questionnaires, etc.

STEP 12 – COMPLETE THE ROW LINKING THE INDICATOR, BASELINE, MILESTONES, TARGET & SOURCE Indicator Baseline 2011 Milestone 2012 0 Doctor 5 Doctors (2 F; 3 M) Number of 4 Nurses (1 F; 3 M) 8 Nurses (4 F; 4 M) health professionals at selected District Hospitals Source: trained on HIV • NGO training reports (quarterly) testing • Hospital records (annual) Milestone 2013 Target 2014 10 Doctors (5 F; 5 15 Doctors (8 F; 7 M) M) 12 Nurses (6 F; 6 M) 20 Nurses (10 F; 10 M)

STEP 13 – DEFINE INPUTS § This level indicates the inputs and related costs needed to fund the project activities, as well as the breakdown of costs by donors (in case the project is funded by several donors). § It also shows the staff who will be involved in the project implementation, and if possible the proportion of their time allocated to the project Inputs ($) Costs • Contract to rehabilitate hospital • ARVs US$5, 000 50% US$3, 000 20% Inputs (HR) 1. Health advisor 2. Communications officer Donor 1 Share % Donor 2 Share % 50% 80%

LOGFRAME TEMPLATE

LOGFRAME TEMPLATE (cont. )

MAKING CHANGES TO THE LF § Logframes are dynamic, subject to change throughout the active life of the project to which they refer. § Changes to a logframe are normally made during a formal review, or in response to circumstances, and should be done in close consultation with all key stakeholders.

RECOMMENDED WEBSITES TO LEARN MORE ON THE LF http: //www. dfid. gov. uk/Documents/publications 1/how-to-guid-rev -log-fmwk. pdf http: //www. dfid. gov. uk/Documents/funding/gpaf/Logframe-howto-note 2011. pdf http: //forexsoftwarebusiness. com/tags/guidance-on-using-therevised-logical-framework. html

EXERCISE § Subject: the Nbiya region is severely affected by HIV/AIDS, with children and women among the most HIV infected populations. You wish to apply for a tender launched by a major donor in order to address HIV/AIDS related health problems in that region. § Methodology: each working group is expected to follow all the above-presented steps to eventually design and present a complete LF with 2 outputs. Use the fact sheet as main background.

M&E SCOPING AND FRAMEWORK

M&E SCOPING Have a focus for your M&E because you cannot measure everything § Possible focus: ü Particular project or activity ü Particular target group ü Inputs, process, outputs, outcomes (changes) and/or impact ü Particular evaluation questions § Criteria to define your M&E scope: ü Info needed for further funding ü You know exactly how the info will be used and for what ü Level of efforts required to collect the information (always consider the info which is already being collected)

M&E FRAMEWORK Indicators and their sources of evidence constitute the basis of an M&E framework § Other elements that should be included: ü Staff responsibilities ü Resources needed and Budget ü Frequency of data collection

INFORMATION COLLECTION TOOLS § Sometimes, sources of evidence are not immediately available and you will have to create a tool to collect the required information (e. g. survey, interview, etc. ). TIPS ü Turn the indicators into questions v Indicator: type of food eaten v Question: how many times do you eat meat per week? ü Select the best tool to collect this information. Criteria for selection tools: v Reliability v Validity v Ease of collection & analysis

DATA MANAGEMENT, ANALYSIS AND USE

§ Management: ü How to store information? v Paper v Simple database (e. g. Excel) or sophisticated database (e. g. Access) v Special database: standard or tailored to your needs (financial and training requirements) § Analysis: ü Data cleaning and harmonization ü Coding if necessary ü Interpreting* ü Recommending § Use: ü Sharing results with staff, users and other key stakeholders ü Making decisions on what must we do differently to increase our success

PROPOSAL WRITING Some tips that should increase your chances to get money … $ $ $

Process overview: 1. Draft proposal: project team 1 3. Finalize proposal: project team leader 2 2. Review draft proposal: PME expert 3 5. Develop work plan: project team 4 4. Disseminate proposal: project team leader 5

Key requirements § Involve all staff concerned and other stakeholders as needed § START designing the project framework & test its validity/logic with main stakeholders § Write the narrative, making sure it matches the project framework (LF) § Ensure consistency throughout the proposal* § Ask colleagues/partners to read it & adjust it if needed § Make sure all teams are properly informed of what they are supposed to deliver as part of the new project, including reporting duties

PROPOSAL STRUCTURE § Background and rationale § Project framework (Logframe) § Activities § Implementation/Coordination (mechanisms) § Risks § M&E § Resources & Budget § Reporting § Appendices (detailed budget, To. Rs, etc. )

Background and rationale This section should describe clearly: ü The problem that the project aims to address (in other words, why you propose this project) and if applicable, the extent to which the project will contribute to the national targets and MDGs. ü Which key stakeholders were involved in the design of this project. ü If the project is a follow-up action of a previous project, intervention, workshop or other event. In that case, provide an overview of this previous project, intervention, workshop or other event (you may use Appendixes to provide further information). ü The role of the organization in this project (leadership, partnership, representation or affiliation function) and why it is strategically positioned to lead or take part to this project (in other words, highlight your comparative advantages/added values). ü The extent to which the project fits into the organization’s strategy. ü In which way(s) your key constituencies will be engaged in this project.

Project framework ü Insert the logframe ü Because activities and inputs are described in a separate section of the proposal, the logframe may only includes the 3 highest levels (i. e. Impact, Outcome and Outputs).

Activities This section should describe the main activities that will be implemented to deliver the expected results. TIPS ü Use bullet points (one per activity) or the Activities Log Table. ü Match activities with expected outputs as stated in the LF. ü Don’t provide too many details (they will be part of the detailed project work plan you will define at a later stage). For example, instead of detailing all sub-activities you will implement to organize a training (such as assess needs of participants, develop curricula, identify speakers, book training venue and accommodation, etc. ), just write as a main activity: Organize a training on [topic] for [target group]. ü Once the proposal is approved, develop a project work plan or Gantt chart (Excel sheet) with detailed activities and sub-activities.

Implementation/Coordination This section should describe clearly: ü Who will be the project leader and what will be his/her role/responsibilities. ü Who will be member of the project team and what will be the role/responsibilities of each member. ü Role/responsibilities of your Communication team (if not part of the project team). ü Role/responsibilities of the staff who are not in the project team and do not belong to the Communication team (only if relevant to the project). ü Mandate and composition of the project steering committee, advisory group, working group or any other kind of consultative body set up for this project. ü Coordination mechanisms put in place to create synergies and avoid overlapping with other similar initiatives (only if relevant to the project).

Risks This section should describe any external factors (i. e. risks) that may have a negative impact on the project and cause its partial or total failure. It should also describe how you will monitor and mitigate them. M&E This section should describe clearly: ü Role/responsibilities of the project team. ü Role/responsibilities of the M&E officer/team. ü What the evaluation will focus on (i. e. process, outcomes and/or impact). You may already specify your key evaluation questions: • What worked well? • What did not work so well? • What could be improved in the design phase? • What could be improved in the implementation phase? • What follow-up actions are needed to make project results sustainable?

M&E (cont. ) Other key evaluation questions: • Are the [items] delivered by the project used? • Are the [items] delivered by the project used in the most efficient and correct way? • Who are the users? etc. This section should also describe: v Which M&E methodologies are envisaged (e. g. tracking records, surveys, interviews, observations, desk reviews, etc. ). TIPS ü Make sure the proposed methodologies will allow measurement of all indicators listed in the project framework. ü If you plan an impact evaluation (i. e. assessment of changes resulting from the project), make sure the budget will include enough funds for this and include it in the project timeline (it should be done at least 1 year after the end of the project).

Resources and Budget This section should describe human resources that will be mobilized for the project and all necessary inputs (goods & services). Provide if possible an outline of the budget (breakdown of total amount by budget line, including a certain % of the total budget for 1) overheads, 2) M&E and 3) communications, if relevant). Reporting This section should specify: ü The frequency of progress and final report(s). ü The outline/structure of progress and final report(s) if possible. ü Try to use you organization’s standard budget lines whenever possible.

Other suggestions to improve HIDA’s visibility, improve work practices & build networks • Endorse the Code of Good Practice for NGOs responding to HIV/AIDS http: //www. hivcode. org/ • Join relevant consortiums • Take part to online discussions, groups, communities of practice • Subscribe to relevant email lists • Attend relevant conferences, meetings, workshops, etc.