Monitoring and Evaluation Frameworks For Malaria Programs Learning

Monitoring and Evaluation: Frameworks For Malaria Programs

Learning Objectives At the end of the session, participants will be able to: 1. Identify and differentiate between conceptual frameworks, results and logical frameworks, and logic models 2. Define goals and objectives for specific intervention programs 3. Design frameworks for specific intervention programs 4. Discuss how frameworks are used for M&E planning

Module Outline Introduction to frameworks o Conceptual Frameworks o Goals and objectives o Results Frameworks o Logical Frameworks o Logic models Exercises developing frameworks Interface between frameworks and M&E planning Work on country group project

Types of Frameworks Conceptual Frameworks Results Frameworks Logical Frameworks Logic Models Why is it important to know about each of these frameworks?

Why are frameworks useful? Designing frameworks assists to develop: Clearly understood program/project goals and measurable, long-term, short-term, and intermediate objectives Clearly defined relationships between program/project: inputs, processes, outputs, and outcomes, and between program/project activities and the external context (environmental factors) Sound implementation and M&E plans

Conceptual Frameworks

Conceptual Frameworks (aka. research or theoretical frameworks) What is a conceptual framework? Diagram that identifies and illustrates the relationships between all relevant systemic, organizational, individual, or other salient factors that may influence program/project operation and the successful achievement of program or project goals M&E Purpose: To show where program fits into wider context To clarify assumptions about causal relationships To show program components will operate to influence outcomes To guide identification of indicators To guide impact analysis (causal pathways) Similar frameworks: Proximate Determinants

Conceptual Frameworks Individual characteristics Service utilization Program supply Technical inputs Institutional capacity Healthy practices Health status Program sustainability

Conceptual Framework: Malaria Burden External factors: • Environmental (ecological, climate) • Socio-economic (economic status, movement, occupation, housing condition, war, population displacement, etc) • Demographic ( age, immunity, gender) Health care system: Accessibility Affordability Quality of care Efficiency Demand/utilization Program factors: • Health policy • Antimalarial drug policy • Support/partnership • National MCP Malaria infection Prevention: • LLINs, IRS, IPT • Environmental management Treatment: Early diagnosis & treatment Malaria knowledge: • Cause • Prevention methods • Early treatment • Cultural beliefs • Information Malaria morbidity Malaria mortality

Conceptual Framework: Malaria Control and Elimination

Activity Get into country groups for 30 minutes Review your Problem Statement Develop a conceptual framework for your malaria control program Identify factors that influence malaria control and elimination Arrange factors into a conceptual framework

Outputs from Activity Reviewed Problem Statement A list of factors influencing malaria control and elimination A conceptual framework for your malaria control/elimination program

Goals and Objectives

Goals and Objectives What is the difference between goals and objectives? Goal: a broad statement of a desired, longterm outcome of the program Objectives: statements of desired, specific, realistic and measurable program results

Goals 1. To reduce malaria incidence and to significantly reduce deaths due to malaria by the end of 20115 2. Through the attainment of a 75% reduction, malaria control will ultimately contribute to the reduction of all-cause mortality by 20% in children under five by 2020

SMART Objectives SMART Specific: identifies concrete events or actions that will take place Measurable: quantifies the amount of resources, activity, or change to be expended and achieved Appropriate: logically relates to the overall problem statement and desired effects of the program Realistic: Provides a realistic dimension that can be achieved with the available resources and plans for implementation Time-based: specifies a time within which the objective will be achieved

Class Activity: Are these objectives SMART? 1. To ensure that at least 80% of people sleep under LLINs in every district 2. At least 80% of pregnant women have access to the package of interventions to reduce the burden of malaria in pregnancy by December 2008 3. At least 85% of people sleep in sprayed structures in eligible areas of the 36 selected districts by December 2008, an upward revision from the 15 initially planned districts in the 2006 -2010 National Malaria Strategic Plan 4. 100% of malaria patients in all districts are receiving prompt and effective treatment according to the current drug policy within 24 hours of onset of symptoms by next month

Small group work Get into your groups for 30 minutes and begin to develop goals and objectives for your program (First review your problem statement and Conceptual Framework)

Results Frameworks

Results Frameworks What is a results framework? Diagram that identifies steps or levels of results and illustrate the causal relationships linking all levels of a program’s objectives. Purposes: Provides a clarified focus on the causal relationships that connect incremental achievement of results to the comprehensive program impact Clarifies project/program mechanics and factors’ relationships that suggest ways and means of objectively measuring the achievement of desired ends Other terms used: Strategic frameworks

Results Frameworks Goal: Improved Health Status Strategic Objective: Improved Use of Health Services or Practices IR 1: Access/Availability IR 2: Quality IR 3: Sustainability IR 4: Demand IR 1. 1: Commodities/ Facilities IR 2. 2: Provider Performance IR 3. 1: Policy IR 4. 1: Attitude IR 1. 2: Equity IR 2. 3: Training/Supervision IR 3. 2: Health Care Finance IR 4. 2: Knowledge IR 2. 4: Information System IR 3. 3: Private Sector IR 4. 3: Community Support

Results Framework: Malaria Control Program SO 1: Reduced Malaria Burden IR 1: Improved malaria prevention IR 2: Improved malaria epidemic prevention & management IR 1. 1 Access to & coverage by ITNs increased IR 2. 1 Early detection & appropriate response improved IR 1. 2 Improved access to IPT IR 2. 2 Epidemic preparedness improved IR 1. 3 IRS coverage increased in epidemic prone areas IR 2. 3 Surveillance system improved IR 1. 4 Use of source reduction/ larviciding increased IR 2. 4 Early warning system strengthened IR 3: Increased access to early diagnosis & prompt treatment of malaria IR 3. 1 Quality of care improved IR 3. 2 Efficiency in service delivery improved IR 3. 3 Utilization of care improved IR 3. 4 Access to services improved

Results Frameworks Assist you in • Identifying/selecting appropriate interventions to achieve strategic objectives • Defining results/intermediate/operational/ outcome objectives • Revisit the intervention you have selected, update them and discuss with your Supervisors

Logic Models

Logic Models What is a Logic Model? Diagram that identifies and illustrates the linear relationships flowing from program inputs, processes, outputs, and outcomes. Inputs or resources affect Processes or activities which produce immediate results or Outputs, ultimately leading to longer term or broader results, or Outcomes. Purposes: Provides a streamlined interpretation of planned use of resources and desired ends Clarifies project/program assumptions about linear relationships between key factors relevant to desired ends Other terms used: M&E Frameworks, Logical Frameworks

Components of Logic Model • The logic model has 5 essential components Logic Model Component Inputs Processes Outputs Outcomes Impacts Description The resources we invest in an intervention The activities carried out in order to achieve the intervention’s objectives The immediate RESULTS achieved by the intervention at the intervention level through the execution of activities The set of results at the (target) population level induced by the interventions. People level effect The long term effects or end results of the intervention eg. changes in health status

Logic Models: ITN Distribution INPUT PROCESS OUTPUT OUTCOME IMPACT Human and financial resources • Establish distribution points for ITNs distributed ITN use Malaria prevalence malaria ITNs • BCC staff trained BCC materials BCC conducted

Logic Model: Malaria Control Programs Inputs • Strategies • Policies • Guidelines • Funding • Materials • Facilities • Commodities • Supplies • Staff Process • Training • Services • Education • Treatments • Interventions Outputs • Services delivered • Knowledge, skills, practice • ITNs distributed • HH sprayed • IPTs delivered • Antimalarials delivered • RDTs/slides delivered and reads taken Adapted from: RBM. 2008 Global Malaria Action Plan. Available at: http: //www. rollbackmalaria. org/gmap. pdf Outcomes • IITN/IRS ownership • ITN use • Treatment based on parasitological confirmation Impact • Malaria Incidence/ Prevalence • Mortality • Socioeconomic wellbeing • Economic Impact

Logical Frameworks

Logical Frameworks What is a logical framework? A table which presents a standardized summary of the project and its logic. Purposes: Summarizes what the project intends to do and how Summarizes key assumptions Summarizes outputs and outcomes that will be monitored and evaluated Other terms used: Logframe matrix

Logical Frameworks Project Description Performance Indicators Means of Verification Assumptions Goal: The broader development impact to which the project contributes - at a national and sectoral level. Measures of the extent to which a sustainable contribution to the goal has been made. Used during evaluation. Sources of information and methods used to collect and report it. Purpose: The development outcome expected at the end of the project. All components will contribute to this Conditions at the end of the project indicating that the Purpose has been achieved and that benefits are sustainable. Used for project completion and evaluation. Sources of information and methods used to collect and report it. Assumptions concerning the purpose/goal linkage. Component Objectives: The expected outcome of producing each component's outputs. Measures of the extent to which component objectives have been achieved and lead to sustainable benefits. Used during review and evaluation. Sources of information and methods used to collect and report it. Assumptions concerning the component objective/purpose linkage. Outputs: The direct measurable results (goods and services) of the project which are largely under project management's control Measures of the quantity and quality of outputs and the timing of their delivery. Used during monitoring and review. Sources of information and methods used to collect and report it. Assumptions concerning the output/component objective linkage. Activities: The tasks carried out to implement the project and deliver the identified outputs. Implementation/work program targets. Used during monitoring. Sources of information and methods used to collect and report it. Assumptions concerning the activity/output linkage.

Logical Frameworks: Malaria Control Program Project Description Performance indicators Means of verification Assumptions Goal: To reduce malaria morbidity and mortality by 50% by 2015 • Malaria prevalence • All-cause under –five mortality rate • Annual reports • Surveys • DSS (INDEPTH) • DHS • Political stability • Environmental stability (no natural disasters) Overall Objective: To achieve universal coverage of malaria control interventions % of individuals with access • Annual to an LLIN in their HH reports • Surveys • Record reviews Component Objective: • To achieve universal coverage and increase use of LLINs % of individuals with access to an LLIN in their HH % of individuals who slept under an LLIN the previous night • Populationbased survey • Health facility surveys • Community surveys • Availability of effective and affordable drugs • Strong political support • Availability of LLINs

Logical Frameworks: Malaria Control Program Project Description Performance Indicators Means of Verification Outputs: LLINs Distributed to target population Number of LLINs distributed to target population LLIN hang up demonstrated to individuals in target population Number of individuals who observed LLIN hang up demonstration Activity Funds available reports/program for distribution records and communication campaign Activities: LLIN Distribution Campaigns Number of LLIN distribution campaigns LLIN hang up demonstrations Number of LLIN hang up demonstrations Assumptions Community support Activity • Funds available reports/program for distribution records and communication campaign

Discussion 1. Does your program use any of these frameworks for planning? How? 2. What are other ways that these frameworks can be used for planning?

Frameworks for M&E Planning Purposes: • Clarifying assumptions, goals, and interrelationships between factors relevant to the project or program • Defining objectives • Selecting activities • Defining levels of performance and desired results in terms of planned activities and realistic, objective impacts Monitoring and evaluation plans incorporate: • Program managers’ assumptions and objectives, in a given context • A schematic design displaying the directional linkages between key program elements and/or planned results, and other relevant factors

Summary of Frameworks Type of Framework Brief Description Program Management Basis for Monitoring and Evaluation Conceptual Interaction of various factors Determine which factors the program will influence No. Can help to explain results Results Logically linked program objectives Shows the causal relationship between program objectives Yes – at the objective level Logically linked Shows the causal program objectives, relationship between outputs, and activities and objectives Yes – at the output and objective level Logic model Logically links inputs, processes, outputs, and outcomes, Yes – at all stages of the program from inputs to process to outputs to outcomes/ objectives Shows the causal relationship between inputs and the objectives

References Aus. Guide. The Logical Framework Approach. http: //www. ausaid. gov. au/ausguidelines/1 -1 -1. cfm Bertrand, Jane T. , Magnani, Robert J, and Rutenberg, Naomi, 1996. Evaluating Family Planning Programs, with Adaptations for Reproductive Health, Chapel Hill, N. C. : The EVALUATION Project. Global AIDS Program. (2003) Monitoring and Evaluation Capacity Building for Program Improvement Field Guide, Version 1. U. S. Centers for Disease Control and Prevention, Atlanta, GA. Ministry of Health, Zambia. 2008 National Malaria Control Action Plan: Actions for Scale-up for Impact on Malaria in Zambia. http: //www. nmcc. org. zm/files/2008 Zambia. NMCPAction. Plan-5 -20. pdf Marsh, David. 1999. Results Frameworks & Performance Monitoring. A Refresher by David Marsh (ppt) http: //www. childsurvival. com/tools/Marsh/sld 001. htm Measurement, learning and evaluation project for the urban reproductive health initiative. Measuring success toolkit. Tsui, Amy. 1998. Frameworks (ppt). Presented at the Summer Institute, University of North Carolina, Chapel Hill. Tsui, Amy. 1999. Frameworks (ppt). Presented at the Summer Institute, University of North Carolina, Chapel Hill. UNICEF. 1998. State of the World’s Children.

Group Projects 1. Get into your groups 2. For your malaria control program final project, complete the following for your final presentation: – Problem Statement – Conceptual framework – Goals and objectives with targets – The Logic model, results framework, or Logical Framework

MEASURE Evaluation is a MEASURE program project funded by the U. S. Agency for International Development (USAID) Through Cooperative Agreement GHA-A-00 -08 -00003 -00 and is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill, in partnership with Futures Group International, John Snow, INC. , ICF International, Management Sciences for Health, and Tulane University. VISIT US ONLINE AT HTTP: //WWW. CPC. UNC. EDU/MEASURE.