Monitoring and Evaluation Learning Objectives By the end
Monitoring and Evaluation
Learning Objectives By the end of the webinar, participants will be able to… o Distinguish between Monitoring and Evaluation and how they are used; o Differentiate goals, objectives, and activities; and impact, outcomes, and outputs; o Frame good indicators o Describe the SPARC approach
What’s M&E? o Monitoring and Evaluation is a systematic process for managing programs by tracking and producing results. o As a management tool, M&E tells us whether a program should continue, be revised, stopped, scaled-up, or replicated. o It communicates to key stakeholders about project achievements and progress. o It ensures resources are used and expended as planned (i. e. accountability)
Monitoring Defined o Refers to the routine process of collecting data about a program’s activities o It looks at the efficiency and quality of program inputs and services (timeliness, adequacy, comprehensiveness), as well as participant satisfaction. o Usually involves counting what a program is doing
Typical Monitoring Questions o Is the target population being reached? o Are activities carried out as planned? o What services are provided to whom, when, how often, for how long, and in what context? o Are the services accessible? o Is the quality of services outstanding?
Evaluation Defined o Refers to the use of social research methods to investigate the effectiveness of a program. o Evaluation sometimes requires a control or comparison group o Evaluation involves measurements over time (e. g. baseline and follow-up).
Typical Evaluation Questions o o o What outcomes are observed in the target group/s? Does the program make a difference? To what extent is the program responsible for the observed changes?
Focus of M&E Goals Objectives Activities/ Inputs Focus of Monitoring Outputs Outcomes Focus of Evaluation Impacts
Core of a Good M&E
Goals • A compelling vision of success, broad in scope, and paints a vivid picture of the world we would like to create. • General statement that describes the hoped-for result of a program for a wider population (i. e. , impact) • e. g. , reduced HIV-related disparities and health inequalities, a hunger-free world, enduring democracies, etc. • Goals are achieved over a long time horizon and through the combined efforts of multiple programs. • Goals = Impact
Objectives • Stated in terms of changes (i. e. , outcomes) to be achieved for target stakeholders (e. g. knowledge, attitude, skills, behaviour, etc. ) • SMART (Specific , Measurable, Achievable, Relevant, Time-bound) • Objectives = Outcomes
Examples GOOD: By the end of the project (time-bound), selected women of reproductive age in rural Nepal (specific) will have improved their knowledge on contraceptive methods and have reduced unplanned pregnancies. (measurable, achievable, relevant). BAD: The project seeks to conduct trainings with women of reproductive age in rural Nepal (This is an activity. To turn this into an objective, ask why trainings need to be conducted. So what? So that what…? )
Logic Model o A logic model conceptualizes how a program is understood to contribute to a chain of results (i. e. outputs, outcomes, impacts). o It follows a cause-and-effect and if-then logic (i. e. if we do this, then this will be the result)
Logic Model Template Goal: A compelling vision of success. Concrete, yet broad in scope, it paints a vivid picture of the world the program seeks to create. Objectives: A concrete statement of what the program desires to see change among its target groups within a given timeframe. Inputs Activities Outputs Short-term Outcome Medium-term Outcome Longer-term Outcome or Impact Identify resources (e. g. staff expertise, funding, social capital and linkages, equipment etc. )—the organization has to run the program. Identify the different actions, events the program will undertake to meet its objectives. Identify the specific products the program will produce in relation to its activities. Identify the changes in attitudes, perceptions, knowledge or understanding of your primary target group/s. Identify the changes in behaviour or practices of your primary target group/s. Identify the ultimate change (populationlevel) the program seeks to contribute to achieving.
What are Indicators? Impacts Outcomes Indicators Outputs Equity Effectiveness Efficiency Activities Inputs Economy Source: National Treasury (2007) Framework for Managing Programme Performance Information Indicators are also alternately referred to as measures and metrics. They are signals of change that give a simple and reliable means to reflect the changes associated with the program.
Quantitative and Qualitative Indicators Quantitative indicators measure quantities or amounts. Examples: o Number of household heads enrolled in a program o Percent of participants who passed an online exam Qualitative indicators measure judgments and perceptions of people about a given topic. May contain a number but they relate to an opinion, belief, or way of thinking Examples: o Level of confidence of GPP online course participants in identifying essential characteristics of GPP in clinical/research trials o Women’s perception about their engagement in community decisionmaking processes o Number of women who rate their community engagement positively
Elements of an Indicator Basic Elements: o What will be measured – what is going to change o Unit of measurement used to describe change (e. g. , number, percent, quality, level, etc. ) o Target groups or population Optional Elements: o Pre-program status or baseline, where possible o Size and magnitude or dimension/target of the intended change o Quality or standard of the change to be achieved o Timeframe
Example of a Detailed Indicator Basic: The number of women’s groups participating in at least 3 consultation meetings with health policy makers Optional: …from 4 women’s groups in 2016 to at least 20 women’s groups by the end of the program What is measured? The number of women’s groups participating in at least three consultation meetings. Unit of measurement? Number of women’s groups Baseline? From 4 women’s groups in 2018 Size/Magnitude/Target? At least 20 women’s groups by the end of the program Quality or Standard of Change? At least 3 consultation meetings Target Population? Women’s groups Timeframe? Between 2016 and the end of the program
Characteristics of Good Indicators Clear Described in concrete, operational language Relevant Tightly linked to essential variables that drive performance Economical Affordable to measure Adequate Sufficient for the collection of essential performance data Monitorable Measurable within capacities of the organization itself, not needing external evaluators
Data Collection o Quantitative (Random of Purposive Sampling) o o o o Qualitative (Random or Purposive Sampling) o o o o Project records, service statistics Observation (with standardized checklist) Mini-surveys / exit interviews Community-based KAP surveys Census Chart or record review Key-informant interviews Focus groups Participant observation Visual Anthropology (photography and film) Community mapping Document review Participatory and Mixed Methods
Simple Intervention Vision & Mission IMPACT OUTCOMES OUTPUTS Plan ACTIVITIES INPUTS Time
Complex Intervention Vision & Mission OUTPU T E COM T U O OUTPUT OUTCOME OUTCOM E Implementing GPP Stakeholder Engagement ACTIVIT Y INPUT S Plan INPUT OUTCOME ACTIVIT Y OUTCOME OUTPUT Time OU TC OM E OUT E COM
SPARC What is SPARC? o Simple: Jargon-free, accessible, maybe even fun o Participatory: All coalition members contribute o Assessment: Qualitative evaluation process o Real Change: Captures outcomes
SPARC and Outcome Harvesting o SPARC is a very simple version of Outcome Harvesting, a qualitative evaluation approach. o It’s a response to a need to capture the richness and range of changes in complex program environments.
What is the starting point of SPARC? o Attribution: Traditional M&E normally tracks changes by looking at a set of project activities and then situating the change that was produced within those activities. o Contribution: SPARC begins at the very end (what has changed? ) and then confirms/ negates contribution
What do we mean by outcomes? o What a social actor did, or is doing, that reflects a significant change in their behavior, relationships, activities, actions, policies or practice, which in turn may have implications for the project’s goals and objectives o Outcomes are not about what a project did (i. e. , activities) but about the effects of those actions on its stakeholders and the field it seeks to influence.
SPARC Guiding Question o Setting aside whether your project may have had anything to do with these changes, what do you think have been the most significant signs of progress related to our project goals and objectives?
SPARC Story Guide Questions o What happened? Basic description of the outcome, including what changed, who was involved, and where and when it happened o Significance? Why the change is relevant to the project’s goals and objectives or the challenge and barrier it helps address o Contribution? How has the project influenced, directly or not, the change o Evidence? Plausible documentation of change and contribution
- Slides: 28