Monitoring and Evaluation Indicators ME Indicators Module Objectives

Monitoring and Evaluation: Indicators

M&E Indicators: Module Objectives At the end of the session, participants will be able to: • Critique indicators • Identify criteria for selection of sound indicators • Understand how indicators are linked to the frameworks covered in the Frameworks Module • Select indicators and complete an Indicator Reference Sheet

An Indicator is… • a variable • that measures • one aspect of a program/project or health outcome An appropriate set of indicators includes at least one indicator for each significant aspect of the program or project (i. e. at least one per box in an M&E framework)

Characteristics of Good Indicators • Valid: accurate measure of a behavior, practice or task • Reliable: consistently measurable in the same way by different observers • Precise: operationally defined in clear terms • Measurable: quantifiable using available tools and methods • Timely: provides a measurement at time intervals relevant and appropriate in terms of program goals and activities • Programmatically important: linked to a public health impact or to achieving the objectives that are needed for impact

Characteristics of Good Indicators: Valid Accurate measure of a behavior, practice or task • Indicator measures what it is supposed to measure – Direct measures – Proxy measures – Straightforward interpretation: change in value signals a change in focal concept or behavior

Validity: Class Activity • Is ideal family size a valid indicator of fertility demand? • Is the maternal mortality ratio a valid indicator of the impact of a family planning program on women’s health? • Is “% Willing to buy vegetables from HIV+ shopkeeper” a valid measure of AIDS stigma? • Is “More than one non-marital partner in past year” a valid measure of “risky sex”?

Characteristics of Good Indicators: Reliable Consistently measurable in the same way by different observers • Types of measurement error – Sampling Error: over-representation of urban populations because access is easier – Non-Sampling Error: survey estimates of abortion incidence, due to response bias – Subjective Measurement: indicators that ask for personal judgment such as “quality, ” “environment” and “progress”

Characteristics of Good Indicators: Precise Operationally defined in clear terms Activity: Develop definitions for • New user • Knowledge of AIDS • Quality of care • Trained provider

Characteristics of Good Indicators: Measurable Quantifiable using available tools and methods • Examples of measures – Days of drug stock outs – Total Fertility Rate – Indicators related to Poverty Reduction Strategy

Characteristics of Good Indicators: Timely Provides a measurement over periods of time of interest with data available for all appropriate intervals • Timeliness Considerations – Reporting schedules – Recall periods – Survey schedules – Length of time over which change can be detected

Characteristics of Good Indicators: Programmatically Important Linked to a public health impact or to achieving the objectives needed for impact • Example: Condom distribution program – Indicator: # of sex partners in past 4 weeks • Example: Program to increase access to oral rehydration salt for childhood diarrhea through community based distributors – Indicator: # of ORS packets distributed in past month

Factors to Consider When Selecting Indicators • Logic/link to framework • Programmatic needs/information for decision making • Resources • External requirements (government, donor, headquarters) • Data availability • Standardized indicators

Operationalizing Indicators • To operationalize an indicator is to identify how a given concept or behavior will be measured: • Challenges – Subjective judgment – Local conditions – Unclear yardsticks

Common Indicator Metrics • Counts – Number of providers trained – Number of condoms distributed • Calculations: percentages, ratios – % of facilities with trained provider – Maternal mortality ratio, Total fertility rate • Index, composite measures – Quality index comprising the sum of scores on six quality outcome indicators – DALY (Disability Adjusted Life Years) • Thresholds – Presence, absence – Pre-determined level or standard

Anatomy of an Indicator Metric Indicator 1: # of ANC care providers trained “providers” include any clinician providing direct clinical services to clients seeking ANC at public health facilities “trained” refers to attending every day of a five-day training course offered by the program Indicator 2: % of facilities with a provider trained in ANC Numerator: # of public facilities with a provider who attended all 5 days of ANC training offered by the program Denominator: total # of public facilities providing ANC services

Always Specify the Details! Threshold indicator for post-abortion family planning • Facility provides post-abortion FP if – 90% PAC clients received FP counseling – 90% PAC clients who want no more children were informed about sterilization services – Facility has all of 6 relevant IEC materials

Composite Indicators • How to measure, “Infection Prevention at facilities providing obstetric care” ? Could construct an index, have various measurement options: – “Facility Readiness, ” from facility audit • Has regulation sharps container • Up to date guidelines on hazardous and medical waste disposal • Leak-proof lidded container for medical waste, etc. – Knowledge, from Provider interview – Practice, from Provider-client observation

Issues with Composite Indicators • How to “scale” the index – Weighted v. un-weighted components – Value of each component: yes/no, present/absent, count of sub-components (e. g. , drugs in stock) – Zero/one total indicator value? % of maximum possible score? At least 8 out of 10 positive (=1) scores? – How to interpret and report?

Example of Composite Indicator: Proper Disposal of Medical Waste Percent of Facilities

Sources of Indicators: Using Pre-Defined Indicators • From past years of the program • From related or similar programs • From lists of global or recommended indicators – Millennium Development Goals – HIV/AIDS: UNAIDS/WHO – Reproductive Health: Compendium of Indicators for Evaluating Reproductive Health Programs – Poverty Reduction Strategy Goals (World Bank)

Indicator Pyramid Decreases Global Compare countries Overview world-wide situation National/Sub-national Number of Indicators Increases Assess effectiveness of response Reflect goals/objectives of national/sub-national response District or Facility Identify progress, problems, and challenges

Indicator Matrix Indicator Data Source Frequency Decision Points/Comments Outputs % of facilities with a provider trained in ANC Program records Quarterly Disaggregate by district. Annual Does not provide information about quality of care provided. Outcomes % of pregnant women HMIS and attended at least once during census pregnancy by skilled health personnel for reasons related to the pregnancy

Indicator Reference Sheet Compile detailed documentation for each indicator: • • • Basic information Description Plans for data collection Plans for data analysis, reporting and review Data quality issues Performance data table (baseline and targets)

Indicator Strengths & Limitations All indicators have limitations, even those commonly used: • Low birthweight: cannot use in rural areas where few births are in facilities • Sexual behavior (e. g. condom use, number of partners): self reporting bias • Maternal Mortality Ratio: maternal death is infrequent event • Couple Years Protection: information on program activity but not on number of people benefiting

Setting Indicator Targets: Useful Information Sources • • • Past trends Client expectations Donor expectations Expert opinion Research findings What has been accomplished elsewhere

Common Pitfalls in Indicator Selection • Indicators not linked to program activities • Poorly defined indicators • Indicators that do not currently exist and cannot realistically be collected • Process indicators to measure outcomes & impacts • Indicators that are not very sensitive to change • Too many indicators

Pitfalls with Selecting Indicators Indicator not linked to program activities IR: Expanded access to AIDS prevention & treatment services Activities: train providers in current clinical protocols Inappropriate Indicator: % of facilities with adequate conditions to provide care Better indicators: # of clinicians trained, # of facilities with a trained provider The program is not aiming to affect facility conditions, only provider skills.

Pitfalls with Selecting Indicators Indicator poorly defined Activity: radio campaign to dispel myths about AIDS transmission and prevention Inappropriate Indicator: % of population with AIDS knowledge Better indicators: % of population with no incorrect beliefs about AIDS (defined as: % correctly rejecting the two most common local misconceptions about AIDS and who know that a healthy-looking person can transmit AIDS) AIDS knowledge does not indicate what is desirable: knowledge that AIDS exists? How it is transmitted?

Pitfalls with Selecting Indicators Data needed for indicator not available Inappropriate Indicator: % of days per quarter that service delivery points have stock-out of drugs Data issue: Information on stock-outs may not be collected daily Better indicators: % of service delivery points that had a stock out of drugs at some time during the last quarter If relying on routine data, indicator definition must depend on how data are collected.

Pitfalls with Selecting Indicators Indicator does not accurately represent desired outcome IR: Expanded access to antiretroviral treatment (ARVs) for pregnant women to prevent mother to child transmission of HIV Inappropriate Indicators: % of women on ARVs who are pregnant; % of people on ARVs who are pregnant women Better indicator: % of HIV positive pregnant women who are on ARVs What does it mean if inappropriate indicators increase? Decrease? Do they reflect the desired program effect?

Indicator systems -- How much is enough? Rule of thumb • At least one or two indicators per key activity or result (ideally, from different data sources) • At least one indicator for every core activity (e. g. , training, BCC, CSM) • No more than 8 -10 indicators per area of significant program focus • Use a mix of data collection strategies/source

Not everything that can be counted counts, and not everything that counts can be counted. Albert Einstein

References • Bertrand, Jane T. , Magnani, Robert J, and Rutenberg, Naomi, 1996. Evaluating Family Planning Programs, with Adaptations for Reproductive Health, Chapel Hill, N. C. : The EVALUATION Project. • Bertrand, Jane T. and Escudero Gabriela, 2002. Compendium of Indicators for Evaluating Reproductive Health Programs, vols. 1 and 2, Chapel Hill, N. C. : MEASURE Evaluation. • Tsui, Amy. 1998. Frameworks (ppt). Presented at the Summer Institute, University of North Carolina, Chapel Hill. • Tsui, Amy. 1999. Frameworks (ppt). Presented at the Summer Institute, University of North Carolina, Chapel Hill. • UNICEF. 1998. State of the World’s Children. • USAID/Tanzania Country Strategic Plan, 2005 -2014. • WHO, 1999. The Evolution of Diarrhoeal and Acute Respiratory Disease Control at WHO: Achievement 1980 -1995 Research, Development and Implementation (WHO/CHS/CAH/99. 12).

Group Project • Form groups • For your project: – Identify indicators and define metrics – If frameworks are not finished, continue working on frameworks – For two indicators, complete indicator reference sheet