Indicators and Calculating Coverage indicators ME Indicators For

  • Slides: 64
Download presentation
Indicators and Calculating Coverage indicators

Indicators and Calculating Coverage indicators

M&E Indicators For Malaria Programs

M&E Indicators For Malaria Programs

M&E Indicators: Module Objectives At the end of the session, participants will be able

M&E Indicators: Module Objectives At the end of the session, participants will be able to: 1. Critique indicators 2. Identify criteria for selection of sound indicators 3. Understand how indicators are linked to the frameworks covered in the Frameworks Module 4. Select indicators and complete an Indicator Reference Sheet

What is an Indicator? An Indicator is… – a variable – that measures –

What is an Indicator? An Indicator is… – a variable – that measures – one aspect of a program/project or health outcome An appropriate set of indicators includes at least one indicator for each significant aspect of the program or project (i. e. at least one per box in an M&E framework)

Anatomy of an Indicator Metric Ø Proportion of Households with at Least One ITN*

Anatomy of an Indicator Metric Ø Proportion of Households with at Least One ITN* – Numerator: Number of households surveyed with at least one ITN. – Denominator: Total number of households surveyed. *An ITN is 1) a factory treated net that does not require any treatment, 2) a pretreated net obtained within the past 12 months, or 3) a net that has been soaked with insecticide within the past 12 months.

Common Indicator Metrics Counts – Number of providers trained – Number of ITNs distributed

Common Indicator Metrics Counts – Number of providers trained – Number of ITNs distributed Calculations: percentages, ratios – % of facilities with trained provider – Under 5 mortality rate, case fatality rate, annual blood examination rate (ABER) Index, composite measures – Quality index comprising the sum of scores on six quality outcome indicators – DALY – Wealth index Thresholds – Presence, absence – Pre-determined level or standard – Cut-off point

Characteristics of Good Indicators Valid: accurate measure of a behavior, practice or task Reliable:

Characteristics of Good Indicators Valid: accurate measure of a behavior, practice or task Reliable: consistently measurable in the same way by different observers Precise: operationally defined in clear terms Measurable: quantifiable using available tools and methods Timely: provides a measurement at time intervals relevant and appropriate in terms of program goals and activities Programmatically important: linked to a public health impact or to achieving the objectives that are needed for impact

Characteristics of Good Indicators: Valid Accurate measure of a behavior, practice or task •

Characteristics of Good Indicators: Valid Accurate measure of a behavior, practice or task • Indicator measures what it is supposed to measure – Direct measures – Indirect/Proxy measures – Straightforward interpretation: change in value signals a change in focal concept or behavior

Validity: Class Activity 1. Is parasitemia a valid measure of morbidity? 2. Is fever

Validity: Class Activity 1. Is parasitemia a valid measure of morbidity? 2. Is fever a valid measure for malaria? 3. Is parasite testing a valid measure for parasite prevalence? 4. Is the number of people reached by BCC campaigns a valid measure of malaria knowledge?

Characteristics of Good Indicators: Reliable Consistently measurable in the same way by different observers

Characteristics of Good Indicators: Reliable Consistently measurable in the same way by different observers Types of measurement error – Sampling Error: over-representation of urban populations because access is easier – Non-Sampling Error: survey estimates of bed net use, due to response bias – Subjective Measurement: indicators that ask for personal judgment such as “quality, ” “environment” and “progress”

Characteristics of Good Indicators: Precise Operationally defined in clear terms Activity: Develop definitions for:

Characteristics of Good Indicators: Precise Operationally defined in clear terms Activity: Develop definitions for: Effective treatment Population at-risk Suspected cases of malaria

Characteristics of Good Indicators: Measurable Quantifiable using available tools and methods Are the following

Characteristics of Good Indicators: Measurable Quantifiable using available tools and methods Are the following indicators measurable? 1. 2. 3. 4. Number of ITNs distributed Compliance to antimalarial treatment Anemia Parasitemia

Characteristics of Good Indicators: Timely Provides a measurement over periods of time of interest

Characteristics of Good Indicators: Timely Provides a measurement over periods of time of interest with data available for all appropriate intervals Timeliness Considerations – Reporting schedules – Recall periods – Survey schedules – Length of time over which change can be detected

Characteristics of Good Indicators: Programmatically Important Linked to a public health impact or to

Characteristics of Good Indicators: Programmatically Important Linked to a public health impact or to achieving the objectives needed for impact Ø Are the following indicators programmatically important? Example 1: ITN distribution program – Indicator: # ITNs distributed in past quarter Example 2: Program to increase access to ACTs through community-based health workers – Indicator: Number of ACT sales points with antimalarial drugs

Factors to Consider When Selecting Indicators Link to framework Programmatic needs/information for decision making

Factors to Consider When Selecting Indicators Link to framework Programmatic needs/information for decision making Resources (Time) External requirements (government, health partner, headquarters) Data availability Standardized indicators

Operationalizing Indicators Establish exactly how a given concept / behavior will be measured –

Operationalizing Indicators Establish exactly how a given concept / behavior will be measured – Precise definition and metric – How the value will be reliably calculated – Anyone using the same data will arrive at exactly the same indicator value – Challenges • • Subjective judgment Local conditions Unclear yardsticks Skills of the users

Sources of Indicators: Using Pre-Defined Indicators What are some sources for pre-defined indicators? –

Sources of Indicators: Using Pre-Defined Indicators What are some sources for pre-defined indicators? – Past years of the program – Related or similar programs – Lists of global or recommended indicators • Roll Back Malaria. Guidelines for Core Population-based Indicators, 2009 • Global Fund Indicator Guide • Global Fund performance framework

Indicator Pyramid Global Compare countries Overview world-wide situation National/Sub-national Assess effectiveness of response Reflect

Indicator Pyramid Global Compare countries Overview world-wide situation National/Sub-national Assess effectiveness of response Reflect goals/objectives of national/ sub-national response District or Facility Identify progress, problems, and challenges Decreases Number of Indicators Increases

Indicator Matrix Data Source Frequency Level Decision points/comments Quarterly Facility Disaggregate by district National

Indicator Matrix Data Source Frequency Level Decision points/comments Quarterly Facility Disaggregate by district National To be used to determine where to target ITN distribution and BCC activities Output Number of health Program personnel and Records community health care agents trained in case management Outcome Proportion of children under 5 years old who slept under an ITN the previous night Representative Periodic household (Every 1 -5 survey (ex. years) DHS, MICS, MIS)

Indicator Reference Sheet Compile detailed documentation for each indicator: Basic information Description Plans for

Indicator Reference Sheet Compile detailed documentation for each indicator: Basic information Description Plans for data collection Plans for data analysis, reporting, and review Data quality issues Performance data table (baseline and targets)

Indicator Reference Sheet: Example Name of Indicator DESCRIPTION Rationale Definition of the Indicator: Numerator:

Indicator Reference Sheet: Example Name of Indicator DESCRIPTION Rationale Definition of the Indicator: Numerator: Denominator: Measurement: Frequency: Interpretation: Data Source(s): Strengths: Limitations: THIS SHEET LAST UPDATED ON: 07/18/2011

Indicator Strengths & Limitations All indicators have limitations, even those commonly used: • Household

Indicator Strengths & Limitations All indicators have limitations, even those commonly used: • Household spraying, net impregnation: Recall bias this can result in considerable ‘heaping’ of dates • Net Use: self-reporting bias, seasonality of survey may affect net use

Indicators: How they link to frameworks

Indicators: How they link to frameworks

Logic Model Indicators INPUT PROCESS OUTPUT OUTCOME IMPACT • Human and financial resources •

Logic Model Indicators INPUT PROCESS OUTPUT OUTCOME IMPACT • Human and financial resources • Establish distribution points for ITNs • ITNs sold and distributed • Use of ITNs • Prevalence of malaria • ITNs Indicator: Number of ITNs sold and distributed Indicator: Prevalence of malaria parasite Indicator: Proportion of household infection members who slept under an ITN the previous night

Results Framework Indicators IR-1: Improved malaria prevention IR-1. 1: Access to and coverage by

Results Framework Indicators IR-1: Improved malaria prevention IR-1. 1: Access to and coverage by ITNs increased IR-1. 2: Improved coverage of IPTs IR-1. 3: IRS coverage increased Number of malaria cases Proportion of household members who slept under an ITN the previous night Proportion of women who received IPT during antenatal care visits during their last pregnancy Proportion of households that received spraying through an IRS campaign within the last 12 months

Setting Indicator Targets: Useful Information Sources • Past trends • Client expectations • Donor

Setting Indicator Targets: Useful Information Sources • Past trends • Client expectations • Donor expectations • Expert opinion • Research findings • What has been accomplished elsewhere • International conventions

Common Pitfalls in Indicator Selection • Indicators not linked to program activities • Poorly

Common Pitfalls in Indicator Selection • Indicators not linked to program activities • Poorly defined indicators • Indicators that do not currently exist and cannot realistically be collected • Process indicators to measure outcomes & impacts • Indicators that are not very sensitive to change • Too many indicators

Pitfalls with Selecting Indicators Indicator not linked to program activities IR: Expanded access to

Pitfalls with Selecting Indicators Indicator not linked to program activities IR: Expanded access to malaria treatment services Activities: train providers in current clinical protocols Inappropriate Indicator: % of facilities with adequate conditions to provide care Better indicators: # of clinicians trained, % of facilities with a trained provider v. The program is not aiming to affect facility conditions, only provider skills.

Pitfalls with Selecting Indicators Data needed for indicator not available • Inappropriate Indicator: %

Pitfalls with Selecting Indicators Data needed for indicator not available • Inappropriate Indicator: % of days per quarter that service delivery points have stock-out of drugs • Data issue: Information on stock-outs may not be collected daily • Better indicators: % of service delivery points that had a stock out of drugs at some time during the last quarter v If relying on routine data, indicator definition must depend on how data are collected

Pitfalls with Selecting Indicators Indicator does not accurately represent desired outcome IR: Access to

Pitfalls with Selecting Indicators Indicator does not accurately represent desired outcome IR: Access to effective treatment among children <5 years old with malaria • Inappropriate Indicators: % of children <5 years old who received ACTs; % of people who received ACTs for malaria infection who are children<5 • Better indicator: % of children <5 years old who were diagnosed with malaria in the past 2 weeks who received ACTs v. What does it mean if inappropriate indicators increase? Decrease? Do they reflect the desired program effect?

Indicator systems -- How much is enough? Rule of thumb At least one or

Indicator systems -- How much is enough? Rule of thumb At least one or two indicators per key activity or result (ideally, from different data sources) At least one indicator for every core activity (e. g. , ITN distribution, IRS, training, BCC) No more than 8 -10 indicators per area of significant program focus Use a mix of data collection strategies/source

Choosing the right number of indicators

Choosing the right number of indicators

Good indicators: • Provide information useful for program decision-making • Are consistent with international

Good indicators: • Provide information useful for program decision-making • Are consistent with international standards and other reporting requirements, as appropriate • Are defined in clear and unambiguous terms • Are non-directional, “independent” • Have values that are: – Easy to interpret and explain – Precise, valid and reliable measures – Comparable across relevant population groups, geography, other program factors, as needed

NOT EVERYTHING THAT CAN BE COUNTED COUNTS, AND NOT EVERYTHING THAT COUNTS CAN BE

NOT EVERYTHING THAT CAN BE COUNTED COUNTS, AND NOT EVERYTHING THAT COUNTS CAN BE COUNTED. Albert Einstein

SUMMARY: GUIDING PRINCIPLES TO SELECTING INDICATORS Ensure that the indicators are linked to the

SUMMARY: GUIDING PRINCIPLES TO SELECTING INDICATORS Ensure that the indicators are linked to the program goals and are able to measure change Ensure that standard indicators are used to the extent possible Consider the cost and feasibility of data collection and analysis Keep the number of indicators to the minimum and include only those needed for program and management decisions or for reporting

Calculating and Interpreting Coverage Indicators For Malaria Programs

Calculating and Interpreting Coverage Indicators For Malaria Programs

Learning Objectives By the end of the session, participants will be able to: •

Learning Objectives By the end of the session, participants will be able to: • Identify sources of data for calculating coverage indicators • Estimate denominators for routine coverage estimates • Calculate and interpret coverage indicators from routine data • Use online resources for estimating coverage indicators • Assess the quality of relevant data sources • Reconcile coverage estimates from different data sources

ITN Coverage Indicators • Proportion of households with at least one ITN/LLIN • Proportion

ITN Coverage Indicators • Proportion of households with at least one ITN/LLIN • Proportion of population with access to an ITN within their household • Proportion of households with at least one ITN for every two people • Proportion of the population/children under 5 years old/pregnant women who slept under an ITN/LLIN the previous night

IPTp Coverage Indicator • Proportion of women who received two, three and four or

IPTp Coverage Indicator • Proportion of women who received two, three and four or more doses of intermittent preventive treatment for malaria during their last pregnancy in the last two years

Diagnostics and Treatment Coverage Indicator • Proportion of children under 5 years old with

Diagnostics and Treatment Coverage Indicator • Proportion of children under 5 years old with fever in the last 2 weeks who had a finger or heel stick • Proportion receiving ACTs (or other first line treatment), among children under five years old with fever in the last two weeks who received any antimalarial drugs

IRS Coverage Indicators • Proportion of Households which Received Spraying through an IRS Campaign

IRS Coverage Indicators • Proportion of Households which Received Spraying through an IRS Campaign within the Last 12 Months • Coverage of vector control: Proportion of Households with at least one ITN/LLIN and/or sprayed by IRS in the last 12 months

Why Coverage Indicators Are Important • Understand how effective program is • See if

Why Coverage Indicators Are Important • Understand how effective program is • See if one target group is reached more effectively than another • Identify underserved areas/regions

Estimating Coverage From Routine Data

Estimating Coverage From Routine Data

Indicators for Program: Numerators • HMIS and routine reports give information on numerators •

Indicators for Program: Numerators • HMIS and routine reports give information on numerators • Numerators: number of houses sprayed with IRS, number of LLIN distributed through antenatal care, number of women receiving at least two doses of SP during antenatal care • Denominators: ?

Example: Importance of denominator Numerator • IPTp Provided to – Town A= 200 women

Example: Importance of denominator Numerator • IPTp Provided to – Town A= 200 women – Town B= 400 women Denominator Question: What will be the denominator? Response: Number of women that need IPTp who visited ANC clinics in each town – Town C= 600 women • Number of pregnant women: Question: Can we say that Town C has the highest coverage? Please justify your response. – Town A= 10, 000 Answer: No. We need the denominator for each town – Town B= 30, 000 – Town C= 60, 000

Indicators for program: Denominators • Population that are targeted by given intervention – –

Indicators for program: Denominators • Population that are targeted by given intervention – – – District population Women of childbearing age Pregnant women visiting ANC Children under the age of five Children under 5 years old who had a fever

How Do We Get Denominators? • • • Population registers Censuses Population projections Population

How Do We Get Denominators? • • • Population registers Censuses Population projections Population growth rate (r) Rate of natural increase = crude birth rate (CBR) minus the crude death rate (CDR) • Net migration rate: inmigration - outmigrants per 1000 population • Population growth = rate of natural increase + net migration rate

Estimating population size • Where: – P(t) is the population size after t years

Estimating population size • Where: – P(t) is the population size after t years – P(0) is the population size at the last census – r the annual population growth rate Example: 300, 000 people at census Growth rate = 3% (0. 03), What is the population after 10 years? 404, 958 people • Use the national statistics office project national and sub-national level • Use UN population, World Bank estimates for national level • Use the official figures and only make projections if they are not available

Defining Population at Risk A group of people who share a characteristic that causes

Defining Population at Risk A group of people who share a characteristic that causes each member to be susceptible to a particular event, such as people living in an endemic area who are exposed to malaria • Mid- term population (Mid-year) • Expresses the population at the middle of the year • Person-time • • Estimate of the actual time-at-risk in years, months, or days that all persons contributed to the period/under a particular intervention. Only possible if individuals are follow-up

Mid-Year Population vs. Person-Years Person time Individual 5 3/12=0. 25 Individual 4 5/12=0. 42

Mid-Year Population vs. Person-Years Person time Individual 5 3/12=0. 25 Individual 4 5/12=0. 42 1 Individual 3 7/12=0. 58 1 Individual 2 10/12=0. 83 1 Individual 1 12/12=1 Dec 31 Jan 01 Mid year pop: 1+1+1=3 Person year: 0. 25+0. 43+0. 58+0. 83+1=3. 08

Estimating Target Population • A District has 10, 000 inhabitants in 2009, and 3%

Estimating Target Population • A District has 10, 000 inhabitants in 2009, and 3% are children under 1 year of age – What is the annual target population for ITN distribution for infants? – What is the monthly population for ITN distribution for infants? Answers: Annual target population = 10, 000 x 0. 03 = 300 Monthly target population = 1, 800/12 = 25

Challenges in Estimating Coverage from Routine Data • Limited knowledge of target pop/denominators •

Challenges in Estimating Coverage from Routine Data • Limited knowledge of target pop/denominators • Low timeliness & completeness of reporting • Poor data quality – Lack of written standard reporting procedures – No systematic supervision on data management • Dual reporting systems (EPI, HMIS) • Data from private sector not often included

Assessing Reliability of Routine Coverage Indicators • • Understand how denominators are derived Understand

Assessing Reliability of Routine Coverage Indicators • • Understand how denominators are derived Understand the process of collecting the information Look for inconsistencies and surprises Look for reliable data from other sources to use as a basis for comparison • Cross-check

Estimating Coverage From Survey Data

Estimating Coverage From Survey Data

Tools for Coverage Estimation • Large-scale population-based surveys • Malaria Indicator Surveys - MIS

Tools for Coverage Estimation • Large-scale population-based surveys • Malaria Indicator Surveys - MIS • Demographic and Health Surveys- DHS • Multiple Indicator Cluster Survey- MICS • Post-campaign coverage Surveys • Other local surveys • Lot quality sample coverage surveys

Routine Data vs. Survey Data

Routine Data vs. Survey Data

Reconciling Coverage Estimates from Different Data Sources • • Age group & geographic scope

Reconciling Coverage Estimates from Different Data Sources • • Age group & geographic scope Health cards versus recall Different sources for different purposes Not all coverage data can be compared in constructive way • Differences in inclusion of private sector • Selectivity

On-line Resource: STATcompiler • Innovative online database tool • Allows users to select numerous

On-line Resource: STATcompiler • Innovative online database tool • Allows users to select numerous countries and hundreds of indicators to create customized tables that serve specific needs • Accesses nearly all malaria and population and health indicators published in MIS/DHS final reports http: //www. statcompiler. com

STATcompiler www. statcompiler. com

STATcompiler www. statcompiler. com

Challenges with Routine-based Coverage Advantages: • Provides information on more timely basis • Makes

Challenges with Routine-based Coverage Advantages: • Provides information on more timely basis • Makes use of data routinely collected • Can be used to detect and correct problems in service delivery Disadvantages: Denominator errors Poor quality reporting

Challenges with Survey-based Coverage Advantages • Avoids problems with denominators • Includes community based

Challenges with Survey-based Coverage Advantages • Avoids problems with denominators • Includes community based information Disadvantages Larger standard errors at sub-national levels Irregular and expensive Survey timing may affect coverage rates

GROUP PROJECT Form country groups For your project: Identify 4 -6 indicators based on

GROUP PROJECT Form country groups For your project: Identify 4 -6 indicators based on your framework and define metrics Create an indicator matrix for your indicators If frameworks are not finished, continue working on frameworks For two indicators, complete indicator reference sheet

References Bertrand, Jane T. , Magnani, Robert J, and Rutenberg, Naomi, 1996. Evaluating Family

References Bertrand, Jane T. , Magnani, Robert J, and Rutenberg, Naomi, 1996. Evaluating Family Planning Programs, with Adaptations for Reproductive Health, Chapel Hill, N. C. : The EVALUATION Project. Bertrand, Jane T. and Escudero Gabriela, 2002. Compendium of Indicators for Evaluating Reproductive Health Programs, vols. 1 and 2, Chapel Hill, N. C. : MEASURE Evaluation. Roll Back Malaria. 2009. Guidelines for core population-based indicators. January 2009. MEASURE Evaluation: Calverton, MD. Tsui, Amy. 1998. Frameworks (ppt). Presented at the Summer Institute, University of North Carolina, Chapel Hill. Tsui, Amy. 1999. Frameworks (ppt). Presented at the Summer Institute, University of North Carolina, Chapel Hill. UNICEF. 1998. State of the World’s Children. USAID/Tanzania Country Strategic Plan, 2005 -2014. WHO, 1999. The Evolution of Diarrhoeal and Acute Respiratory Disease Control at WHO: Achievement 19801995 Research, Development and Implementation (WHO/CHS/CAH/99. 12).

MEASURE EVALUATION IS A MEASURE PROGRAM PROJECT FUNDED BY THE U. S. AGENCY FOR

MEASURE EVALUATION IS A MEASURE PROGRAM PROJECT FUNDED BY THE U. S. AGENCY FOR INTERNATIONAL DEVELOPMENT (USAID) THROUGH COOPERATIVE AGREEMENT GHA-A-00 -08 -00003 -00 AND IS IMPLEMENTED BY THE CAROLINA POPULATION CENTER AT THE UNIVERSITY OF NORTH CAROLINA AT CHAPEL HILL, IN PARTNERSHIP WITH FUTURES GROUP INTERNATIONAL, JOHN SNOW, INC. , ICF MACRO, MANAGEMENT SCIENCES FOR HEALTH, AND TULANE UNIVERSITY. VISIT US ONLINE AT HTTP: //WWW. CPC. UNC. EDU/MEASURE.