Monitoring and Evaluation Calculating and Interpreting Coverage Indicators

Monitoring and Evaluation: Calculating and Interpreting Coverage Indicators

Learning Objectives By the end of the session, participants will be able to: • Identify sources of data for calculating coverage indicators • Estimate denominators for routine coverage estimates • Calculate and interpret coverage indicators from routine data • Use online resources for estimating coverage indicators • Assess the quality of relevant data sources • Reconcile coverage estimates from different data sources

Maternal Health Coverage Indicators • Proportion of pregnant women who received at least two antenatal care visits • Proportion of deliveries occurring in a health facility • Proportion of deliveries with skilled attendant at birth • Proportion of women attended at least once during postpartum period (42 days after delivery) by skilled health personnel for reasons related to childbirth

Why Coverage Indicators Are Important • Understand how effective program is • See if one target group is reached more effectively than another • Identify underserved area/regions

Child Health Coverage Indicators • Immunization Programs – DTP 3 vaccine coverage – Measles vaccine coverage – BCG vaccine coverage – OPV 3 coverage – Hep. B 3 coverage – Fully immunized child • Nutrition programs? • Control of diarrheal disease programs?

Coverage Indicators for HIV/AIDS Care & Treatment Programs • Number of clients receiving public/NGO VCT services • Number of clients provided with ARVs • Percent of children in need receiving cotrimoxazole prophylaxis • Percent of HIV patients receiving DOTS • Coverage of PMTCT programs?

Where Do We Get the Data? • • • Censuses Surveys Registrations Health management information systems Program statistics Patient registers

Estimating Coverage From Routine Data

Indicators From Program Statistics: Numerators • HMIS and routine reports give information on numerators • Numerators: number of deliveries in health facilities, measles vaccinations, pills distributed, voluntary counseling and testing clients etc. • Denominators: ?

Example: Importance of denominator • Town A vaccinated 200 infants • Town B vaccinated 400 infants • Town C vaccinated 600 infants • Population size: – Town A= 10, 000 – Town B= 30, 000 – Town C= 60, 000

Indicators From Program Statistics: What Denominators Are Needed? • Denominators: population composition – Population composition – How many women are of childbearing ages? – How many children are under five? – How many adolescents? 15 -19? 20 -24? – How many men are 15 -59 years? – How many children are of school going age? – How many infants are there? – How many babies are born each year?

How Do We Get Denominators? • • • Population registers Censuses Population projections Population growth rate (r) Rate of natural increase = crude birth rate (CBR) minus the crude death rate (CDR) Net migration rate: inmigration - outmigrants per 1000 population CBR: no. of births per 1000 population in 1 year CDR: no. of deaths per 1000 population in 1 yr Population growth = rate of natural increase + net migration rate

Spectrum Model • Dem. Proj: projects population of country/region by age and sex based on assumptions about fertility, mortality, and migration – Urban and rural population projections can also be prepared • Easy. Proj: supplies data needed to make a population projection from estimates provided by the Population Division of the UN www. tfgi. com

Spectrum

Calculating Denominators • Population at time t: P(t) = P(0) * exp(r*t), where: – P(t) is the population size after t years – P(0) is the population size at the last census • Example: – 300, 000 people at census – Growth rate = 3% (0. 03), – What is the population after 10 years? – 404, 958 people

Estimating Number of Live Births • Where data on the number of live births are unavailable: Total expected births = Total population x crude birth rate • Where the crude birth rate (CBR) is unknown: Total expected births = Total population x 0. 035 Source: WHO 1999 a; WHO 1999 b

Estimating Number of Surviving Infants • Target population for childhood immunization: Surviving infants <12 months of age in a year • Where data on the number of surviving infants are unavailable: Total expected number of surviving infants = Total population x CBR x (1 – infant mortality rate)

Estimating Number of Surviving Infants: CBR Known Total population: 5, 500, 000 CBR: 30/1000 Infant mortality rate (IMR): 80/1000 Number of surviving infants = Total population x CBR x (1 – IMR) = 5, 500, 000 x 30/1000 x (1 - 0. 080) = 5, 500, 000 x 0. 030 x 0. 920 = 151, 800 Source: Immunization Essentials: A Practical Field Guide (USAID, 2003)

Estimating Number of Surviving Infants: CBR Unknown • Where data on the number of surviving infants, CBR or IMR are unavailable, multiply total population by 4%: Expected no. of surviving children < 12 months = Total population x. 04 • If the total population is 30, 000, then the number of children under one year = 30, 000 x 4/100 = 1200 Source: WHO, 2002 b

Estimating the Monthly Target Population Monitoring immunization and vitamin A coverage should be done monthly at the facility and district levels, requiring estimations of the monthly target population Monthly target population = Estimated number of children under 1 year of age divided by 12 Example: • Annual target population of children < 12 months = 1200 • Monthly target = 1200/12 = 100

Example: Immunization Coverage From Routine Data • • • Total population of district in 1990 = 99, 000 CBR = 40 per thousand IMR = 80 per thousand Population growth (r) = 3% per year 3, 000 measles vaccinations were given to infants in district in 1998 • What is the measles coverage rate for 1998? – Numerator: No. immunized by 12 months in a given year – Denominator: Total no. of surviving infants < 12 months in same year

Immunization Coverage From Routine Data: Answer • Estimate district total population in 1998 Pop 1998 = 99, 000 * exp(. 03*8) = 125, 854 • Estimate number of surviving infants in 1998 125, 854 x (40/1000) x (1 -. 080) = 4615 • Estimate measles coverage rate Measles coverage = 3000/4615 x 100 = 65%

Challenges in Estimating Coverage from Routine Data • Limited knowledge of target pop/denominators • Low timeliness & completeness of reporting • Poor data quality – Lack of written standard reporting procedures – No systematic supervision on data management • Dual reporting systems (EPI, HMIS) • Inclusion of data from private sector

Assessing Reliability of Routine Coverage Indicators • Understand how denominators are derived • Understand the process of collecting the information • Look for inconsistencies and surprises

Assessing Reliability of Routine Coverage Indicators • Look for reliable data from other sources to use as a basis for comparison • Cross-check

Estimating Coverage from Survey Data

Survey Tools for Coverage Estimation • WHO-EPI surveys • Lot quality coverage surveys • Large-scale population-based surveys • • USAID Demographic and Health Surveys UNICEF Multiple Indicator Cluster Survey Arab League PAPCHILD surveys CDC Reproductive Health Surveys • Seventy-five household survey • Knowledge-Practice-Coverage Surveys • Other local surveys

How Do Administrative Data Compare With Survey Data?

Reconciling Coverage Estimates From Different Data Sources • • Age group & geographic scope Health cards versus recall Different sources for different purposes Not all coverage data can be compared in constructive way • Differences in inclusion of private sector • Selectivity

On-line Resource: STATcompiler • Innovative online database tool • Allows users to select numerous countries and hundreds of indicators to create customized tables that serve specific needs • Accesses nearly all population and health indicators published in DHS final reports http: //www. measuredhs. com/statcompiler

STATcompiler

On-line Resource: DOLPHN • DOLPHN: Data Online for Population, Health and Nutrition • Online statistical data resource • Quick access to frequently used indicators from multiple sources, including: – DHS, BUCEN, CDC, UNAIDS, UNESCO, UNICEF, World Bank, WHO www. phnip. com/dolphn

Advantages and Disadvantages of Routine-based Coverage Advantages • Provides information on more timely basis • Makes use of data routinely collected • Can be used to detect and correct problems in service delivery Disadvantages • Denominator errors • Poor quality reporting

Advantages and Disadvantages of Survey-based Coverage Advantages • Avoids problems with denominators • Includes information from non-reporting facilities Disadvantages • • Coverage survey has low precision Larger standard errors at sub-national levels Irregular and expensive Survey timing may affect coverage rates

References • WHO. 1999 a. Indicators to Monitor Maternal Health Goals: Report of a Technical Working Group, Geneva, 8 -12 November 1993. Division of Family Health Geneva: WHO. • WHO. 1999 b. Reduction of Maternal Mortality: A Joint WHO, UNFPA, UNICEF, World Bank Statement. Geneva: WHO. • WHO (2002) Increasing Immunization at the Health Facility Level. Geneva, Switzerland: World Health Organization

Case Study 1: Immunization Coverage from Facility Data • Estimate total population in 2003 • Calculate coverage for DTP 1, DPT 3, and measles vaccine in 2003 • Evaluate trends in coverage • Estimate drop-out rates • Analyze the problems in 2003 – Is coverage low or falling? – What are possible causes? – What are the differences in coverage in different areas? • What action can managers take if coverage data indicate problems?