ROUTINE HEALTH INFORMATION SYSTEMS A Curriculum on Basic
ROUTINE HEALTH INFORMATION SYSTEMS A Curriculum on Basic Concepts and Practice MODULE 5: RHIS Data Analysis SESSION 2: Overview of Steps 1 -4 of Data Analysis The complete RHIS curriculum is available here: https: //www. measureevaluation. org/our-work/ routine-health-information-systems/rhis-curriculum 1
Learning Objectives and Topics Covered Objectives By the end of this session, participants will be able to: • Select appropriate indicators for data analysis • Conduct a basic desk review of data quality and adjust data if necessary • Select appropriate denominators • Compare findings from routine data with findings from other data sources • Analyze routine data to produce information products (tables, graphs, and maps) Topics Covered • Selection of indicators for analysis • Desk review of data completeness and internal consistency • Selection of appropriate denominators • Comparison of findings from routine data with findings from other data sources 2
5 Steps of Data Analysis 1. Select a limited set of core indicators 2. Review data quality 3. Select appropriate denominators 4. Reconcile findings with estimates from other data sources 5. Communicate key findings (to be discussed in Module 5, Session 3) 3
S 1: First Step Select a Limited Set of Core Indicators 4
What Makes an Indicator “SMART”? (Characteristics of Good Indicators) • • • Specific: Indicator is concrete, detailed, focused, and welldefined Measurable: Indicator tells how many or how much and can be measured with identified measurement sources Agreed upon: Stakeholders vested in a specific M&E question should agree that indicator is relevant Relevant: Indicator generates data that can answer the question of interest Time-bound: Indicator specifies time frame of what it is measuring 5
S 1: Select a Limited Set of Core Indicators Core indicators should: • Be based on M&E framework for the national health sector strategy • Be programmatically relevant, and facilitate program management • Reliably and comprehensively assess the performance of the health system (whether national or subnational levels) • Be clearly defined • Have numerators that are measurable with routine health data 6
S 1: Core Indicator with 2 Data Sources: DTP 3 Routine health information systems Numerator: Number of infants immunized with DTP 3 by 12 months of age in a given year Denominator: Total number of surviving infants <12 months of age in same year Population-based survey Numerator: Number of children ages 12– 23 months who received three doses of DTP 3 vaccine by age 12 months Denominator: Total number of children ages 12– 23 months surveyed 7
S 1: Core Indicators Measured Reliably with One Data Source Antiretroviral therapy (ART) retention rate Source: Routine health information systems Numerator: Number of adults and children with HIV, alive and on antiretroviral therapy (ART) 12, 24, 36 months (etc. ) after initiating treatment Denominator: Total number of patients initiating ART during a specific period Fully-immunized child Numerator: Number of children ages 12– 23 months who received 3 doses of OPV, 3 Source: Population-based doses of DTP, and 1 dose each of BCG and measles vaccine before age 12 months survey Denominator: Total number of children ages 12– 23 months surveyed 8
Some Indicators Are Not Clearly Defined or Cannot Be Reliably Measured with Routine Data Some indicators are not clearly defined: • Percentage of health facilities with a skilled provider • Percentage of health facilities with an adequate supply of drugs Some health indicators can be reliably measured with a household survey or health-facility survey but not with data reported routinely by health facilities: • Proportion of population using an improved drinking water source • Proportion of health facilities with treatment guidelines • Infant mortality rate Some indicators are not clearly defined and usually cannot be reliably measured with data reported routinely by health facilities: • % of health facilities with health equipment 9
Practicing the First Step Exercise: Part 1 • Review the list of core health indicators (on the next slide (Handout 5. 2. 1) • Identify which indicators can/cannot be reliably measured with routine data o Can the numerator be defined with routine data? o Do you need additional data sources to measure the indicator? 10
List of Health Indicators 1. Percentage of pregnant women attending antenatal clinics who are screened for syphilis 2. Neonatal mortality rate 3. Number of neonatal tetanus cases 4. Cost of one month’s supply of contraception as a percentage of monthly wages 5. Percentage of infants born protected against neonatal tetanus in a specified period 6. Measles vaccine coverage rate 7. Percentage of registered new and relapse tuberculosis (TB) patients with documented HIV status 8. Percentage of children ages 12– 59 months who were dewormed in the past six months 9. Percentage of HIV-positive infants born to HIV-positive women 10. Maternal mortality ratio 11. Number of health facilities providing comprehensive emergency obstetric care functions per 500, 000 population 12. Exclusive breastfeeding rate 13. TB treatment success rate 14. Percentage of health facilities with systems that support quality service delivery 15. Percentage of districts with current trend analysis for selected priority diseases at a given time (e. g. , month) 11
S 1: Practicing the First Step • Exercise, Part 2 • Review the WHO’s Global Reference List of Core Health Indicators (Handout 5. 2. 3) or the list of standard indicators in Handout 5. 2. 2. • Select five indicators from the list. • For each indicator: o Specify the numerator o Specify the denominator o What is the data source for the numerator? o What is the data source for the denominator? o How can the indicator be interpreted? 12
S 2: Second Step Review Data Quality 13
Step 2: Review Data Quality (see also Module 4) • Completeness and timeliness o Completeness of reports o Completeness of data o Timeliness of reports • Internal consistency o Accuracy o Outliers o Trends o Consistency among indicators • External consistency o Data triangulation o Comparison with data surveys o Consistency of population trends • External comparisons (population denominators) 14
S 2: Demonstration, Excel How to use Excel to: • Create a chart Participants are invited to practice simultaneously. Here are two websites with 5 -minute videos explaining how to create a chart using Excel: https: //www. youtube. com/watch? v=Bcs. Dn. RClz. CY http: //excelcentral. com/excel 2007/essential/lessons/05010 create-a-simple-chart-with-two-clicks. html 15
S 2: Demonstration DHIS 2 How to use DHIS 2 to: • Verify the definition of an indicator o Numerator and denominator • Create a pivot table • Create a chart Participants are invited to practice simultaneously. 16
S 3: Third Step Select Appropriate Denominators 17
S 3: How Do We Get Coverage Denominators? • First, estimate the size of the target population. • Common target populations for health-facility-based indicators: o Total population, children< 5 years, infants, pregnancies, women of reproductive age, live births at health facilities • Size of target populations is often estimated (such as projections/modeled estimates from national population census). • Limitations of estimates: • o Reliability declines with years since last census o Internal migration may make estimates of populations of regions and districts unreliable Some programs may use their own denominators. 18
How Do We Get Denominators? • Document how the denominator value was obtained: o Methods and assumptions used to calculate the denominator o Annual rate of growth if denominators are based on projections of census figures o Present these along with rest of analysis • If service coverage >= 95% and data are of high quality, use ANC 1 or DTP 1 to estimate the number of surviving infants. o Use of service statistics to estimate size of target population can modify conclusions reached about which districts are strong performers and which are weak performers. 19
Estimating Denominators Estimating the number of surviving infants: Total population: 5, 500, 000 Crude birth rate (CBR): 30/1, 000 Infant mortality rate (IMR): 80/1, 000 Number of surviving infants Total population x crude birth rate x (1 - IMR) = 5, 500, 000 x 30/1000 x (1 - 0. 080) = 5, 500, 000 x 0. 030 x 0. 920 = 151, 800 20
Data Quality Check for Denominators • Pregnancies = births + pregnancy loss (2% to 10%); • Surviving infants = births – infant mortality 14000 12000 Number of pregnancies, deliveries, live births, infants Tanzania district example, 2014 12415 10000 11512 11286 11400 11058 Pregnancies, late Deliveries Live births Infants 8000 6000 4000 2000 0 Pregnancies, early 21
S 4: Fourth Step Reconcile Findings with Estimates from Other Data Sources 22
S 4: Reconcile Findings with Estimates from Other Data Sources • Compare data from parallel systems that routinely report the same health events. • Compare estimates from routine health-facility data with estimates from household surveys at the national and regional levels. • Compare available data with statistics that have been officially reported to WHO. 23
S 4: Estimates from Two Sources of Routine Health Facility Data Administrative estimates of 2014 DTP 3 coverage, by district, DPI/JRF versus HMIS/DHIS data 140% DTP 3 -- HMIS (DHIS) 120% 100% Linear(Equal) Linear(Plus 25%) 80% Linear(Minus 25%) 60% . 40% 60% 80% 100% 120% 140% DTP 3 - EPI (JRF) Note: Red marks are for districts with a negative dropout rate, according to EPI/JRF data.
S 4: Compare Estimates from Household Surveys DTP 3 immunization coverage, Tanzania, 2009– 2012 Sources: Routine EPI facility data, 2010 DHS, and 2011 Immunization Coverage Survey 25
Reconciling with Survey Findings Is Not Always Easy 2014 WHO-UNICEF report on trends in DTP 3 coverage in Ethiopia Administrative/official estimates (red stars and circles) versus surveys (vertical red lines) versus WHO-UNICEF estimate (blue line) 26
Reconciling with Survey Findings Is Not Always Easy 2014 WHO-UNICEF report on trends in DTP 3 coverage in India Administrative/official estimates (red stars and circles) versus surveys (vertical red lines) versus WHO-UNICEF estimate (blue line) 27
Small Group Exercise (Steps 1 -4 ) • Distribute Handouts 5. 2. 4 a and b. • Form small groups of 4– 5 participants. • Using Excel and the spreadsheet provided (Penta, 2014 data from Tanzania, Handout 5. 2. 4 B), calculate indicator values by region for: o Penta 1 coverage rate o Penta 3 coverage rate o Penta 1 -Penta 3 dropout rate • For each region, specify whether access is good or poor. • For each region, specify whether utilization is good or poor. • Categorize the immunization problem in each region (if any). • Brainstorm the differences in coverage between regions. • Discuss what action managers can take if coverage and dropout rates indicate problems. 28
P 2: Small Group Exercise, Option B (DHIS 2) • Distribute Handout 5. 2. 6. • Form small groups of 4– 5 participants. • Choose either ANC 4 or DTP 3 as your indicator. • Using DHIS 2, verify the definition of the indicator. • Create a pivot table with indicator values by district. • Graph trends in the completeness of reporting by year for the past four years by: o Facility (hospital versus health center) and o Managing authority (public versus private) • Discuss how the trend in completeness would affect the apparent trend in the indicator. 29
ROUTINE HEALTH INFORMATION SYSTEMS A Curriculum on Basic Concepts and Practice This presentation was produced with the support of the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement AIDOAA-L-14 -00004. MEASURE Evaluation is implemented by the Carolina Population Center, University of North Carolina at Chapel Hill in partnership with ICF International; John Snow, Inc. ; Management Sciences for Health; Palladium; and Tulane University. The views expressed in this presentation do not necessarily reflect the views of USAID or the United States government. 30
- Slides: 30