A Comparison of Different Methods of Data Collection

A Comparison of Different Methods of Data Collection Health System Innovations Workshop Abuja, Jan. 25 -29, 2010

Three Pillars of M&E 1. Setting Objectives and selecting Indicators √ 2. Collecting data using appropriate methods 3. Using the data to make decisions and improve performance 2

Important Methods Different methods of data collection required, each with benefits & issues 1. Routine reporting system (HMIS): reported data coming from service providers 2. Household surveys (HHS): statistical sample of households in the community 3. Health Facility Assessments (HFA): a survey of a sample of health facilities to assess QOC 4. Quantitative supervisory checklist 3

1. Routine Reporting - HMIS Advantages • Usually near real time • Highly Disaggregated • Local managers can their manage own performance Disadvantages • often inaccurate • service providers have incentive to overstate • difficult to provide data on equity, quality • can’t assess effects at community level • Expensive (HW time) 8 x cost of HHS • Misses info from private sector 4

Excessive & Exclusive Reliance on HMIS is Fraught with Danger • Murray et al have found that GAVI ISS may have over-paid by 93% • Data Quality Audit (DQA) was no help in discovering discrepancies between HMIS & surveys • People want to tell their superiors good news • May worsen when tied to some form of bonus 5

HMIS vs. Household Survey DPT 3 Coverage in Pakistan Average difference = 14% points 6

2. Household Surveys Advantages • More accurate • Can be independent of service providers • Provides data on coverage, equity, expenditure, CPR, ITN use, satisfaction, etc. Disadvantages • Relatively expensive (cheaper than HMIS but an incremental cost) • Episodic, not done in real time • Level of disaggregation 7

M&E: More Frequent Household Surveys • In a rapidly changing system need more frequent data • Data required to verify HMIS • Lots of data best obtained from HHS – Household behaviors (use of ITNs, CPR, etc. ) – Utilization, patient satisfaction, payments – Knowledge and behaviors – Source of care (private, public, non-qualified) • Suggestion: Every 2 years with disaggregation (LQAS or cluster surveys at state level) 8

3. Health Facility Assessments Advantages • Can assess quality of care, more than anecdotes • Can be independent of service providers • Can be done more often than HHS • Relatively inexpensive Disadvantages • Complex to design • Lots of data, can overwhelm managers • Cannot provide information on coverage, equity 9

4. Quantitative Supervisory Checklist • A reduced version of a health facility assessment • Objectively assesses a variety of indicators to come up with total score. • Takes about 1 -3 hours to complete • A copy of results left in the health facility, easy to track progress • QSC is both a management intervention and tool for M&E 10

4. Quantified Supervisory Checklists Advantages • Can assess QOC. • Can be independent of service providers • Can be done often • Inexpensive • Clarifies what is expected of HWs • Can be adapted to conditions as they change Disadvantages • Challenging to design • Cannot provide information on coverage, equity • Ensuring continued use is difficult 11

Recommendations • Use all four approaches – none are perfect • Relying excessively on HMIS is unwise • Allocate sufficient budget to carry out various approaches to data collection (5 -7% of projects) • Make M&E somebody’s job 12