Zambia Health Results Based Financing RBF Project Results

Zambia Health Results Based Financing (RBF) Project Results from the Impact Evaluation Presenters Jed Friedman, Principal Investigator Jumana Qamruddin, Task Team Leader 1

Results based financing in Zambia 1. Provider-Purchaser split – Quantity and Quality data verification – Steering Committees (SCs) as Independent Verifiers – Periodic External Verification 2. “Performance based financing” through public health sector contracting. One of the very few examples of “contracting-in” – “Fee-for-service” on a set of Maternal and Child Health indicators 3. Managerial and financial autonomy of health facilities 4. Nine (9) health facility indicators targeting improvements in MCH 5. Health centre quality indicators in 10 areas 6. Performance package at District Medical Office 2

Impact Evaluation: Design The IE seeks to determine the causal impact of HRBF on priority service provision and population health indicators Intervention (RBF) (10 Districts) Control 1 (10 Districts) Control 2 (10 Districts) RBF program (including Enhanced financing Business-as-usual (status incentives) +Em. ONC (equal to RBF incentives) quo) equipment + Em. ONC equipment 3

Three-arm experimental Study Design 30 districts matched in groups of three on key health systems and outcome indicators and randomly allocated to each arm: 10 districts per arm o 10 Intervention Districts (RBF) o 10 Enhanced (Input-Based) Financing Districts (C 1) o 10 Business-as-usual (status quo) Districts (C 2) • District triplets selected within each province by matching on: – – – geographical accessibility (i. e. rural and remoteness) number and level of health facilities average facility catchment population proportion of staff in position health services utilization rates • Difference-in-difference estimator between matched districts in treatment and control groups estimates program impact 4

District Selection 5

Implementation & Learning Platforms Program Inception (April 2012) Process Evaluation (May – June 2013) Baseline (Nov – Dec 2011) Program Ends (Oct 2014) Endline (Nov – Dec 2014) Routine Performance Review (Quarterly) – Operational Data 6

Questions investigated What is the causal effect of the RBF on targeted health indicators and other population outcomes of interest? – What are the effects on coverage of health services? – What are the effects on quality of care? – What are the effects on health system functionality? 7

Data Source 1: Household Survey • Population representative survey of health behavior and health outcomes • Baseline and endline data at community and household levels covering – 18 districts – 307 enumeration areas – 3064 households in BL and 3087 in EL • After full community listing, random sample of all households with a pregnancy related outcome in the two years before survey 8

Data Source 2: Health Facility Survey • A comprehensive review of the structure, provision, and quality of care at facility level • 213 facilities in both baseline and follow up • Instruments – Facility checklist – Health worker tool (330 in BL; 402 in EL) – Exit interview tool – ANC (900 in BL; 1256 in EL) child illness (1064 in BL; 1273 in EL) • Data collected by independent contractor: University of Zambia 9

Data Source 3: External Verification A review of the completeness, accuracy and validity of reporting at facility level • 140 facilities: 105 in RBF districts and 35 in C 2 districts • Instruments – Facility document review checklist – Client tracer tool • Data collected by independent contractor: Zambia Institute for Policy Analysis and Research (ZIPAR) 10

Data Source 4: Cost-effectiveness analysis Ø To estimate the cost-effectiveness of RBF program in Zambia: § RBF versus status quo (C 2) § RBF versus enhanced financing (C 1) § C 1 versus status quo (C 2) Ø To assess the cost-effectiveness with and without adjustment for quality improvement over a broad number of MCH services 11

Results Outline 1. Healthcare coverage 2. Quality of services 3. Health systems (incl. HRH, Finance) 4. Cost-effectiveness analysis 12

Healthcare coverage 13

Institutional & Skilled Deliveries RBF vs. Control 1 RBF vs. Control 2 Control 1 vs. Control 2 Impact estimate Facility delivery -0. 049 0. 128* 0. 175* Skilled provider and facility -0. 043 0. 101 0. 142* Note: Statistical significance determined by Fisher exact standard errors, * p <. 12 • Deliveries at the facility increased by 12. 8 percentage points in RBF districts, and by 17. 5 percentage points in enhanced financing districts 14

Antenatal care coverage RBF vs. Control 1 RBF vs. Control 2 Control 1 vs. Control 2 Impact estimate -0. 015 -0. 004 -0. 372* -0. 015 -0. 034 -0. 476* 0 -0. 029 -0. 108 Any ANC 4 or more ANC visits Timing of first ANC visit Note: Statistical significance determined by Fisher exact standard errors, * p <. 12 • Many ANC indicators are already relatively well performing in Zambia before the RBF pilot period, and show little change as a result of the RBF program or enhanced financing • However one important exception: pregnant women present significantly earlier for their first ANC visit in RBF districts as compared to the controls 15

Post-natal care coverage RBF vs. Control 1 RBF vs. Control 2 Control 1 vs. Control 2 Impact estimate -0. 051 0. 082 0. 132* Any PNC Note: Statistical significance determined by Fisher exact standard errors, * p <. 12 • Both RBF and especially C 1 increase coverage relative to C 2 16

Family planning outcomes RBF vs. Control 1 RBF vs. Control 2 Control 1 vs. Control 2 Impact estimate -0. 024 -0. 045 -0. 078* -0. 039 0. 002 0. 083* -0. 017 0. 046 0. 159* Any Contraception Modern Contraception (of FP users) Any FP outreach Note: Statistical significance determined by Fisher exact standard errors, * p <. 12 • Little effect on contraceptive take-up (although sample is not fully representative) • Increase in FP outreach in both RBF and especially C 1 areas 17

Vaccination outcomes RBF vs. Control 1 RBF vs. Control 2 Control 1 vs. Control 2 Impact estimate 0. 116 -0. 066 0. 031 -0. 01 0. 052 0. 015 0. 07* 0. 061* -0. 046 0. 081* 0. 028 0. 056* Fully vaccinated Any vaccinations BCG inject ever received DPT ever received Note: Statistical significance determined by Fisher exact standard errors, * p <. 12 • RBF performed better than C 1 and C 2 in fully vaccinated coverage but the impact estimates are not precise. • For some of the other measures of immunization, both the RBF and C 1 performed better than C 2 18

Quality of services 19

Structural Quality Facility infrastructure variables Facility experiences no power outage Facility experiences no water outage Infrastructure index RBF vs. Control 1 Impact p-value estimate -0. 019 0. 881 0. 041 0. 688 0. 195 0. 470 RBF vs. Control 2 Impact pestimate value 0. 194 0. 159 0. 051 0. 476 0. 483* 0. 099 • Little change in individual measures of structural quality, however an aggregate index suggests gains in RBF compared with pure control districts • Gains in structural quality of care-specific indices RBF vs. Control 1 RBF vs. Control 2 Impact estimate p-value Curative Care 0. 39 0. 204 0. 28** 0. 042 Family planning Delivery Room 0. 15 0. 578 0. 08 0. 546 0. 61** 0. 010 0. 57*** 0. 000 20

Availability of drugs RBF vs. Control 1 Iron tabs Folic acid tabs Artemisinin-Based Combination Therapy (ACT) Drug availability index RBF vs. Control 2 Impact estimate p-value -0. 03 -0. 09 0. 722 0. 455 -0. 03 0. 13 0. 824 0. 259 0. 04 -0. 08 0. 693 0. 844 0. 27*** 0. 06 0. 008 0. 893 * p<0. 1 ** p<0. 05 *** p<0. 01 • With the exception of ACT, little relative gain in drug availability for either RBF or enhanced financing 21

Availability of equipment RBF vs. Control 1 Tape measure Baby scale (infant weighing scale) Forceps, artery Needle holder Equipment availability index Impact estimate 0. 15* 0. 05 0. 08 -0. 09 0. 03 p-value 0. 097 0. 643 0. 406 0. 389 0. 917 RBF vs. Control 2 Impact estimate 0. 11 0. 22*** 0. 16** 0. 25*** 0. 37* p-value 0. 399 0. 007 0. 011 0. 001 0. 088 * p<0. 1 ** p<0. 05 *** p<0. 01 • Select equipment for delivery and neo-natal care more available in RBF districts 22

Quality of ANC (Source: Exit interviews) RBF vs. Control 1 Weighed Blood pressure measured Abdomen palpated Advice on diet Quality of ANC index Impact estimate p-value -0. 02 -0. 03 0. 07 0. 00 0. 14*** 0. 02 0. 632 0. 809 0. 152 0. 987 0. 009 0. 921 RBF vs. Control 2 Impact estimate 0. 06 0. 08 0. 09* 0. 12* 0. 02 0. 33 pvalue 0. 251 0. 452 0. 063 0. 083 0. 850 0. 165 * p<0. 1 ** p<0. 05 *** p<0. 01 • Process measures of ANC quality for a few measures are improved in RBF as compared to C 1 and C 2, but little gain in overall index 23

Quality of child health care (Source: Exit interviews) RBF vs. Control 1 Asked age Weighed child Measured height Physically examined Quality of care index Impact estimate -0. 01 -0. 07 -0. 10 -0. 09 p-value 0. 880 0. 378 0. 104 0. 327 0. 669 RBF vs. Control 2 Impact estimate 0. 02 0. 06 -0. 02 -0. 08 0. 14 p-value 0. 776 0. 498 0. 577 0. 350 0. 565 * p<0. 1 ** p<0. 05 *** p<0. 01 • No apparent gain in process quality of child health visit 24

Satisfaction on ANC (Source: Exit interviews) The health worker spent a sufficient amount of time with the patient You trust the health worker completely in this health facility Satisfaction index RBF vs. Control 1 RBF vs. Control 2 Impact estimate p-value 0. 08* 0. 067 0. 08* 0. 081 0. 07* 0. 04 0. 066 0. 826 0. 03 0. 12 0. 569 0. 574 * p<0. 1 ** p<0. 05 *** p<0. 01 • Higher levels of patient satisfaction in selected dimensions of ANC (but not all) in RBF as compared to the two controls • Little apparent increase in overall satisfaction 25

Satisfaction on child health care (Source: Exit interviews) RBF vs. Control 1 The amount of time you spent waiting to be seen by a health provider was reasonable You trust the health worker completely in this health facility Satisfaction index RBF vs. Control 2 Impact estimate p-value -0. 02 0. 823 -0. 06 0. 477 0. 11* 0. 09 0. 057 0. 617 0. 04 0. 504 0. 858 • Little apparent increase in overall satisfaction for child care 26

Health systems 27

Level of RBF revenue, RBF vs C 1 5, 000 90% 4, 500, 000 4, 000 Amount in US$ 80% 78% 70% 3, 500, 000 56% 3, 000 2, 500, 000 38% 2, 000 43% 60% 50% 40% 30% 1, 500, 000 1, 000 20% 500, 000 10% - 2012 RBF C 1 2013 2014 Total 0% Funds disbursed to C 1 in propotion to RBF 28

Proportion of GRZ grant to RBF grant 250% 200% 30% 26% 25% 21% 150% 230% 100% 13% 171% 50% 15% 10% 34% 0% 2012 -18% 2014 2013 -50% 5% 0% Growth of RBF grant Growth of GRZ grant Proportion of GRZ grant to RBF grant 29

Use of RBF Funds, and Proportion of RBF staff incentives to Govt. staff salaries 70. 0% 59% 60. 0% 51% 50. 0% 47% 40. 0% 30. 0% 20. 0% 10. 0% 14% 10% 0% 1% 2012 2013 2014 Actual over Period Proportion of RBF staff incentives to GRZ staff salaries Proportion of RBF funds used for RBF staff incentives 30

Facility governance RBF vs. Control 1 RBF vs. Control 2 Impact p-value estimate Number of Health Center Committee meetings held in the last 12 months Number of visits made by a district hospital representative for supervision Number of times performance of staff assessed internally Number of times performance of staff assessed externally Number of times performance of the facility as a whole assessed externally 1. 03 0. 103 1. 26* 0. 093 0. 82* 0. 065 0. 66 0. 542 3. 41*** 0. 002 4. 41*** 0. 002 1. 40** 0. 046 2. 33*** 0. 003 0. 365 0. 022 1. 15 2. 64** • Increases in supervisory visits and performance assessments 31

Autonomy RBF vs. Control 1 able to allocate my facility budget choice over who I allocate for what tasks. choice over what services are provided in the facility. Autonomy index RBF vs. Control 2 Impact estimate 0. 022 -0. 067 0. 873 0. 565 Impact p-value estimate 0. 092 0. 147 0. 035 0. 556 0. 036 0. 797 0. 127* 0. 055 0. 06 0. 880 0. 26** 0. 037 p-value • Gains in some measures of facility autonomy in RBF compared with C 2, but not C 1 districts * p<0. 1 ** p<0. 05 *** p<0. 01 32

Job Satisfaction RBF vs. Control 1 RBF vs. Control 2 Work conditions N 448 β (s. e. ) 6. 393 (5. 121) N 464 β (s. e. ) 4. 366* (2. 183) Compensation 448 8. 639** (4. 081) 464 3. 880* (1. 994) Recognition 448 1. 439 (2. 842) 464 0. 086 (1. 324) Opportunities 448 4. 686 (4. 183) 464 3. 641* (2. 004) * p<0. 1 ** p<0. 05 *** p<0. 01 • Dimensions of job satisfaction generally higher in RBF districts, especially as compared with C 2 33

Accuracy of reporting Indicator Out-patient visit Delivery Ante-natal care HIV testing and counseling PMTCT proportion of control facilities under-reporting services 0. 643 0. 833 0. 654 0. 655 0. 875 Relative likelihood of RBF facilities underreporting 0. 049 -0. 154 0. 029 -0. 093 -0. 156 P-value of relative likelihood 0. 634 0. 100 0. 787 0. 369 0. 082 • Most services in general are underreported – even in RBF districts! • For select services, RBF appears to improve accuracy of reported information 34

Cost-effectiveness analysis 35

Incremental cost effectiveness ratios Cost/life saved (US$) Cost/QALY gained (US$) Mid-point (lower bound; upper bound) RBF vs C 1 (unadjusted) 35, 802 (17, 143; 594, 308) 1, 513 (724; 24, 544) RBF vs C 1 (quality adjusted) 30, 219 (14, 882; 27, 8582) 1, 277 (628; 11, 820) RBF vs C 2 (unadjusted) 24, 423 (11, 452; 52, 626) 1, 031 (484; 2, 223) RBF vs C 2 (quality adjusted) 20, 434 (10, 233; 35, 940) 863 (433; 1, 518) C 1 vs C 2 (unadjusted) 14, 786 (7, 461; 144, 906) 624 (182; 5, 482) C 1 vs C 2 (quality adjusted) 12, 280 (4, 396; 310, 513) 518 (168; 16, 221) Comparison 36

Summary: Context, Coverage, Quality • One of the first 3 -armed IE designs in the portfolio • Project was implemented during a period of several changes in GRZ leadership and ministry organization • RBF and C 1 compared to C 2 had considerable gains across a number of indicators • RBF vs C 1 on health care coverage indicators were comparable • Structural quality: Results were mostly inconclusive but RBF better than C 2 on the status of infrastructure and medical equipment; and both controls on quality of delivery rooms • Process quality: Minimal progress on process quality of maternal health care in RBF and C 1 districts 37

Summary – HRH • Few gains in client satisfaction except: – Clients who visited RBF health facilities were more satisfied with the time that the health workers spent with them as compared to C 1 and C 2. – Clients trusted health workers in RBF facilities more than those in C 1 facilities for both maternal and child health services • Job satisfaction and retention of health workers increased in both RBF and C 1 but the gains were higher in RBF as compared to C 1. 38

Summary: Cost Effectiveness Analysis • RBF delivered greater health gains, in terms of lives saved or QALYs gained, than C 1 when compared with C 2. • However these gains were supplied at a higher unit cost. In $/QALY, C 1 is more cost-effective. • Both interventions can be considered cost-effective when compared with the annual per-capita income for Zambia. • However, cost-effectiveness analysis does not explicitly account for health system strengthening investments – certain dimensions of effectiveness with regards to RBF may have been missed by the analysis. 39

Summary: Incentives and RBF Grants • RBF incentives as a percentage of staff salaries: equal to 10% of staff salaries by end of implementation period – Intended ratio was higher but GRZ increased staff salaries for all civil servants ranging from 100% to 200% – Low powered incentives likely result in reduced ability of RBF to affect targeted outcomes • RBF grants as a percentage of operational expenditures: – At facility level, The RBF grant at facility level was spent more on operational activities as compared to staff incentives – Comparison to GRZ grants at facility level suggests that RBF grants may have played a substitutional role instead of being additional (as intended) 40

Summary: Provider Payment Mechanism • RBF was being implemented in a health system that already had relatively high coverage in some indicators – implications for efficiency of spending • Rather than fee-for-service paying for all services rendered, it may have been more effective to have used a target or coverage-based provider payment mechanism 41

Summary: Disbursement Mechanisms • By using two different mechanisms, the study was able to measure the success of each system in terms of overall level of RBF funding disbursed and used by facilities. • Disbursement of RBF performance grants directly to health facilities enabled fiscal decentralization and increased autonomy. • Results show that health facilities in the C 1 districts did not receive the same amount as the RBF districts due to delayed retirement and low absorptive capacity. • By the end of the RBF program, the proportion of disbursement to C 1 districts was only 56% of what the RBF districts had received. To note, health facilities in the RBF intervention group allocated 47% of the total RBF funds for staff incentives, and 53% for investment. 42

Conclusions • Both the RBF and the C 1 arms contributed to some very important health gains as compared to business-as-usual “C 2”, and C 1 at even lower $/QALY • But the RBF observed relatively more gains in health systems governance, client perception, and health worker satisfaction • The health systems gains under the RBF may translate into population and health gains over a period longer than the 2 year measured under the pilot • Enhanced financing is not just money in a vacuum, involves signaling and direction. Better understanding the effectiveness of these mechanisms can inform policy and program development 43

Considerations and Implications -Focusing RBF mechanisms on improving quality as the primary focus -Design of National Health Insurance Scheme and other health sector priorities -Setting agenda for next generation of learning/operational research - Current IDA Lending operation: RBF component with a heavy process evaluation 44

Thank You! 45

Primary data collected from 18 districts Districts sampled for household survey for Phase I Questions Province Districts Central Chibombo; Kapiri Mposhi; Mumbwa Eastern Chadiza; Lundazi; Nyimba Luapula Kawambwa; Milenge; Mwense Northern Chinsali; Chilubi; Mporokoso North-Western Chavuma; Mufumbwe; Mwinilunga Southern Mazabuka; Namwala; Siavonga 46

Work Motivation RBF vs. Control 1 Teamwork Autonomy Changes in facilities Work environment Self-concepts Recognition Well-being Leadership of facility N 447 448 448 448 450 431 β (s. e. ) 0. 385 (3. 132) 0. 822 (4. 311) -2. 096 (2. 664) -1. 788 (2. 597) -0. 727 (1. 866) -0. 380 (3. 282) 1. 100 (2. 981) -3. 075 (4. 885) RBF vs. Control 2 N 462 463 463 461 465 446 β (s. e. ) 0. 925 (1. 429) 1. 314 (1. 768) 1. 026 (1. 240) 1. 257 (1. 260) 0. 774 (1. 075) -0. 837 (1. 330) 2. 418* (1. 236) 1. 210 (2. 613) * p<0. 1 ** p<0. 05 *** p<0. 01 • Work motivation largely unchanged 47

Distribution of RBF program costs 48