Impact Evaluation of the Rwanda Community PerformanceBased Financing

Impact Evaluation of the Rwanda Community Performance-Based Financing Program College of Medicine and Health Sciences School of Public Health

Background: Health in Rwanda has experienced dramatic changes in indicators of health outcomes and utilization of health services in recent years DHS 2005 DHS 2010 DHS 2014 -15 Under-5 mortality (per 1, 000 live births) 152 76 50 Maternal Mortality (per 100, 000 live births) 750 476 210 Delivery at Health Facility 28% 69% 91% ANC by skilled provider 94% 98% 99. 8% Total Fertility Rate 6. 1 4. 6 4. 2

Background: IE of PBF (First Generation) Basinga, Gertler et al. 2012; Gertler and Vermeersch 2012; de Walque, Gertler et al 2015 Positive Impact • Quantity: • Institutional deliveries • Preventative care visits by children • VCT for couples • Quality of prenatal care • Health Outcomes of children No Impact • Prenatal care utilization • Use of modern contraceptives PBF at the health facility level was scaled up nationally in 2008

Background: Community PBF (Second Generation) • Since 2009, Community Health Workers (CHWs) were paid for reporting on health indicators in their communities • Additional components were added through the Community Performance. Based Financing Program in order to promote targeted services • This study evaluates the impact of 2 interventions that were added to the scheme: 1. Performance incentives for CHW cooperatives 2. Demand-side in-kind incentives

Background: organization of CHWs in Rwanda Each village has 3 volunteers serving as Community Health Workers (CHWs). Criteria CHW in Charge of Maternal and Neonatal Health Multidisciplinary CHWs • Can read and write • Age 20 -50 • Lives in the village • Elected by the village residents

Background: organization of CHWs in Rwanda All the CHWs within the catchment area of a health center are organized in a CHW cooperative. Cooperative

Background: organization of CHWs in Rwanda 70% of payments received by a cooperative must be invested in income generating activities (IGAs). 30% of the payments and revenues from the IGAs are given to cooperative members. It is up to the cooperatives to determine distribution rules.

Intervention #1: Performance Incentives for CHW Cooperatives CHW cooperatives received financial rewards for: 1. Nutrition monitoring: # children 6 -59 months monitored 2. Timely Antenatal Care: # of women accompanied/referred within first 4 months of pregnancy 3. In-Facility Delivery: # of women accompanied/referred for assisted delivery 4. New Family Planning users: # referred to health center 5. Regular Family Planning Users: # regular users at health center • 4 indicators related to TB and HIV were added at a later stage and not evaluated

Intervention #2: Demand-Side In-Kind Transfers Women received gifts for seeking care for the following services: Eligibility* Initiation of Antenatal Care during first 4 months of pregnancy Delivery in health center Initiation of Postnatal Care during the 10 days after delivery Value (Ceiling) Suggested Package 5 USD Adult cloth and water treatment tablets OR baby cloth package and water treatment tablets 6. 67 USD Baby soap, baby shawl and baby bed sheets An umbrella and water treatment tablets OR Adult cloths 3. 33 USD * Women can only receive the gifts for one pregnancy every 3 years.

Research Questions 1. Do the demand-side in-kind transfers and the performance incentives to CHW coops increase • • Initiation of prenatal care within first 4 months of pregnancies? Total prenatal care visit? In-facility deliveries? Rate of postnatal care within 10 days after delivery? 2. Is there a multiplicative effect when both interventions are implemented? 3. Do the performance incentives to CHW coops affect • Behavior and motivation of the CHWs? • Use of modern contraceptives? • Growth monitoring of children under 5

Study Design: RCT 198 sectors (sub-districts) were randomly allocated into 4 study arms: Payments to CHW Coops For Reporting* Performance No C S Yes D D+S Demand-Side Transfers * Coops paid for reporting received the average amount received by the coops paid for performance

Study Design: Selection and Randomization of Sectors • Study conducted in 19 districts in 4 provinces • Excluding Kigali • 18 districts of previous IE • A sector usually corresponds to a catchment area of a health center • Include only sectors with a public or non-for-profit faith-based health centers • Exclude 30 sectors where the demand-side intervention was piloted • Sectors blocked by district and poverty ranking

Study Design: Randomization of Sectors

Study Design: Baseline Survey • 12 villages were randomly selected within the catchment area of each health center • Household survey • A field supervisor consulted village leader and/or CHW to identify household with most recent birth or pregnancy (N = 2, 376) • Household questionnaire • Interview with “core” respondent • CHW survey • Target of 2 CHWS in each selected village (N = 4, 668) • CHW in charge of maternal and neo-natal health should have been included • Cooperative survey (N = 197)

Study Design: Follow Up Survey • Same villages were visited • Household survey: two samples! • Women with most recent birth in their village (N = 2, 343) • “Baseline” Women (N = 2, 157) • CHW survey • CHW in charge of maternal and neo-natal health (N = 2, 220) • Cooperative survey (N = 197) • Health Center Assessment (N = 197)

Timeline February. May 2010 November 2013 -June 2014 • Follow-up Survey • Baseline Survey 2010 2011 October 2010 • Interventions Introduced 2012 2013 February 2013 • Last transfer of funds for in -kind transfers 2014

Descriptive Analysis: Baseline Sample Characteristics Core Respondents Total D S DS C 28. 2 28. 1 27. 8 28. 2 28. 7 (6) (5. 8) (5. 9) (6. 2) 0. 91 None 0. 18 0. 16 0. 18 0. 19 Primary 0. 70 0. 72 0. 69 0. 67 0. 69 > primary 0. 13 0. 12 0. 13 Covered by Mutelles 0. 90 0. 93 0. 9 0. 88 Number of live births 3. 03 3. 11 2. 95 2. 97 3. 10 (2. 12) (2. 18) (2. 03) (2. 18) 0. 4 0. 37 0. 41 Age Married Education Ever used modern FP

Descriptive Analysis: Baseline Sample Characteristics Care during last pregnancy At least one ANC visit 98% First ANC visit in first 4 months of pregnancy 63% At least 4 ANC visits 37% Skilled-attended in-facility delivery 79% PNC within 10 days of birth 38% Gift received for ANC* 5% Gift received for delivery* 4% Gist received for PNC 2% *The reported value of gifts is about a third of those set by the program for the different services.

Descriptive Analysis: Comparison with the RDHS 2010 Comparing core respondents to RDHS sample of women with a pregnancy during the two previous years Baseline RDHS 2010 28. 2 29. 1 None 0. 17 0. 18 Primary 0. 69 0. 75 > primary 0. 14 0. 07 Number of births 3. 01 3. 34 At least one ANC visit 0. 98 First ANC visit in first 4 months of pregnancy 0. 63 At least 4 ANC visits 0. 37 0. 36 Skilled-attended in-facility delivery 0. 79 PNC within 10 days of birth 0. 38 0. 18 Age Education Care during last pregnancy:

Descriptive Analysis: Baseline Sample Characteristics of CHWs in charge of maternal and neonatal health Mean SD Age 39. 1 (9. 4) Female 0. 94 Education Primary 0. 61 > primary 0. 39 Number of years as CHW 2. 6 (3. 7) Number of HHs in village 122 (117) Number of hours spent on health activities in past week 17. 7 (28) Distance to health center (in km) 4. 1 (4. 9)

Descriptive Analysis: Time Trends Baseline and Follow-up rates among Control group 100% 98% 99% 94% 90% 79% 80% 72% 70% 63% 60% 50% 36% 40% 34% 30% 20% 13% 10% 0% Any ANC Timely ANC ≥ 4 ANC Baseline Follow-up In-facility Delivery Timely PNC

Descriptive Analysis: Program Implementation Incentive amounts reduced during the implementation of the program Incentivized indicators Nutrition Monitoring Timely Antenatal Care In-facility Delivery New Family Planning User Regular Family Planning User 2010 Unit Fees (USD) 2011 2012 2013 2014 3. 24 0. 57 0. 43 2. 24 1. 12 0. 81 2. 73 1. 37 0. 99 2. 90 1. 45 1. 05 2. 11 1. 06 0. 77

Descriptive Analysis: Program Implementation « D'habitude, est-ce que votre centre de sante donne des cadeaux femmes qui viennent pour les services suivants: » Indicator D S DS C Timely ANC 0. 94 0. 16 0. 90 0. 08 In-facility delivery 0. 92 0. 18 0. 88 0. 13 Timely PNC 0. 90 0. 09 0. 88 0. 08

Descriptive Analysis: Program Implementation Experience of stock outs by health centers Reason for stock outs Supplier 14% 6% 19% No stock outs Sometime 18% Procurement 23% Often Very Often 49% Availability of funds 69%

Descriptive Analysis: Program Implementation Share of women who received gifts

Descriptive Analysis: Program Implementation Share of eligible women who received gifts, by year of birth Timely ANC In-Facility. Delivery Timely PNC 40% 40% 35% 35% 30% 30% 25% 25% 20% 20% 15% 15% 10% 10% 5% 5% 5% 0% Promise Gift 2011 3% 33% 2012 3% 19% 2013 2% 14% 0% Promise Gift 2011 1% 34% 2012 2% 14% 2013 4% 12% * Combining samples of recently pregnant women and “baseline” women; only pregnancies ending in live births 0% Promise Gift 2011 3% 33% 2012 3% 19% 2013 2% 14%

Results: Maternal Health Services • Indicators: • Timely ANC • In-facility deliveries • Timely PNC • Sample of women with most recent birth in their village • Pregnancies resulting in a live birth

Results: ANC visit within first 4 months of pregnancy Timely ANC 85% 80% 75% 70% 65% 60% 55% 50% Timely ANC Control 72% Demand 82% Supply 74% D + S 80% • A positive and significant (at the 1% level) impact of the demand-side in-kind incentives of about 10 percentage points • The CHW incentives are not found to have a significant effect • No difference between the ‘Demand’ and the ‘Demand+Supply’ treatment arms

Results: at least 4 ANC visits Four or more ANC visits 50% 45% 40% 35% 30% 25% 4 ANC Control 40% Demand 46% Supply 43% D + S 45% • Not targeted by the program! • Higher in the intervention sectors, but not statistically significant at the 10% level

Results: Skilled-attended in-facility delivery In-Facility Delivery 100% 95% 90% 85% 80% 75% 70% Deliv Control 94% Demand 95% Supply 96% D + S 94% • No statistically significant difference between the treatment arms • Rate has increased substantially in the duration of the study for other reasons

Results: PNC within 10 days after delivery 25% 20% 15% 10% 5% 0% Timely PNC Control 13% Demand 22% Supply 11% D + S 20% • A positive and significant (at the 5% level) impact of the demand-side in-kind incentives of about 7 percentage points • Not targeted by the CHW incentives intervention

Results: targeted maternal health indicators • Results are robust • Inclusion of controls • Diff-in-diff specification • Exclusion of 3 misclassified sectors • Negligible changes when sample weights are used • Using data on number of households in each sector

Results: Fertility and Family Planning • Indicators • Birth since baseline • Ever used modern family planning method • With and without adjustment for baseline response • Sample of baseline women • Caveat: not optimal for estimating this • Caveat: higher attrition among treatment groups implementing the supplyside intervention

Results: Fertility and Family Planning 80% 70% 60% 50% 40% 30% 20% 10% 0% Pregnancy Modern* Control 53% 52% 68% Demand 55% 57% 69% Supply 59% 50% 64% • No statistically significant impact on any of the indicators D + S 55% 54% 70%

Results: Nutritional monitoring • Indicator: • Child was measured in the past 6 months to determine nutritional status • Sample of children 6 -59 months • Of all women (recent pregnancies and baseline women)

Results: Fertility and Family Planning Growth Monitoring in past 6 months 90% 80% 70% 60% 50% 40% 30% Child measured Control 79% • No statistically significant impact Demand 80% Supply 75% D + S 75%

Other results: CHW • No Impact on CHW behavior • Hours spent on health activities • Number of households visited • Consulting other CHWs • No Impact on measures of satisfaction and motivation of CHWs • No impact on CHW behavior reported by the women • • CHWs referred or accompanied to ANC, Delivery or PNC services CHWs provided information on these services Interacted with CHWs in the past three months (in the community or home visits) Satisfaction from CHWs • Women in Demand D+S sectors report CHWs were aware earlier of their pregnancies

Other results: Cooperatives • No Impact on. . • • • Number of meetings Number of members recruited, dismissed, resigned in the past 12 month Cooperative income Whether payments are allocated by performance Internal assessment of coop members

Key Findings: Demand-Side In-Kind Incentives • The demand-side in-kind incentives caused an increase in timely ANC and PNC services • Although some challenges in procurement and frequent stock outs • Although some health centers independently implemented their own demand-side incentives strategies to promote utilization • Although funding ended before end-line data collection • Consistent with findings in other countries that implemented demand-side cash transfers

Key Findings: Performance Incentives to CHW Coops • No impact of incentives to CHW cooperative on targeted indicators, CHW behaviors and CHW motivation. • Potential reasons for lack of impact – Incentives were too low – Collective reward but individual effort – Pay-for-reporting could have already oriented the CHWs towards targeted indicators – Limited scope given the many supply-side programs targeting the same indicators

Caveats • Self-reported indicators • Timing of survey relative to termination of the demand-side intervention • Hoping to include analysis of administrative data • Not optimal sample for the analysis of fertility and family planning indicators

Research Team • Ministry of Health • Fidel Ngabo • Cathy Mugeni • University of Rwanda • Ina R. Kalisa • James Humuza • Jeanine Condo • Vedaste Ndahindwa • The World Bank • Gil Shapira • Netsanet W. Workie • Jeanette Walldorf The study was funded by the Health Results Innovation Trust Fund (HRITF)

Thank you!