Impact Evaluation in the Real World One nonexperimental

  • Slides: 20
Download presentation
Impact Evaluation in the Real World One non-experimental design for evaluating behavioral HIV prevention

Impact Evaluation in the Real World One non-experimental design for evaluating behavioral HIV prevention campaigns

Implementation realities • BCC program: – Has already started – Builds on the previous

Implementation realities • BCC program: – Has already started – Builds on the previous campaign (not the first one addressing behaviour) – Is being rolled out in communities that have other HIV prevention interventions – There are endogenous ‘interventions’ (e. g. conversations on the way to school, or in the waiting line at the clinic) – Diffusion is a good thing – Cannot (and does not want to) control implementation

Difference in Differences Example

Difference in Differences Example

Effect = 3. 47 – 11. 13 = - 7. 66 Participants 66. 37

Effect = 3. 47 – 11. 13 = - 7. 66 Participants 66. 37 – 62. 90 = 3. 47 57. 50 - 46. 37 = 11. 13 Non-participants 6

Effect = 8. 87 – 16. 53 = - 7. 66 Before 66. 37

Effect = 8. 87 – 16. 53 = - 7. 66 Before 66. 37 – 57. 50 = 8. 87 62. 90 – 46. 37 = 16. 53 After 7

Counterfactual assumption: Without intervention participants and nonparticipants’ pregnancy rates follow same trends 8

Counterfactual assumption: Without intervention participants and nonparticipants’ pregnancy rates follow same trends 8

74. 0 16. 5 9

74. 0 16. 5 9

74. 0 7. 6 10

74. 0 7. 6 10

Matching Example

Matching Example

Implementation realities • BCC program: – Has already started – Builds on the previous

Implementation realities • BCC program: – Has already started – Builds on the previous campaign (not the first one addressing behaviour) – Is being rolled out in communities that have other HIV prevention interventions – There are endogenous ‘interventions’ (e. g. conversations on the way to school, or in the waiting line at the clinic) – Diffusion is a good thing – Cannot (and does not want to) control implementation

What do we need to know? • Can a specific set of communication messages

What do we need to know? • Can a specific set of communication messages manipulate a specific set of sexual behaviors? • What magnitude of behaviour change will give what magnitude of changes in incidence?

Approach decided on • NON-intervention approach: – We are NOT trying to prove that

Approach decided on • NON-intervention approach: – We are NOT trying to prove that one campaign works…. . BUT we are trying to see whether a specific set of messages work, irrespective of the method of delivery or transmission of the method • Observation approach – We are not trying to force one intervention to work; not focusing on implementation of one intervention

So what WILL we do? • • • Non-experimental design – Researcher does not

So what WILL we do? • • • Non-experimental design – Researcher does not manipulate the independent variable (message exposure) – No control group in the community; create the control group statistically through matching Collection of exposure, behavioural and biological data from random sample of individuals and their sexual partners Develop a measurement of intensity of exposure (‘doses’ of exposure) Determine the probability of having a specific dose of exposure Match individuals with similar covariates, but different doses of exposure

So what WILL we do? 1. Survey to measure demographic covariates (or use pop

So what WILL we do? 1. Survey to measure demographic covariates (or use pop survey data) 2. Measure type and intensity of exposure to messages – Different doses of exposure to MCP campaign messages among the population – Detailed measurement of method of exposure to messages during surveys: Direct channels (# times heard messages on radio…); AND indirect channels (conversation with friends, relatives, etc. ; as shown to be important in accounting for HIV declines in Uganda) – Construct message exposure scale (low vs. high, or more detailed) using statistical techniques (e. g. , principal components analysis) – Every individual has a single score for message exposure

So what WILL we do? 3. Survey to measure exposure, behavioural outcomes, couple and

So what WILL we do? 3. Survey to measure exposure, behavioural outcomes, couple and social network norms and HIV incidence amongst random selection of individuals 4. Nested sub-study to trace partners of those who reported one or more sexual partners, and collect same data from them 5. Parallel measurement of ‘social norms’ – hearsay ethnography or other methods

So what WILL we do? 6. Analyses – – – Use covariates to calculate

So what WILL we do? 6. Analyses – – – Use covariates to calculate an individual’s propensity (scalar summary of all covariates) to receive a specific ‘dose of treatment’ (message exposure scale) Match pairs of participants (index cases and their sexual partners) with similar propensity scores and different doses of treatment (control and treatment groups) Now, we can calculate impact (behavioural and biological outcomes) by comparing the means of outcomes across participants and their matched pairs 7. Modeling – – – Has the density of the sexual network changed over time, and to what extent has it changed? How 'much' behaviour change is needed, over what period of time in how many individuals, to bring about what levels of reductions in new infections What are the individual and the combined effects of MC, ART,

ure ’ ex po s ‘Hi gh ‘Low Density of scores for low exposure

ure ’ ex po s ‘Hi gh ‘Low Density of scores for low exposure expo sure ’ Density of scores for high exposure Low probability of exposure given X High probability of exposure given X

What we will know • Can a specific set of communication messages (delivered in

What we will know • Can a specific set of communication messages (delivered in different ways) manipulate a specific set of sexual behaviors? • What magnitude of behaviour change will give what magnitude of changes in incidence?