Measuring Impact Course Overview 1 What is Evaluation

Measuring Impact

Course Overview 1. What is Evaluation? 2. Measuring Impact 3. Why Randomize? 4. How to Randomize? 5. Sampling and Sample Size 6. Threats and Analysis 7. Cost-Effectiveness Analysis 8. Project from Start to Finish

Women as Policymakers CASE STUDY

What was the main purpose of the 73 rd Amendment of India’s constitution? A. To reserve leadership positions for women (and caste minorities) B. To formalize local institutions of leadership C. To give women the right to vote in local elections

Theory of Change from Reservations to Final Outcomes Women are empowered Public goods show women’s preferences Women have different preferences More female Pradhans Reservations for women Pradhan’s preferences matter Democracy imperfect

Data Used Covers All Aspects of Theory of Change Purpose of Measurement Sources of Measurement Indicators Reservations Policy and GP Priorities Administrative Data • Reservation • Budgets • Balance sheets Preferences of Women and Men Transcript from village meeting • Who speaks and when (gender) • Issues raised Public Investments Village Leader Interview • Political experience • Investments undertaken Public Investments Village PRA • Village infrastructure + investments • Perception of public good quality • Participation of men and women Household (HH) Survey • Declared HH preference • HH perceptions of quality of public goods and services

Results: Reservations Increase Female Leadership West Bengal Of Pradhans: Total Number Proportion female Rajasthan Reserved Unreserved 54 107 40 60 100% 6. 5% 100% 1. 7%

Results: Women Raise Different Issues at GP Meetings West Bengal Rajasthan Women Men Drinking water 31% 17% 54% 43% Road improvement 31% 25% 13% 23% Of Pradhans:

Results: Public Investments Show Women’s Preferences West Bengal Issue W M Reserved Investment Rajasthan Issue W M Reserved Investment Issue Investment Drinking Water # facilities 31% 17% 9. 09 54% 43% 2. 62 Road Improvement Road 31% 25% Condition (0 -1) 0. 18 13% 23% -0. 08 Irrigation # facilities 4% 20% -0. 38 2% 4% -0. 02 Education Informal education center 6% 12% -0. 16 5% 13%

Why do You Need Data? Follow Theory of Change § Our theory and hypothesis helps us define the set of outcomes § Need to find indicators that map the outcomes well § Characteristics: Who are the people the program works with, and what is their environment? • Sub-groups, covariates, predictors of compliance § Channels: How does the program work, or fail to work? § Outcomes: What is the purpose of the program? Did it achieve that purpose?

Lecture Overview § Theory of Change: What Do You Want to Measure? § Theory of Measurement: What Makes a Good Measure? § Practice of Measurement: Measuring the Immeasurable § Collecting Data

Theory of Measurement WHAT MAKES A GOOD MEASURE

The Main Challenge in Measurement: Getting Accuracy and Precision More precise More accurate

Terms “Biased” and “Unbiased” Used to Describe Accuracy More accurate “Biased” On average, we get the wrong answer “Unbiased On average, we get the ”right answer

More precise Terms “Noisy” and “Precise” Used to Describe Precision “Noisy” Random error causes answer to bounce around “Precise” Measures of the same thing cluster together

A Noisy and a Precise Measure Can Both Be Biased More precise More accurate “Noisy” Random error causes answer to bounce around “Precise” Measures of the same thing cluster together

Choices in Real Measurement Often Harder More precise More accurate “Noisy” but “Unbiased” Random error causes answer to bounce around “Precise” but “Biased” Measures of the same thing cluster together

The Main Challenge in Measurement: Getting Accuracy and Precision More precise More accurate

Is this introducing noise or bias? A. Noise / Random Error B. Bias A surveyor doesn´t follow the exact phrase of the question: Surveyor- “you feel unsecure in this neighborhood, right? ” Respondent- “well, sometimes yes and sometimes, well, no…” Surveyor- “so, is more of a yes, right? ” - “mmm…well” Respondent - “ok, thanks. Next question. ” code 1=yes

Accuracy § In theory: • How well does the indicator map to the outcome? (e. g. IQ test intelligence) § In practice: Are you getting unbiased answers? • Respondent bias • Recall bias • Anchoring bias • Social desirability bias (response bias) • Framing effect / Neutrality

Precision and Error § In theory: The measure is precise, but not necessarily accurate § In practice: • Length, fatigue • “How much did you spend on broccoli yesterday? ” (as a measure of annual broccoli spending) • Ambiguous wording (definitions, relationships, recall period, units of question) Eg. Definition of terms – ‘household’, ‘income’ • Recall period /units of question • Type of answer -Open/Closed • Choice of options for closed questions Ø Likert (i. e. Strongly disagree, neither agree nor disagree, . . . ) Ø Rankings Ø Quantitative scales. Numbers or bins.

Random Error § Mistakes in estimation or recall § Question wording § Surveyor training/quality § Data entry § Length, fatigue of survey § How do you generalize from certain questions?

Which is worse? A. Bias (Low Accuracy) B. Noise (Low Precision) C. Equally bad D. Depends E. Don’t know/can’t say

A biased measure will bias our impact estimates A. True B. False C. Depends D. Don’t know

Practice of Measurement MEASURING THE UNMEASURABLE

What is Hard to Measure? (1) Things people do not know very well (2) Things people do not want to talk about (3) Abstract concepts (4) Things that are not (always) directly observable (5) Things that are best directly observed

How much juice did you consume last month? A. <2 liters B. 2 -5 liters C. 6 -10 liters D. >11 liters

1. Things people do not know very well § What: Anything to estimate, particularly across time. Prone to recall error and poor estimation • Examples: distance to health center, profit, consumption, income, plot size § Strategies: • Frame the question in a way people are likely to know and remember. • Consistency checks – How much did you spend in the last week on x? How much did you spend in the last 4 weeks on x? • Multiple measurements of same indicator – How many minutes does it take to walk to the health center? How many kilometers away is the health center?

How many glasses of juice did you consume yesterday A. 0 B. 1 -3 C. 4 -6 D. >6

What is Hard to Measure? (1) Things people do not know very well (2) Things people do not want to talk about (3) Abstract concepts (4) Things that are not (always) directly observable (5) Things that are best directly observed

How frequently do you yell at your spouse A. Daily B. Several times per week C. Once per week D. Once per month E. Never

2. Things people don’t want to talk about § What: Anything socially “risky” or something painful • Examples: sexual activity, alcohol and drug use, domestic violence, conduct during wartime, mental health § Strategies: • Don’t start with the hard stuff! • Consider asking question in third person • Always ensure comfort and privacy of respondent

Asking Sensitive Questions: List Randomization § Randomly ask part of the sample the question with / without a sensitive option § Response only a count, not the specific options How many of these statements are true for you? § This morning I took a shower. § My nearest bank branch office is within walking distance. § I have tea every day. How many of these statements are true for you? § This morning I took a shower. § My nearest bank branch office is within walking distance. § I have tea every day. § I use my loan for nonbusiness expenses.

Asking Sensitive Questions: List Randomization 2. 8 – 2. 1 = 0. 7 full TRUE difference (on average) 70% used their loan for non-business purposes. Average number of true statements: 2. 1 2. 8

List Randomization Shows Big Differences in Some Real Cases

Asking Sensitive Questions: List Randomization § Some questions are sensitive and people are hesitant to answer truthfully § List randomization is a way to get the answer on average without knowing confidential information on any one person

What is Hard to Measure? (1) Things people do not know very well (2) Things people do not want to talk about (3) Abstract concepts (4) Things that are not (always) directly observable (5) Things that are best directly observed 37

“I feel more empowered now than last year” A. Strongly disagree B. Disagree C. Neither agree nor disagree D. Agree E. Strongly agree

3. Abstract concepts § What: Potentially the most challenging and interesting type of difficult-to-measure indicators § Examples: empowerment, bargaining power, social cohesion, risk aversion § Strategies: • Three key steps when measuring “abstract concepts” • Define what you mean by your abstract concept • Choose the outcome that you want to serve as the measurement of your concept • Design a good question to measure that outcome § Often choice between choosing a self-reported measure and a behavioral measure – both can add value!

“I am involved in the decision to send my child to private vs public school” A. Strongly disagree B. Disagree C. Neither agree nor disagree D. Agree E. Strongly agree

How “Socially Connected" do You Feel to the Other People in this Room? You A B C D E Everyone else in this room

How likely are you to take a taxi with a driver you don’t know after dark? A. Very unlikely B. Unlikely C. Likely D. Very likely

How likely are you to take a taxi with a driver you don’t know after dark? A. Very unlikely B. Unlikely C. Likely D. Very likely

Perceptions and Attitudes § Ask directly • “How effective is your leader? ” (ineffective, somewhat effective, very…) § Indirect approaches often have better accuracy • Listen to a Vignette (Male v. Female) • Revealed preference – voting behavior • Implicit Association tests

Implicit Association Test § People simplify the world for efficiency • Use thumb rules to draw connections • May not even be aware themselves § For some important outcomes, may be worth trying to measure these indirectly • Implicit association one technique

Implicit Association Test: Match on Left or Right?

Implicit Association Test: Match on Left or Right? Parliament

Implicit Association Test: Match on Left or Right? Home

Implicit Association Test: Match on Left or Right? Parliament

Implicit Association Test § People simplify the world for efficiency • Use thumb rules to draw connections • May not even be aware themselves § Actually based on response time, not accuracy • Are respondents faster to select “Parliament” when associated with a man than a woman?

What is Hard to Measure? (1) Things people do not know very well (2) Things people do not want to talk about (3) Abstract concepts (4) Things that are not (always) directly observable (5) Things that are best directly observed 51

What proportion race X people are denied jobs due to racial discrimination? A. 0% B. 1 -20% C. 21 -40% D. 41 -60% E. >60%

4. Things that aren’t Directly Observable § What: You may want to measure outcomes that you can’t ask directly about or directly observe • Examples: corruption, fraud, discrimination § Strategies: • Audit studies, e. g. Rajasthan police experiment to register cases, Delhi Driver’s Licenses, Delhi doctors • Multiple sources of data, e. g. inputs of funds vs. outputs received by recipients, pollution reports by different parties • Don’t worry – there have already been lots of clever people before you – so do literature reviews!

5. Things that are Best Directly Observed § What: Behavioral preferences, anything that is more believable when done than said § Strategies: • Develop detailed protocols • Ensure data collection of behavioral measures done under the same circumstances for all individuals • Example: how often do you wash your hands? Ø Strategy: collect the data directly while disrupting participants’ lives as little as possible Ø E. g. , put movement sensors in soap, measure the weight of the soap

The Problem § With the following questions…

Question: After the last time you used the toilet, did you wash your hands? (The problem with this indicator is…. ) A. Accuracy B. Precision C. Both D. Neither

Outcome: Gender Bias Question: How effective are women leaders? (ineffective, somewhat effective, very…) A. Accuracy B. Precision C. Both D. Neither

Examples § Study: Double-Fortified Salt (Duflo et al. forthcoming) • Where: Bihar, India • Intervention: selling low-cost iron-fortified salt to at-risk-foranemia families. • Indicators: Ø Physical fitness (directly observable; step test) Ø Physical development (directly observable; anthro measures) Ø Cognitive development (indirectly observable; puzzles, tests) Ø Health history (indirectly observable; surveying on immunizations received, etc. ) 58

COLLECTING DATA

When to Collect Data § [ Baseline ] : Before you start § During the intervention § End line : After you’re done § [ Scale-up, intervention ]

Methods of Data Collection: Not Just Surveys § Surveys- household/individual § Administrative data § Logs/diaries § Qualitative – eg. focus groups, RRA § Games and choice problems § Observation § Health/Education tests and measures

Where can we get Data? The good. . and the bad Administrative data • Collected by a government or similar body as part of operations • May already be collected and thus free • Can be extremely accurate (e. g. electricity bills) • May not exist or not answer the question you want • May itself change behavior (e. g. taxes) Other secondary data • Collected for research or other purposes not admin • May already be collected and thus free • Can inform the larger context of a project • May not exist or not answer the question you want • Dubious quality Primary data • Collected by researchers for study • Address the exact question of interest • Cover channels and assumptions • Very costly and time consuming • May be biased answers

Primary Data Collection § Surveys • Questions • Exams, tests, games, vignettes, etc. § Direct Observation • Personal or technological (e. g. smoke sensor, vibration sensor) § Diaries/Logs

Common Survey Modules Can be Adapted for a Particular Project § Demographics § Economic • Income, consumption, expenditure, time use • Yields, production, etc. § Beliefs • Expectations or assumptions • Bargaining power, patience, risk § Anthropometric § Cognitive, Learning

Primary Data Collection Considerations § Quality • Reliability and validity of the data § Costs / Logistics • Surveyor recruitment and training • Field work and transport, interview time • Electronic vs. paper • Data entry, reconciliation, cleaning, etc. § Ethics § Human subjects, data security

Reliability of Data Collection § The process of collecting “good” data requires a lot of efforts and thought § Need to make sure that the data collected is precise and accurate. avoid false conclusions, for any research design § The survey process: § Design of questionnaire Survey printed on paper/electronic filled in by enumerator interviewing the respondent data entry electronic dataset § Where can this go wrong?

Things to Take-away : § Theory of change guides measurement • Want to measure each step § Data collection all about trade-offs • Quality and cost • Bias (accuracy) and noise (precision) § Creative techniques can sometimes help • Think about what outcomes are most important

Thank you!