Advanced methodology and statistics Field experiments multilevel models

Advanced methodology and statistics: Field experiments, multilevel models, interrupted time series, and metaanalysis Week 2. 2

Key questions Say you have an idea to solve a problem. 1. How do you design a field experiment to know if your idea works? 2. After you design an experiment, how are you going to analyze the data? • Are there new techniques more appropriate to analyze field data than conventional techniques?

Lab vs. field experiments High internal validity High external validity

Behavioural change and field experiments • Ideally you want your intervention to have real-world impact • Hence you want to test your theories in the real world • But the real world is “messy”

Experiments in the larger context Lab and field experiments are complementary Hypothesis Test in lab Test in field

Issues in field experiments

Where are your participants? • Incentivizing • What kind of incentives will you give? • Are people motivated by money? • What features of the intervention might cause potential participants to decline to enroll, such as lack of child care, fear of husbands’ retaliation, etc. • How many people do you need, and in what structure? (next section on multilevel modeling) • 500 people from 5 villages, or 5 people from 500 villagers? • Either way you get N = 2500, so does it really matter? • Attrition • What are the reasons for attrition? Can you prevent them?

Who are your “participants”? • You want to know whether people who are exposed to a social norm saying “most people do not bring cutlery out of the mess” would return a stray cup. How do you define who a participant is?

Is randomization feasible? • In field settings, control groups are often geographically distinct to prevent communication between treatment and control groups. • You may not be able to find such geographically distinct groups, or if you do, they may differ in significant ways from the treatment group and as such, confounding your findings (in other words, such groups creates more problems than it solves) R i v e r The situation This is not the solution This is ideal, but not always possible

Measurement reliability and validity • Measurement is fundamental. • Take a measurement as ‘simple’ as toilet usage in an open defecation intervention. • Q: How would you measure if people actually use the toilets that were built? • Some possibilities: • By asking them? • By installing cameras? • Using Passive Latrine Use Monitor (PLUM)? Clasen et al. (2012). Making sanitation count: Developing and testing a device for assessing latrine use in low-income settings. Environ Sci Technol.

Language barriers • Bear in mind that not everyone • Understands likert scales the way you do • Understands concepts the way you do • Concrete concepts: # chickens • Abstract concepts: love

Class Exercise • You want to study whether a litter-filled carpark would lead drivers to litter more (heard of the ‘broken windows’ theory? ). • You want to study whether a litter-filled park would lead visitors to litter more.

My experience • Anything that can go wrong will go wrong. Prepare for contingencies. • Get out of your office (or study room). Be on the ground. • Even if it means waking up at 6 am in the morning.

Multilevel modeling

Clustering of real world data in field experiments • When individuals come into the lab, they usually come in alone (unless it is a group interaction experiment) • Any manipulation (e. g. , watching Mr Bean) affects A, B, C, D, etc. independently from one another. • What if A, B, C, D come in together?

Nested data in the real world • Researchers want to know whether Textbook A or B results in better student learning. They randomly assigned Class IV classrooms to either use Textbook A or B. • What’s the problem? • Students in each classroom experience the same teachers, same friends, etc. The data is inherently dependent within each classroom

Visualizing clusters and variances Open defecation intervention: Say you want to measure toilet usage of a particular individual over a period of time. There will be many sources of variance.

Nested data in the real world: A simple 2 -level model In SRM II, you learnt to recognize nested models. In this course, you learn to account for nesting in your analysis. School Level 3 (but we are not considering this) Teacher Y Teacher X Level 2 Level 1 (individual) Student A Student B Student C Student D Student E Student F Variables Praise type Gender of teacher Academic performance Gender of student You want to know whether a particular instructional intervention (e. g. , praising results vs. effort) administered by teachers influence academic performance. • Gender of teacher • Gender of student

Multilevel modeling: Key questions • How can we account for within-group similarities (jargon: intraclass correlation)? • • Students within classrooms Residents within dorms Inhabitants within households Dwellers within villages • How can we know whether our intervention applied at a higher level works, after accounting for within-group similarities?

Multilevel modeling has broad applications • Ecology: bugs|trees • Criminology: victims|households • Marriage: individuals|couples • Family: children|households • Education: schools|district • Medicine: patient|doctors • Public health: people|districts • I/O: employees|managers

Important: Sampling size issue • Which is better? 30 students from each of 5 classes Easier to execute Cheaper 5 students from each of 30 classes But this is the correct answer. Why?

The math behind MLM • The only thing you need to know from this slide is that there are equations to express MLM. But you do need to know how to read a regression table (review SRM I).

Interrupted time series

What is time series? • A time series is a series of data points indexed (or listed or graphed) in time order. (Think of a stock market graph. ) • Sometimes we want to know whether an intervention that occurred at a particular point in time produces the intended outcome • An interrupted time series is a time series of a particular outcome of interest is used to establish an underlying trend, which is ‘interrupted’ by an intervention at a known point in time.

Why use time series? • Traditional pre- / post-type of data does not allow for ‘seasonal’ variations or cyclical trends • E. g. , open defecation rates differ in weekends vs. weekdays • Analysis of archival data • E. g. , number of road deaths after a mandatory seat belt law is passed

Why won’t you have a control group? • You are working with a small population • For a medium effect size, d = 0. 50, to detect a statistically significant result with 80% power between groups (expt vs. controls), you need N = 100 (two groups, n = 50 each) • If you are working in a small community with 30 people, a between-subjects design with n = 15 is statistically unsound (if your effect sizes are small) • No one wants to be in a control group • Especially if they know that ‘the other person is receiving treatment’

Possible impact models of interrupted time series Types of impact models (a)Level change (b)Slope change (c) Level and slope change (d)___________ (e)___________ (f) ___________ Q: Can you think of other possible models?

Segmented regression model •

Meta-analysis

What is meta-analysis? • A statistical procedure for combining data from multiple studies in order to estimate the ‘true’ effect size in the population

Effectiveness of psychotherapy for severe somatoform disorder: Physical symptoms Koelen et al. (2014). Effectiveness of psychotherapy for severe somatoform disorder. Br J Psychiatry.

Effectiveness of psychotherapy for severe somatoform disorder: Psychological symptoms Koelen et al. (2014). Effectiveness of psychotherapy for severe somatoform disorder. Br J Psychiatry.

Why can’t we just stick to the findings of one study? • Would you visit a surgeon whose only experience in surgery was with one patient (even if the patient got well)? • P-value is a conditional probability • Statistical significance a function of your alpha level (typically. 05), sample size, and effect size • P-values ≠ effect sizes • This implies that given the same effect size, a study with a large N will have a small p. • Meta-analyses show the effect sizes (in d, eta, OR, etc. )

One goal of science •

Forest plots represents effect sizes intuitively You usually have a diamond at the bottom. The center of the diamond shows the mean effect size; the edges show the 95% CI of the effect size

Funnel plots and publication bias • Looks good What if the ones in red were not published? How will the funnel look like? Reality of scientific publishing 1. Non-significant findings are hard to publish (“file drawer effect”). 2. There are 1000 ways to screw up your study, and only one way to get it right.

Implications for behavioural change • You are the Director of Education in your country. You want to improve educational outcomes for children. You believe that motivational interventions work. • Which sort of motivational interventions work best? How much will the improvement be? Do the benefits outweigh the costs? Lazowski & Hulleman (2016). Motivation interventions in education: A meta-analytic review. Rev Educ Res.

Meta-analysis and public policy • Most of the time you won’t need to do a meta-analysis • But you do need to know how to read a paper that reports meta-analytic findings, especially if you are a policy maker (evidence-informed policy) • It is irresponsible to make policy decisions based on gut feelings, when you know that the evidence is out there. • When you read these findings, it’s good to translate what the effect sizes mean in impactful quantities (e. g. , how many people could be saved with Drug X).

Take home messages • More methodological/statistical issues need to be considered when dealing with data outside lab • Some of these issues are peculiar to field settings, and hence require a different skillsets. (If you are serious to pursue this as a career, arm yourself with the necessary skillsets. ) • At this stage, you need to learn how to recognize these issues; solving them will require graduate training, but if your grounding in statistics is strong, you can pick up these skills with reading and practice.