Demonstrating the Logic Behind an RCT to Program

Demonstrating the Logic Behind an RCT to Program Staff Session #817 Joel D. Philp The Evaluation Group American Evaluation Association Annual Conference Friday October 17, 2014 Denver, CO

For Today �Background �Demonstrate a simple procedure that might help you explain key concepts behind an RCT to a lay audience �Answer questions and hear your comments

Background - TEG �TEG has evaluated over 150 education-based programs over the last 20 years; �Most are multi-year programs; �Rigorous impact evaluations are increasingly required by many federal agencies (DOE SAMSHA, OJJDP); �DOE follows What Works Clearinghouse standards (WWC).

Background - WWC Designs �RCTs and RDDs meet WWC evidence standards without reservations; �QEDs meet WWC evidence standards with reservations

Background - Challenges when engaging staff in a rigorous evaluation �Terminology Baseline equivalence? Counterfactual? Contrast effect? Implementation fidelity? �Most are unfamiliar with or have forgotten Research Design 101 �Abstract nature of the issue picture, is worth a 1000 words! Demo, like a

Background - Key Requirement for a Successful RCT in the Human Service Field #1. Buy-in from program staff Without buy-in, run the risk of �Sabotage to design fidelity �Disgruntled staff But what does buy-in mean? poor implementation � Giving support willingly, via a shared understanding and agreement

Background - The Question: So, if…. �Staff support is a key ingredient to a successful RCT �Understanding is needed for staff support Q: How do you promote understanding in a nonthreatening, easily digestible way to uninitiated or apprehensive staff? A: By way of simple analogy, using apples and oranges, and referencing the concepts described back to the program.

RCT Demonstration Context to this demonstration: �RCT impact evaluation of a school-based mentoring program serving at risk students �More students applied (312) than can be served (150). �Presentation of RCT to program staff

RCT Demonstration Materials: Bag of apples (8 -10) Bag of oranges (8 -10) A few bananas (optional) Large paper bag Spray bottle with water Time: 20 -30 minutes Objectives: Impart an understanding of: ▪ Baseline equivalence; ▪ Counterfactual and contrast effect; ▪ Treatment fidelity

RCT Demonstration We’ve all heard the term You can’t compare apples and oranges, right? Why not? “In this bag I’ve got everyone who applied to our program (the few bananas over there applied but didn’t meet eligibility criteria). Although we can’t serve them all, we can do a rigorous test of the program. To do that , we will randomly assign our applicants to Tx and control groups. ” Have a staff member reach into the bag and select the first piece of fruit touched (apple or orange) � Alternate placing the selected fruit into either the Tx or control group; � Bananas symbolize non-eligible applicants. �

RCT Demonstration Concept of Baseline Equivalence: 1. Randomization produces two equal groups on: Known characteristics: On the outside they are same color, size, texture, etc. ; Unknown characteristics: they are likely the same on the inside (same color, # of seeds, etc. ); Key point: Randomization cancels out individual differences and creates two equal groups. Program app: 150 TX, 162 control

RCT Demonstration Concept of Baseline Equivalence 2. Randomization is fair. No “creaming” the sample or selection bias. “What happens if we select only perfect apples and oranges into the Tx group? ” Key point: All units have an equal opportunity to be in the Tx group Program app: Randomization done independently via computer

RCT Demonstration Counterfactual and Contrast effect “Now that I’ve got 2 equal unbiased groups, I’m going to give the treatment group the intervention, which is designed to improve the taste of the fruit. ” Spray the treatment group fruit with the spray bottle 5 -6 times Do nothing to the fruit in the control group

RCT Demonstration Counterfactual 1. Given two equal groups, with one getting the Tx and one not, we are able to assess the impact of the intervention compared to no intervention. Key point: without a control group, we would not be able to make causal attribution. Program app: Can say with confidence that mentoring works.

RCT Demonstration Contrast effect 2. But do fruit farmers really do absolutely nothing to improve their fruit? Key point: Probably not, so it’s important to monitor what the control group does get Program app: do students in the control group get anything similar to mentoring?

RCT Demonstration Concept of Treatment Fidelity “Did anyone notice: �How many times I sprayed the fruit? �How close the sprayer was? �The quality of the water I used? ”

RCT Demonstration Concept of Treatment Fidelity Evaluation Term Demonstration Application Dosage With what frequency, intensity, and duration did I deliver the spray? Participation Did all the fruit get sprayed? The same amount of spray? Quality Was it tap water or spring water? Clear or cloudy? Warm or cold? Adherence Was it sprayed correctly? Responsiveness Did the apples and oranges like getting sprayed? Adaptation Did I alter my spray technique?

RCT Demonstration Concept of Treatment Fidelity Evaluation Term Program Application Dosage What is the frequency, intensity, & duration of mentoring? Participation Did all students receive the same amount of mentoring? Quality Was the mentoring of high quality (mentors well trained? On time? Engaging? ) Adherence Was mentoring delivered as planned (1 -1, at school, over lunch)? Responsiveness Did students enjoy being mentored? Adaptation Did mentoring change in response to changing circumstance ?

RCT Demonstration Other adaptations: � To demonstrate outliers, include in the bag a green apple (or a rotten apple!) � To demonstrate multiple outcomes, can taste the fruit; grade its color; measure its size, etc. � To demonstrate overall and differential attrition, some fruit accordingly. remove � To demonstrate contamination and crossover, shift a few pieces of fruit from the control to treatment group.

Questions and Comments Could this be applied to a QED? How might it be to applied to an RDD? How to illustrate differential outcomes? Other adaptations to describe design concepts?

Contact Information Joel Philp Senior Director of Research The Evaluation Group 169 Laurelhurst Ave Columbia SC 29210 (803) 454 -2006 Joel@evaluationgroup. com www. evaluationgroup. com