PRELIMINARY PLEASE DO NOT CITE OR TWEET WITHOUT

PRELIMINARY. PLEASE DO NOT CITE OR TWEET WITHOUT AUTHORS’ PERMISSION Beyond Triage: A Randomized Experiment in Sustained Pre-College Advising ERIC P. BETTINGER BRENT J. EVANS WITH ANTHONY ANTONIO, JESSE FOSTER, EILEEN HORNG, RIE KIJIMA UC DAVIS – 15 MARCH 2015

Motivation Non-financial barriers related to information deter students from preparing for and enrolling in higher education, especially among low-income and minority students (Avery & Kane, 2004) Gauntlets ◦ Financial aid (ACSFA, 2005) ◦ Admission process (Klasik, 2012) High school counselors are overloaded and cannot meet the demand in their schools. Huge investment by college access programs to provide information and guide students through the process ◦ TRIO (> $800 million) ◦ Countless small local programs Models vary substantially

Literature Information and guidance matter at very specific stages in the process ◦ H&R Block Study (Bettinger, Long, Oreopoulos, & Sanbonmatsu, 2012) ◦ Information and guidance related to financial aid improves aid receipt and college enrollment ◦ 2 -8 percentage point effect ◦ Summer Melt Studies (Castleman, Page, & Schooley, 2014; Castleman, Arnold, & Wartman, 2012) ◦ Outreach to students in the summer between high school completion and college enrollment improves likelihood of enrolling ◦ 3 -14 percentage point effect Holistic advising ◦ Mathematica’s Upward Bound Study (Myers et al. , 2004) ◦ Primarily academic supports to promote college enrollment, but also provide information ◦ RCT found no effects on college enrollment although some evidence of shift from two-year to four-year ◦ College Possible (Avery, 2013) ◦ Targeted ACT prep, information, and admission support ◦ High-touch intervention-360 hours ◦ Large effects (15 pp) on shifting from two-year to four-year college but no overall enrollment effects ◦ Dartmouth Outreach (Carrell and Sacerdote, 2013)

Need for evidence of different models Different models of providing information and guidance ◦ Targeted versus school wide ◦ In school versus out of school ◦ Near-peer By investigating wide range of models, we can start to tease apart mechanisms Need well identified, causal estimates of efficacy Randomized controlled trial evidence should be privileged where possible

Advise TX Part of the College Advising Corps model (active in 15 states 350 -400 high schools) Goal is to help low-income, first-generation college students attend and succeed in college Recent college graduates serve as full-time near-peer advisers to entire student body at a high school Full school college advising intervention-concentrate on seniors but some time spent developing college plans for underclassmen Encourage attendance; provide information; assist with selection of colleges, college applications, and FAFSA completion Program pilot in 2010 -2011 in 15 schools with rapid expansion to over 120 schools in 2011 -2012 school year

Experimental Design School level randomization Schools invited to apply ◦ > 35% FRL ◦ < 70% graduates attend college within one year (average is 45% in 2009) ◦ < 55% students undertaking a “distinguished” college-prep curriculum Schools were ranked on these 3 criteria and a qualitative “fit” component 84 schools automatically selected among 237 who applied Next 111 schools eligible for random assignment (experimental sample) 36 randomly selected for treatment assignment, others assigned control status (no program) Blocked on region: individual lotteries were held within each region (23)

Data School level data from Texas Education Agency (TEA) Student level data from Texas Higher Education Coordinating Board (THECB) 2 years of enrollment outcome data ◦ From 2011 -2012 (2012) and 2012 -2013 (2013) high school graduating classes ◦ All in-state public postsecondary institutions ◦ Our results are lower bounds of effect of program if the program had any effect on out of state or private enrollments ◦ Outcomes: 2 -year and 4 -year enrollment and enrollment in Fall after high school graduation Preliminary results from student surveys conducted in treatment/control schools

Treatment-Control Balance Panel A: School Level Variable All TX High Schools Mean All Experiment High Schools Stdev. Mean All Treatment High Schools Stdev. Mean Raw Difference T-C Stdev. Difference T-C Difference with Lottery Controls P-value Difference P-value White 0. 400 0. 311 0. 225 0. 227 0. 214 0. 223 -0. 017 0. 718 -0. 015 0. 665 Black 0. 123 0. 177 0. 171 0. 217 0. 208 0. 068 0. 049 0. 085 0. 006 Hispanic 0. 440 0. 313 0. 568 0. 273 0. 526 0. 276 -0. 062 0. 266 -0. 083 0. 033 Asian 0. 017 0. 045 0. 021 0. 029 0. 037 0. 011 0. 060 0. 013 0. 017 Other race 0. 021 0. 041 0. 015 0. 016 0. 014 0. 011 -0. 001 0. 783 0. 000 0. 948 Low-income 0. 542 0. 258 0. 635 0. 179 0. 636 . 0169 0. 001 0. 980 0. 006 0. 843 Grad Rate* 0. 804 0. 217 0. 800 0. 100 0. 802 0. 096 0. 003 0. 899 0. 014 0. 470 Total Students 728 891 1683 838 1848 956 243 0. 154 194 0. 210 Total Seniors 159 209 370 187 418 220 71. 8 0. 058 62. 2 0. 066 N 1785 111 36

Descriptive Statistics & Balance Relative to all TX high schools, experimental schools ◦ ◦ Are much larger (1683 to 728) Have a higher percentage of low-income status (64% to 54%) Have a higher share of minority students (77. 5% to 60%) Have same graduation rate (80%) Treatment schools relative to control schools ◦ ◦ Have more Black and less Hispanic students (8 percentage points difference) Have slightly more Asian students (1 percentage point difference) Same graduation rates and college enrollment rates We control for race in all models to account for this imbalance

Compliance 5 of 36 treatment schools declined to participate or eventually left the program 9 of 75 control schools eventually received treatment ◦ Although we had a randomly determined waitlist, it was not consistently adhered to by program staff ◦ E. g. Austin loved the program and offered to pay for all control and treatment schools to participate after the first year. Treatment on the treated estimates are approximately 33% higher than Intent to treat estimates Treatment Received Control Received Total Treatment Assigned 31 5 36 Control Assigned 9 66 75 Total 40 72 111 0. 745 (0. 072) Lottery controlled regression of treatment received on treatment assignment

Main Impacts Panel B: Separate Treatment Years Outcomes Enrolled in Higher Education Fall after HS Treatment Year Model 1 2012 0. 021 * (0. 0102) Model 2 0. 0146 (0. 0092) Model 3 0. 0146 (0. 0090) 2013 0. 0133 (0. 0105) 0. 0089 (0. 0094) 0. 0089 (0. 0092) Enrolled in 2 Year Fall 2012 after HS 0. 019 (0. 0118) 0. 0208 + (0. 0112) 0. 0208 + (0. 0110) 2013 0. 0061 (0. 0122) 0. 0084 (0. 0114) 0. 0084 (0. 0112) Enrolled in 4 Year Fall 2012 after HS 0. 0053 (0. 0082) -0. 0029 (0. 0083) 2013 0. 0077 (0. 0082) 0. 0011 (0. 0076) Controls SE Clustered at Schoolx. Year Level X X X N 77823

Intent to treat estimates 1. 2 – 1. 5 pp impacts on enrolling in fall after high school completion (significant at 10% level) ◦ Impacts are observed in first year of treatment ◦ enrollment effects on two-years become more significant if just examine first year If we pool the data, small and insignificant effects (~ 1 pp) on overall enrollment ◦ Concentrated at 2 -years ◦ Point estimates at 4 -years close to 0 All second year impacts attenuated

Impacts on Subgroups Outcomes Coefficient Subgroup Female White Black Hispanic Asian Low. Income Enrolled in Higher Treatment Education Fall Main Effect after HS Interaction Effect 0. 0167 * (0. 0076) 0. 0155 * (0. 0073) 0. 0147 * (0. 0071) -0. 0028 (0. 0095) 0. 0124 + (0. 0064) -0. 01 (0. 0090) -0. 0100 (0. 0081) -0. 0197 (0. 0140) -0. 0157 (0. 0139) 0. 0257 + (0. 0131) -0. 0183 (0. 0270) 0. 0345* (0. 0120) Enrolled in 2 Year Fall after HS Treatment Main Effect 0. 0116 (0. 0080) 0. 0165 + (0. 0087) 0. 0215 * (0. 0086) -0. 0047 (0. 0103) 0. 0155 + (0. 0079) 0. 0041 (0. 0097) Interaction Effect 0. 0059 (0. 0083) -0. 0103 (0. 0127) -0. 0370 * (0. 0163) 0. 0340 ** (0. 0128) -0. 0264 (0. 0263) 0. 0166 (0. 0104) Enrolled in 4 Year Fall after HS Treatment Main Effect 0. 0072 (0. 0061) 0. 0016 (0. 0058) -0. 0045 (0. 0059) 0. 0015 (0. 0085) -0. 001 (0. 0055) -0. 0135 (0. 0085) Interaction Effect -0. 0161 ** (0. 0060) -0. 0127 (0. 0147) 0. 0191 (0. 0133) -0. 0041 (0. 0092) 0. 0045 (0. 0290) 0. 0203 * (0. 0097) N 77823 77823

Subgroup Analysis Impact on Low-income students ◦ More likely to enroll overall, split between two-years and four-years (2 -3 pp) Impact on Hispanic students ◦ More likely to enroll overall, mostly at two-years (2 -3 pp) Impact on Black students ◦ Less likely to enroll in a two-year (3 pp) ◦ Some evidence of substitution into four-year, but four-year enrollments not significant Impact on Female students ◦ Less likely than men to enroll in four-year and start at a four-year in fall after high school (1. 6 pp)

Survey Sample Surveys conducted in spring 2014 ◦ All treatment schools (36 + additional randomized group from 2013) ◦ 42 control schools Treatment/Control Balance ◦ Less representation among Hispanic students (as in the administrative data) Surveys conducted in spring 2014 ◦ All treatment schools (36 + additional randomized group from 2013) ◦ 42 control schools

Preliminary Survey Results Key Differences ◦ ◦ ◦ ◦ Degree expectations improve Plans to work full- or part-time after college or to enter the military decrease Similarly likely to participate in “soft” college prep activities (college visits, website, test prep) More applications submitted and greater likelihood of receiving help More likely to attempt AP exams or to retake college entrance exams More likely to complete FAFSA (self-reported) More likely to have submitted a deposit Less likely to have talked to a high school counselor or teacher about admissions

Discussion Unsurprising that treatment effects are small at the school level Advisers likely spend time with inframarginal students who are already going to college as opposed to targeting support at students who need it most Degree of complementarity is potentially important part of the story Cost-benefit analysis Scalability Future work ◦ Will examine persistence outcomes to observe whether advisers have effect on “fit”