Differential Outcomes of An AfterSchool Program An Evaluation

Differential Outcomes of An After-School Program: An Evaluation of the Combined Effects of Program Participation and Students’Antecedent Characteristics

Authors: Angelica Adams, Communities in Schools of Houston Heidi Brasher, YMCA of Greater Houston Amy Corron, United Way of Greater Houston Emily Faron, United Way of Greater Houston Karla Kessler, Alief Independent School District Cheryl Mc. Callum, Childrens’Museum of Houston The “Houston’s Kids” Team Roger Durand, External Evaluator, Durand Research and Marketing Associates, LLC (Contact: Rogerdurand 21@Hotmail. com)

Abstract We report results from an evaluation of the combined and interactive effects of students’ participation levels and students antecedent characteristics (especially social disadvantages such as poverty status and limited English proficiency) on the outcomes of the Houston’s Kids After-School Program. We found that students’ backgrounds combined in often complicated ways with levels (“dose”) of participation in yielding unexpected outcomes.

Background Houston’s Kids (HK) was created in 2006 in the aftermath of Hurricane Katrina. A collaborative program of the United Way of Greater Houston, the YMCA of Greater Houston, the Alief YMCA, Communities in Schools of Houston, the Alief Independent School District, and the Children’s Museum of Houston. Purpose of HK: To address the needs of disadvantaged children and youth with a particular emphasis on developing the assets of elementary and intermediate school children.

Principal Evaluation Questions Were program participants’ outcomes a result of the HK program or were they a result of students’ backgrounds? (Spurious attribution of outcomes to the program? ) Did program participation matter more to HK outcomes among some students than others? If so, which students?

Data and Methods Data were gathered on a “true panel” of 333 program participants and 300 control subjects from six (6) program sites over an academic year. Data on students’ program participation records were merged with demographic and outcomes (school absences, mandatory state test scores, changes in math, reading, and conduct grades) variables. Multiple statistical methods were used (bivariate models suitable for the level of measurement, OLS, Discriminant Functions, contextual modelling procedures, maximum likelihood and regression discontinuity methods). Basic spuriousness test model: Yi = a + B 2 X 2 + B 1 X 1 + ei, Basic interaction model: Yi = a + B 1 X 1 + B 2 X 2 + B 3 X 1 X 2+ ei, Where Yi is individual I’s outcome of interest, X 1 is participation level and X 2 is a program antecedent condition

Baseline Findings: Dose-Response We found evidence of dose-response relationships at each of the six (6) Houston’s Kids program sites. Regular, most active program participants were found to outperform control subjects and less active participants on all outcome performance measures: (passing state mandatory reading and math tests, fewer school absences, improvement in reading, math, and conduct grades).

Key Analysis and Findings Program outcomes were studied in relation to participants’ attendance (dose) and to the following antecedent, social characteristics: poverty status, Limited English Proficiency, gender, and race/ethnicity. The most common finding: the effect of program dose on a positive performance outcome was reduced slightly, but not eliminated, by introducing a control for individuals’ antecedent conditions. Little evidence that positive program outcomes were spurious owing to background characteristics. Evidence was found in several instances that an antecedent background variable spuriously suppressed a dose-response relationship until that antecedent variable was introduced as a control.

Conclusions The HK program itself – especially the program’s dose -- and not student “mix” in the form of students’ backgrounds -- appears to be largely responsible for positive, program outcomes. But we also found that students’ backgrounds combined in often complicated ways with levels (“dose”) of participation in yielding unexpected outcomes.

Discussion: Action Implications The findings suggest the need to “fine tune” program activities, including support ones, in light of the differential outcomes identified here. Program activities that involve a less intensive use of English (say, sports or number exercises); that require fewer resources than are typical for impoverished families (e. g. , transportation alternatives to a family auto or more learning resources at home); that may be more culturally diverse and inclusive; and that may appeal more to the different interests of boys compared to girls.