The Role of Unmet Financial Need in Retention

The Role of Unmet Financial Need in Retention Reuben Ternes Oakland University MI/AIR Nov. 2017

Overview • Primary Question: • If we met the financial needs of all of our students, what would their retention rates be? • Or, alternately, what is the role of unmet financial need in (first year) retention rates?

Definitions & Demographics • We are mostly concerned with Full-Time, First Time cohorted students. • The same group that is reported for typical IPEDS first year retention rates. • Oakland University is a large Midwestern doctoral research university • 19, 333 students (fall 2017) • Heavy undergraduate population • A large working-class student body • Many of whom work…. A lot

Working Hours Percentage of First Year Students Who Report Working Off Campus (NSSE 2016) 52% OU 32% Great Lakes Public National

What is unmet financial need? • • We define unmet need in the following way: Cost of Attendance – non-loan aid – EFC Or, more specifically: Unmet Need = (Tuition + Housing Cost + Miscellaneous Costs) – (Scholarships + Need Aid + Expected Family Contribution) • Scholarships would include all institutional scholarships and outside scholarships • Need aid includes both federal (Pell, Work Study) and institutional need aid • Tuition should be exact (not estimated). • Housing should be exact (if on campus) or estimated (if off campus) • Misc. costs often depend on housing status (on/off), but is generally estimated by the financial aid office.

A Brief Tangent About Measurement Issues

Are Cost of Attendance Estimates Accurate? • Measurement is an often overlooked but critically important piece of all statistical analyses. • It’s worth examining if the cost of attendance (COA) estimates at your institution are an accurate depiction of real costs. • But, how can we tell if the estimate are accurate? Triangulate! • Find an independent estimate to compare with. • One possible source was recently featured on the Chronicle: • http: //www. chronicle. com/interactives/cost-of-living • Created by the Wisconsin HOPE Lab. • OU’s COA was within $7 of what HOPE Lab found.

COA Differences MI Public Universities Name OU UM-D MSU GVSU LSSU WMU NMU UM-AA FSU EMU CMU SVSU MTU UM-F WSU Institution Estimate HOPE Estimate (County Data) Difference $11, 292 $11, 286 $10, 742 $10, 716 $9, 143 $10, 872 $10, 436 $12, 050 $10, 526 $9, 898 $8, 240 $11, 612 $11, 447 $12, 670 $14, 170 $11, 300 $11, 111 $10, 544 $10, 202 $9, 734 $10, 197 $9, 743 $11, 201 $9, 473 $11, 201 $9, 743 $9, 842 $9, 491 $9, 878 $11, 111 $7 $176 $199 $515 $590 $675 $694 $850 $1, 054 $1, 302 $1, 502 $1, 771 $1, 957 $2, 793 $3, 060

More Measurement Issues! • For many SISs, COA is a variable that can be attached to an individual student’s record. • But we couldn’t use that variable. • Why? • Because, tuition, a huge component of COA, was estimated. • A student taking 18 credits was getting the same tuition estimate as a student with 14 credits. But our tuition is based entirely on credits! • In addition, COA estimates sometime change if the student indicates they are attending only one semester. • So the data needed further cleaning to make sure that COA estimates were comparable for students that attended for one semester vs. those that attended for the whole year. • All COAs were recalculated for students based on actual credits. • Students that attended only for the fall semester had their COA’s doubled (as well as their disbursed aid) to keep them in relative sync with students that attended for both fall and winter.

Measurement issues abound! • Expected family contribution is an important part of the formula used here. • What about students that did not complete the FAFSA? • It was then assumed that these students had no unmet need. • Is that a true statement? Probably not. • But it’s hard to assume otherwise.

Now back to our regularly scheduled program…

The Sample • Examined 2015 FTIAC class • Full time only. • 70 students were excluded because they started in fall 2015 with a part time status. • Also excluded international students and out of state students. Their COA’s are way different, and probably deserve special analysis. • Plus, we don’t have many of them. 56 total students excluded here. • Also excluded students that started full time, but then dropped to part time in winter. • This decision could be criticized. I’m still not sure if it was the right decision or not. But I found it difficult to create an appropriate COA for these students. • 63 students met this exclusion criteria. • Final Sample was 2, 084 • Total exclusions represented only 7% of the total 2015 class.

Ok. Some Demographics Table 1. 2015 Full-Time FTIAC Outcomes by Financial Need Category 1 st Unmet Semester Avg. Financial % of Retentio Probation Pell % HS Need N Total n Rate URM GPA None 1033 40% 79% 8% 2% 9% 3. 48 $1 -$3 k 222 9% 86% 8% 23% 13% 3. 70 $3 k-$6 k 308 12% 84% 5% 45% 12% 3. 56 $6 k-$9 k 355 14% 79% 9% 60% 19% 3. 39 $9 k-$12 k 259 10% 74% 16% 65% 22% 3. 23 $12 k+ 411 16% 50% 30% 72% 43% 3. 06

Table 1 Highlights (A) • That’s a lot of info in a little table. Let’s break it down. 1. Retention rates are (mostly) stable until the highest unmet need category • This suggests that moderate unmet financial need may not be an insurmountable retention hurdle. Extreme need is clearly disruptive.

Table 1 (A) – Higher unmet need = lower retention rates Table 1. 2015 Full-Time FTIAC Outcomes by Financial Need Category 1 st Unmet Semester Avg. Financial % of Retentio Probation Pell % HS Need N Total n Rate URM GPA None 1033 40% 79% 8% 2% 9% 3. 48 $1 -$3 k 222 9% 86% 8% 23% 13% 3. 70 $3 k-$6 k 308 12% 84% 5% 45% 12% 3. 56 $6 k-$9 k 355 14% 79% 9% 60% 19% 3. 39 $9 k-$12 k 259 10% 74% 16% 65% 22% 3. 23 $12 k+ 411 16% 50% 30% 72% 43% 3. 06

Table 1 Highlights (B) 1. Higher unmet need is correlated with a host of other risk characteristics: • • Higher Pell Rates Lower HS GPA Higher first semester probation rates Minority status

Table 1 (B) - Higher unmet need is correlated with a host of other risk characteristics: Table 1. 2015 Full-Time FTIAC Outcomes by Financial Need Category 1 st Unmet Semester Avg. Financial % of Retentio Probation Pell % HS Need N Total n Rate URM GPA None 1033 40% 79% 8% 2% 9% 3. 48 $1 -$3 k 222 9% 86% 8% 23% 13% 3. 70 $3 k-$6 k 308 12% 84% 5% 45% 12% 3. 56 $6 k-$9 k 355 14% 79% 9% 60% 19% 3. 39 $9 k-$12 k 259 10% 74% 16% 65% 22% 3. 23 $12 k+ 411 16% 50% 30% 72% 43% 3. 06

Table 1 (C) – The highest category of financial need is also one of the largest. Table 1. 2015 Full-Time FTIAC Outcomes by Financial Need Category 1 st Unmet Semester Avg. Financial % of Retentio Probation Pell % HS Need N Total n Rate URM GPA None 1033 40% 79% 8% 2% 9% 3. 48 $1 -$3 k 222 9% 86% 8% 23% 13% 3. 70 $3 k-$6 k 308 12% 84% 5% 45% 12% 3. 56 $6 k-$9 k 355 14% 79% 9% 60% 19% 3. 39 $9 k-$12 k 259 10% 74% 16% 65% 22% 3. 23 $12 k+ 411 16% 50% 30% 72% 43% 3. 06

What does Table 1 really tell us? • So, what do these things all mean? • Basically, Table 1 illustrates the complicated relationship between unmet need, student characteristics, and student outcomes. • It’s not going to be as simple as ‘if we give these students more money, our retention problems will go away’. • However, the question remains: if we did address their financial struggles, how much improvement would we expect to see in first year retention rates?

Regression to the Rescue! • With some appropriate control variables, we can start to address this question using a basic logistic regression. • Though we may not be able to answer it with complete confidence. • For the most part, regression is still a correlational tool. And correlation does not imply causation.

Sample (yet again) • • 2015 FTIACs (First Time in Any College) N = 2, 084 Excludes • Part-time students • Non-resident/out of state students • Includes • Students that did not fill out the FAFSA • EFC set to zero in these cases • This is a large group of students!

Variables • Outcome of Interest • First Year Retention (Binary) • Control Variables • • Unmet Need (in thousands of USD) Expected Family Contribution (in thousands of USD) ACT Composite Scores High School GPAs First Term Credits Housing Indicator (binary yes/no) First Generation Status (binary yes/no) Underrepresented Minority Stats (binary yes/no)

Why Not Add More Control Variables? • I could have added more control variables • For example, SES estimates or interaction terms • I chose not to • I have two main concerns with adding a large number of additional control variables 1. Interpretability 2. Overfitting • My goal is not to necessarily develop a comprehensive explanatory model, but rather, to estimate a generalized impact of changing financial aid dollars

Results Variable Unmet Need EFC ACT HS GPA First Term Credits Housing Indicator First Generation Indicator URM Status Constant B -0. 091 -0. 007 -0. 021 1. 044 0. 124 -0. 116 -0. 001 S. E. 0. 010 0. 003 0. 018 0. 147 0. 034 0. 116 0. 154 Wald 78. 652 6. 090 1. 343 50. 760 12. 972 0. 998 0. 000 Sig. 0. 000 0. 014 0. 247 0. 000 0. 318 0. 994 Exp(B) 0. 913 0. 993 0. 980 2. 841 1. 132 0. 890 0. 999 0. 258 -3. 209 0. 141 0. 677 3. 340 22. 483 0. 068 0. 000 1. 295 0. 040 Nagelkerke R 2 = 0. 192

How is this even remotely useful? • Normally in a regression study, one would want to examine the significance level of some variable of interest and then write a really long paper about it. • The only variable I am interested in is unmet financial need. • And I don’t need to do a regression analysis to understand that this is an important variable. I think everyone already knows that. • Instead, I want to probe what happens to the outcome variable (retention rates) when (unmet need) is systematically decreased. • Q: How to do this? • A: Math

Logistic Regression Math • Retention Rate = • Those beta weights correspond to the B column in the results slide. All you have to do is plug them in and go. • In order to get our final answer though, we have to run some estimates on what the average might be for the other variables. • Some variables, like first generation, have basically no impact on the outcome at all. • But you can run a number of different scenarios to probe how the retention rate changes for various student demographics.

Results Variable Unmet Need EFC ACT HS GPA First Term Credits Housing Indicator First Generation Indicator URM Status Constant B -0. 091 -0. 007 -0. 021 1. 044 0. 124 -0. 116 -0. 001 S. E. 0. 010 0. 003 0. 018 0. 147 0. 034 0. 116 0. 154 Wald 78. 652 6. 090 1. 343 50. 760 12. 972 0. 998 0. 000 Sig. 0. 000 0. 014 0. 247 0. 000 0. 318 0. 994 Exp(B) 0. 913 0. 993 0. 980 2. 841 1. 132 0. 890 0. 999 0. 258 -3. 209 0. 141 0. 677 3. 340 22. 483 0. 068 0. 000 1. 295 0. 040 Nagelkerke R 2 = 0. 192

Bottom Line • Each $1000 that we decrease Unmet Need by, retention rates increase by about 1. 6%. • This finding is very consistent with previous research that suggests that every $1000 equates to somewhere between 1% and 2% change in retention rates. • Includes other regression studies • Includes Regression Discontinuity studies • Includes Interrupted Time Series studies • Even if we add $5000 of additional aid to the most needy group, we would expect their retention rates to improve from 50% to less than 60%. • That is a lot smaller of an impact than I suspect many other administrators would expect.

Policy • Results like these can really impact policy. • At $5 k per student, can we do better than a 10% improvement in retention rates with other services? • A money only policy to student retention is unlikely to have the impact that many administrators might intuitively think. • To address retention, especially for those with high unmet need, a more comprehensive approach will be required.

Confidence (Error Estimates) • Q: As this is only a correlational design, how confident am I in my own conclusions? • Can’t use p-values to answer this question • A: Somewhat • Pros • Several control variables were added to reduce confounding factors • Results are consistent with previous research • I’m not testing multiple hypotheses (or any really) • Strong theoretical foundation (we know money matters) • Cons • Lots of data decisions to make. Unlikely that all of them were ideal. • A high amount of ‘Researcher Degrees of Freedom’ existed due to all of the data decisions. • Overall effect size is modest to begin with. Changing the Unmet Need by a single unit of standard error results (approx. ) in a 0. 7% change in retention rates. So two units of standard error would be almost the same as a $1000 change (1. 6%).

Limitations/Criticisms • Doubling the COA (and aid) for students that attended only fall. • I think that one could criticize this decision. Alternatively, you could analyze only fall data (COA and aid). • Doing so is awkward, but it might be more ‘accurate’ • I excluded part time students. • I also excluded non-resident students and out of state students. • I did not include any additional SES estimates – which I usually do. • This means the ACT/HS GPA data is also conflated with general SES. So, the concept of ‘academic preparedness’ is conceptually conflated with general SES. • I also included all students – setting those students without FASFA information to an EFC of zero. • I’m unsure if this is the right thing to do, or the wrong thing to do.

The End • Questions?