Kevin Eagan Sylvia Hurtado Bryce Hughes Tanya Figueroa

Kevin Eagan, Sylvia Hurtado, Bryce Hughes, & Tanya Figueroa, UCLA Association for Institutional Research Annual Forum Orlando, FL May 28, 2014 The Impact of Undergraduate Interventions on STEM Student Outcomes

Introduction �Colleges and universities have been challenged to produce an additional one million STEM degrees over the next decade �The NSF, NIH, and institutions have invested heavily in interventions which have been shown to improve academic performance and retention in STEM �Supplemental instruction and faculty mentoring are two cost-effective types of interventions often already provided at institutions

Purpose �To examine the effect of supplemental instruction and faculty mentoring on STEM identity, intentions to enroll in a STEM graduate program, and commitment to a STEM career �To utilize a quasi-experimental statistical modeling technique to better isolate the effects of these interventions on the three outcomes of interest

Supplemental Instruction �Developed by Deanna Martin at the University of Missouri-Kansas City �Targets “at-risk courses” as opposed to “at-risk students” �Peer-facilitated sessions focused on problem solving and enhancing course material �Voluntary; not remedial �Supplemental instruction has been shown to improve academic performance and term-toterm retention rates in single-institution studies

Faculty Mentoring �Intentional support, as opposed to happenstance faculty-student interactions �Consists of professional and personal support activities �Faculty mentoring also improves academic performance and retention �However, students who typically seek faculty mentoring are students already positioned to succeed

Conceptual Framework � Situated Learning Theory (Lave and Wenger, 1991) STEM as a community of practice Learning as legitimate peripheral participation Through the process of learning new members become more central to the community through identifying with the community � Social Learning Theory (Bandura, 1971) Learning is a cognitive and behavioral process that occurs through both observation and modeling Learning results from a dynamic interaction between cognition, environment, and behavior � Theory of Planned Behavior (Ajzen, 1991) Intentions are a crucial precursor to behavior

Methods �Data Source and Sample Longitudinal Dataset ▪ 2004 CIRP Freshman Survey ▪ 2008 CIRP College Senior Survey NIH and NSF funding augmented participation of MSI’s and STEM-producing institutions 4, 166 longitudinal student cases who intended to major in STEM across 237 institutions

Methods �Dependent Variables STEM identity – Four-item factor ▪ Becoming an authority in my field ▪ Making a theoretical contribution to science ▪ Receiving recognition from others for contributions to my field ▪ Finding a cure for a health problem Commitment to a STEM career (dichotomous) Intentions to pursue STEM graduate study (dichotomous)

Methods �Independent Variables Participation in Supplemental Instruction Receipt of Faculty Mentoring – 9 -item factor Each is dichotomized for the propensity score matching analysis ▪ Supplemental instruction: Students who participated (frequently or occasionally) versus non-participants ▪ Faculty mentorship: Above average (>50) mentorship versus average or below average (<50)

Methods �Control variables Background characteristics Pre-college academic preparation Pre-college aspirations and expectations Initial measures of STEM identity, plans to pursue a STEM career, and expectations of pursuing graduate study in STEM (i. e. Pretest)

Methods �Analytic Strategy Missing data addressed through EM algorithm Propensity score matching ▪ Probit regression ▪ Precollege characteristics and experiences predicting mentorship and supplemental instruction participation ▪ Nearest neighbor matching ▪ T-tests conducted with matched sample for each intervention for each of the three outcomes

Methods �Limitations Secondary data analysis Propensity score matching only as good as the variables available Two outcomes measure intentions rather than actual behavior

Findings – Predicting Supplemental Instruction �HS GPA (+) �STEM identity as a freshman (+) �HPW talking with teachers outside of class in HS (+)

Findings – Predicting Above Average Mentorship �Race: Latino vs. White (-) �Race: Asian American vs. White (-) �HS GPA (+) �Mother’s education (+) �STEM identity as an incoming freshman (+) �Reason for attending college: Prepare for graduate school (+) �HPW: Talking with teachers outside of class in HS (+) �Major: Engineering or computer science (-) �Concerns about ability to pay for college (-)

Effects of SI and Mentorship on Outcomes Supplemental Instruction Mentorship STEM Graduate School STEM Career STEM Identity Aspirations Mean Difference Sig. 0. 63*** 0. 05* 0. 03 1. 10*** 0. 06** 0. 04*

Effects of SI and Mentorship on Outcomes for URM Students STEM Identity STEM Graduate School Aspirations STEM Career Aspirations Mean Difference Sig. Supplemental Instruction 0. 63*** 0. 05 0. 06* Mentorship 1. 30*** 0. 07** 0. 06*

Discussion � Supplemental instruction as a way to establish a community of practice Strengthens students’ STEM identity Increases likelihood to plan to enroll in STEM graduate programs Particularly beneficial for URM STEM identity development � Faculty Mentorship Benefits of mentorship extend even after accounting for the types of students likely to receive or seek out mentorship Mentorship even more impactful for URM students’ STEM identity development

Implications �Undergraduate research can be a resource- intensive intervention Supplemental instruction and faculty mentoring are additional important STEM persistence tools Structure or web of opportunity in STEM �Lends support for further expansion of supplemental instruction offerings and for broader access to intentional faculty mentoring �Propensity score matching provides a method for reducing bias due to participant self-selection when assessing STEM interventions

Future Research �Examine effects of mentorship, supplemental instruction, and other interventions on longer -term outcomes STEM graduate program enrollment Entry into STEM workforce

Contact Information Administrative Staff: Dominique Harrison Faculty/Co-PIs: Sylvia Hurtado Kevin Eagan Graduate Research Assistants: Tanya Figueroa Bryce Hughes Undergraduate Research Assistants: Paloma Martinez Robert Paul Papers and reports are available for download from project website: http: //heri. ucla. edu/nih Project e-mail: herinih@ucla. edu This study was made possible by the support of the National Institute of General Medical Sciences, NIH Grant Numbers 1 R 01 GMO 71968 -01 and R 01 GMO 71968 -05, the National Science Foundation, NSF Grant Number 0757076, and the American Recovery and Reinvestment Act of 2009 through the National Institute of General Medical Sciences, NIH Grant 1 RC 1 GM 09077601. This independent research and the views expressed here do not indicate endorsement by the sponsors.