NIH LOAN REPAYMENT PROGRAM EVALUATION NIH Loan Repayment

NIH LOAN REPAYMENT PROGRAM EVALUATION NIH Loan Repayment Programs Extramural LRPs FY 2003 -2009 Milton J. Hernández, Ph. D. Director Division of Loan Repayment OEP, OER, OD National Institutes of Health Bethesda, MD Laurel Haak, Ph. D. Chief Science Officer Discovery Logic, Inc. Rockville, MD

NIH LRP Evaluation Questions Ø Question 1: Who is applying to the NIH LRP? Ø Question 2: Does participation in the LRP affect career outcomes? Ø Question 3: How does the LRP compare to other NIH career development award programs? NIH Loan Repayment Programs 2

New and Renewal LRP Applications and Awards Trends Analysis, FY 03 -FY 07 2500 RENEWAL NEW Applications Number of LRP Applications and Awards 2028 2000 Applications Awards 1916 1846 1900 1500 1168 1147 1017 1000 Awards 2130 818 758 804 782 460 500 1284 1166 893 842 FY 06 FY 07 390 35 30 0 FY 03 FY 04 FY 05 FY 06 FY 07 FY 03 FY 04 FY 05 The renewal program has decreased the success rate for new applications but has not affected the number of new applications submitted. The success rate for new applications from FY 05 -FY 07 is 40% and for renewals is 70%. NIH Loan Repayment Programs 3

Gender Distribution of New LRP Applicants Female Applicants Male Applicants Success Rate (Female) Success Rate (Male) 70% 60% 50% 40% 30% 20% 10% 0% 0% FY 03 NIH Loan Repayment Programs FY 04 FY 05 FY 06 § The ratio of women to men LRP applicants is reversed from the ratio Success Rate Percent of New Extramural LRP Applications Trends Analysis, FY 03 -FY 07 in recent graduating MD and Ph. D classes. § Success rates are not different. FY 07 4

Gender Distribution of New LRP Applicants, by Degree Trends Analysis, FY 03 -FY 07 About the same number of men and p<0. 001 women applicants are MDs, but 2. 5 times more women applicants are Ph. Ds, and twice as many men applicants hold dual degrees. p<0. 001 Degree Type Example Degrees Physician Doctorate MD, DO Academic Doctorate Ph. D, Sc. D, Ed. D, DPH Physician Doctorate / Academic Doctorate MD/Ph. D, DO/Sc. D Professional Clinical Doctorate DDS, DMD, DVM, Pharm. D, Psy. D Allied Health Professional BSN, MPH, MSN NIH Loan Repayment Programs 5

Race/Ethnicity of New LRP Applicants Trends Analysis, FY 03 -FY 07 Percent of New LRP Applicants 14% 12% 10% 8% 6% 4% 2% 80% Success Rate, New LRP Applicants American Indian or Alaska Native Asian Black or African American Hispanic or Latino More Than One Race Native Hawaiian or Other Pacific Islander Asian Hispanic or Latino Black or African American White 70% 60% 50% 40% 30% 20% 10% 0% FY 03 FY 04 FY 05 FY 06 FY 07 § More Blacks and fewer Asians applied as compared to recent MD and Ph. D graduating classes. § Success rates did not differ from each other or from the overall success rate. NIH Loan Repayment Programs 6

Educational Debt, by Degree Type Trends Analysis, FY 03 -FY 07 Among LRP applicants, academic doctorates have a higher debt load than recent Ph. D graduates, and physician doctorates have a lower debt load than recent MD graduates. MD Ph. D LRP All* LRP All† No Debt n/a 15% n/a 50% <$40 K 10% 13% 38% 40% >$40 K 90% 73% 62% 10% $114 K $120 K $62 K ~$20 K Average Debt *Source: Survey of Graduates, American Association of Medical Colleges (2003 -2007). †Source: Survey of Earned Doctorates, National Science Foundation (2003). Prepared by NIH OER/OEP/DLR 7

Years Since Degree Trends Analysis, FY 03 -FY 07 1400 10+ Years (10%) 4 -6 Years (37%) 7 -9 Years (18%) LRP attracts early career researchers § 72% of new applicants apply within 6 years of receiving their degree § 90% of new applicants apply within 9 1200 Number of New LRP Applicants Up to 3 Years (35%) Physician Doctorate Professional Clinical Doctorate 1000 Academic Doctorate 800 Physician Doctorate / Academic Doctorate Allied Health Professional 600 400 200 0 <1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 15+ Years Since Degree years of receiving their degree Academic doctorates and physician doctorates apply at different times § 67% of academic doctorates apply within 3 years of their degree § 68% of physician doctorates apply 4 -6 years after receiving their degree NIH Loan Repayment Programs 8

NIH LRP Evaluation: Outcome Analysis Ø Question 1: Who is applying to the NIH LRP? Ø Question 2: Does participation in the LRP affect career outcomes? Ø Question 3: How does the LRP compare to other NIH career development award programs? NIH Loan Repayment Programs 9

Outcome Analysis Methodology Ø Outcome analysis: Regression Discontinuity Analysis. Ø We compared two nearly-identical applicant groups. LRP applicants with very similar priority scores were identified, about half of whom were funded and half of whom were not funded. Number of Applicants Priority Score “Bubble” Funded Not Funded Priority Score NIH Loan Repayment Programs 10

Identifying the Comparison Cohort LRP Applicant Cohort, FY 03 -FY 06 Funded Priority Score range, e. g. 190 -214 F 8 NF 9 NIMH, FY 03 Not Funded Identify bubble applicants for each IC and each fiscal year Outcome Analysis Subsequent publications and NIH grant participation F 11 NF 14 NICHD, FY 04 F 8 NF 8 F 9 NIAID, FY 05 NIDDK, FY 04 F 259 NF 254 Combine individual bubbles IC-FY Bubble 513 A 50 -50 mix of funded and not funded applicants with closely-matched priority scores NIH Loan Repayment Programs 11

IC-FY Bubble, Demographic Distribution Outcome Analysis, LRP Applicant Cohort, FY 03 -FY 06 % Funded IC-FY Bubble (n=259) % Not Funded IC-FY Bubble (n=254) % IC-FY Bubble (n=513) % All New Applicants, FY 03 -FY 07 (n=9820) Male 41% 36% 38% 44% Female 57% 62% 60% 53% Physician Doctorate 48% 46% 47% Professional Clinical Doctorate 2% 6% 4% 5% Academic Doctorate 44% 45% 42% Physician Doctorate / Academic Doctorate 5% 2% 3% 6% Allied Health Professional 1% 1% 1% <1% Up to 3 Years 52% 57% 54% 35% 4 -6 Years 34% 35% 37% 7 -10 Years 11% 6% 8% 10+ Years 2% 3% 3% 10% Parameter Gender Degree Type Years Since Degree Category The demographics of the funded and not funded IC-FY Bubble cohorts are indistinguishable, hence they can be used as comparison groups. There are some notable differences compared to the total new applicant group. 12

NIH Grant Participation: IC-FY Bubble Overall p=0. 0102 13

NIH Grant Participation: IC-FY Bubble, by Gender Outcome Analysis, LRP Applicant Cohort, FY 03 -FY 06 Male Percent of New LRP Applicants in the IC-FY Bubble 100% 90% Female Applied and Received Later RPG/K Award 474 p=0. 0 80% Applied and Received Later Non-RPG Award Applied, No RPG/K Award as of FY 07 70% Applied, No Non-RPG Award as of FY 07 60% Other Involvement in RPG/K Awards 50% Other Involvement in Non-RPG Awards 40% Had RPG Award Prior to LRP Award (≤ 5 years) 147 p=0. 0 30% Had Non-RPG Award Prior to LRP Award (≤ 5 years) 20% Matched to Key Personnel in RPG Type 5 Progress Reports, FY 05 -FY 07 10% None of the Roles Above 0% Funded (n = 105) Not Funded (n = 91) Funded (n = 148) Not Funded (n = 158) Women LRP awardees are less likely than men to have applied for an NIH award or to have a subsequent observable role in the NIH extramural program. NIH Loan Repayment Programs 14

NIH Grant Participation: IC-FY Bubble, by Degree Outcome Analysis, LRP Applicant Cohort, FY 03 -FY 06 Physician Doctorate Academic Doctorate 324 p=0. 0 106 p=0. 0 470 p=0. 0 Participation in the LRP program increased the likelihood that a physician doctorate applied for a subsequent NIH award and was retained in the NIH extramural workforce. It also increased the likelihood that an academic doctorate was awarded an NIH grant. NIH Loan Repayment Programs 15

Question 3: Comparison with Other NIH Programs Identification of Comparison Cohorts K Cohort LRP Cohort Funded Type I awardees, FY 03 -FY 06 Top 10% priority score, each FY/IC L awards only LRP+K Cohort K 01, K 08, K 23 Funded Type I awardees, FY 03 -FY 06 Top 10% priority score, each FY/IC K awards only LRP Top-10% LRP+K Top-10% n = 215 n = 218 Demographic parameters n = 96 Research accomplishments Publications Outcome analysis Subsequent NIH grant participation NIH Loan Repayment Programs Funded Type I awardees, FY 03 -FY 06 Top 10% priority score, each FY/IC L and K awards* *L award between FY 03 -FY 06 and a K 01/K 08/K 23 award at any time. Awardees were in the top 10% of priority scores for both award mechanisms. 16

NIH Grant Participation: K and L Programs Outcome Analysis, 0 -6 Years Since Degree, Top-10% Comparison 100% Applied and Received Later RPG Award 90% Applied and Received Later Non-RPG Award 03 =0. 00 p 80% p=0. 0176 Percent of Top-10% Awardee Cohort, 0 -6 Years Since Degree 70% Applied, No RPG Award as of FY 07 Applied, No Non-RPG Award as of FY 07 60% Other Involvement in RPG Awards 50% Other Involvement in Non-RPG Awards 40% Had RPG Award Prior to Award (≤ 5 years) 30% Had Non-RPG Award Prior to Award (≤ 5 years) 20% Matched to Key Personnel in RPG Type 5 Progress Reports, FY 05 -FY 07 p=0. 000 3 10% None of the Roles Above 0% LRP Top-10% (n = 157) K Top-10% (n = 101) LRP+K Top-10% (n = 39) Even after balancing for years since degree, the LRP recipients show less likelihood of applying for subsequent NIH funding than K or LRP+K awardees. LRP awardees are also less likely than K awardees to have a subsequent role on NIH awards. NIH Loan Repayment Programs 17