Using USMLE exam results to predict later intraining

Using USMLE exam results to predict later in-training exam performance among general surgery residents Charles Holden, MD- Associate Program Director Thomas Hartranft, MD – Program Director Darrell Spurlock, Jr. Ph. D – Director of Research Mount Carmel Health System General Surgery Residency Program cholden@phy. mchs. com

Problem Statement Ranking and selecting candidates for general surgery residency programs is an important but complex process Various factors are considered in ranking and selecting candidates but the relative weights assigned to these factors are sometimes anecdotal In an effort to examine the predictive nature of USMLE Step 1 and Step 2 scores on later ABSITE performance, a small study was conducted

Background Several factors are important to consider when ranking and selecting residents for general surgery residency programs Previous academic & professional performance Organizational fit Resident expectations of the program Program expectations of residents Congruence between career goals & training opportunities

Background (cont) Academic achievement is measured in many ways, but perhaps the most objective is by the use of standardized exams, like the USMLE Step 1 and Step 2 of the USMLE have been examined in past research for their relationship with in-training exam scores, but not in general surgery residency programs

Background (cont) Results from research in other specialties (mainly orthopaedics, obstetrics & gynecology, and internal medicine) has shown mixed results Step 1 is a good predictor in some cases, but Step 2 seems to be a better predictor – when studied (which it often is not) We examined Step 1 and Step 2 scores in relation to ABSITE scores during the PGY 1 -5 years

Methods Design: Retrospective, descriptive, correlational study of graduated resident records Participants: N = 34 residency completers from 1999 -2009 from two Midwestern community teaching hospital general surgery residency programs Correlation and regression analyses used to evaluate relationships between USMLE Step 1, Step 2, and ABSITE PGY 1 -5 year scores

Results 250 USMLE Step 1, Step 2, and ABSITE PGY 1 -5 Mean Scores, N = 34 with Standard Deviation Error Bars 217. 09 216. 91 200 150 100 69. 55 55. 59 50 46. 77 58. 63 56. 27 0 USMLE Step 1 USMLE Step 2 ABSITE Year 1 ABSITE Year 2 ABSITE Year 3 ABSITE Year 4 ABSITE Year 5

Results (cont) Intercorrelations among Exam Scores (N = 34) Variable 1 2 3 4 5 6 1. USMLE Step 1 - 2. USMLE Step 2 . 576** - 3. ABSITE Year 1 . 264 . 446* - 4. ABSITE Year 2 . 343 . 444* . 531** - 5. ABSITE Year 3 . 457** . 604** . 355* . 460** - 6. ABSITE Year 4 . 342 . 455* . 319 . 412* . 549** - 7. ABSITE Year 5 . 347 . 550** . 311 . 352* . 555** . 662* * significant at P <. 05 level ** significant at the P <. 01 level

Results (cont) Hierarchical regression analysis was run to examine the incremental predictive value of Step 1, beyond Step 2; no incremental improvement was noted Simple linear regression showed that USMLE Step 2 scores were predictive of all years of ABSITE performance, but most significantly for PGY 3 (R 2 =. 365, F = 11. 491, P =. 003) and 5 (R 2 =. 303, F = 8. 674, P =. 008)

Results (cont) Approximate 30 -36% Shared Variance Between USMLE Step 2 and ABSITE PGY 3/5 Scores USMLE Step 2 ABSITE PGY 3/5

Discussion USMLE Step 2 scores were better predictors of later ABSITE performance than were Step 1 scores Because of the limited range of scores for Step 2 and the small sample size, establishing a “cut-score” was not possible when considering only residents who were accepted into the residency program

Discussion (cont) Standardized test scores can be an important part of the selection criteria for choosing residents, but they must not be weighed too heavily as they account for only ~30 -35% of the later ABSITE performance Factors other than exam performance are important in the making of a good surgeon, and weren’t evaluated here, e. g. , professionalism, ethics, etc.

Discussion (cont) Future research is needed with larger samples to examine the relationship of other selection criteria on later in-training performance With the addition of the USMLE Step 2 “Clinical Skills” component, further evaluation of the usefulness of Step 2 is warranted as well Questions?