How Accurate is the ACGME Resident Survey Comparison

How Accurate is the ACGME Resident Survey? Comparison Between ACGME and In-House GME Survey Bridget N. Fahy 1, S. Rob Todd 1, Judy L. Paukert 2, Melanie L. Johnson 1, Barbara L. Bass 1 1 Department of Surgery, The Methodist Hospital, Weill Cornell Medical College 2 The Methodist Hospital Education Institute Association of Program Directors in Surgery 20 April 2010

Accuracy of ACGME Survey Disclosure • No financial conflict of interest

Accuracy of ACGME Survey Background • Purpose of annual Accreditation Council for Graduate Medical Education (ACGME) Resident/ Fellow Survey: – monitor graduate medical education – provide early warning of potential non-compliance • Not administered in conjunction with program site visit-- results are used at the time of the visit • Used to prompt early site visits for programs with significant areas of non-compliance

Accuracy of ACGME Survey Background • 15/32 questions allow only “yes/no” responses • Wording of some questions is confusing resident misunderstanding of the question* • Graduate Medical Education (GME) Department administers Annual Survey of Residents – similar content to ACGME survey – responses given using a 5 -point Likert scale – opportunities for narrative responses for each question are solicited • Differences in the results obtained from the two surveys identified *Holt et al. Acad Med; 85, 2010.

How Accurate is the ACGME Resident Survey?

Accuracy of ACGME Survey Purpose of Study • Examine systematically the results of 15 questions on the two surveys that assessed similar information – identify areas of significant disparity

Accuracy of ACGME Survey Subjects • 26 general surgery residents – 4 categorical residents at each PGY level – 2 preliminary PGY 2 residents – 4 preliminary PGY 1 residents • Tertiary referral academic training center

Accuracy of ACGME Survey Instruments • ACGME – www. acgme. org/ac. Website/resident_survey – 32 -question survey administered annually – Response options: “yes/no” = 15, 3 -point scale = 4, 5 -point Likert scale = 9 – Residents assured of anonymity

Accuracy of ACGME Survey Instruments • GME – 53 -question survey administered annually – Administered to residents from all programs – Response options: majority given on 5 -point Likert scale + option for narrative responses – Resident assured of anonymity

Accuracy of ACGME Survey Common Survey Items Faculty support 1. Do the faculty spend sufficient time teaching residents/fellow in your program? 2. Do the faculty spend sufficient time supervising residents/fellows in your program? 3. Do your faculty members regularly participate in organized clinical discussions, rounds, journal clubs, conferences? 4. Do you have the opportunity to confidentially evaluate your faculty, in writing, electronically, at least once a year? Rotation / program evaluation 5. Do you receive written or electronic feedback on your performance for each rotation? 6. Do you have the opportunity to confidentially evaluate your program, in writing, electronically, at least once a year? 7. Has your program provided you access to, either by hard copy or electronically, written goals and objectives for the program overall and for each rotation and major assignment? 8. Have residents/fellows had the opportunity to assess the program for the purposes of program improvement? Working / educational environment 9. Does your program and/or institution have a system through which you are able to raise and resolve issues without fear of intimidation or retaliation? 10. How often are you able to access, either in print or electronic format, the specialty specific and other reference materials that you need? 11. To what extent to trainees who are not part of your program (such as residents from other specialties, subspecialty fellows, Ph. D. students and nurse practitioners) interfere, in a negative way, with your education? 12. Do your rotations and other major assignments emphasize clinical education over any other concerns, such as fulfilling service obligations? 13. Does your program offer you the opportunity to participate in research or scholarly activities? Duty hours 14. Have you had sufficient education (from your program, your hospital(s), your institution, or your faculty) to recognize and counteract the signs of fatigue and sleep deprivation? 15. Have you met the ACGME duty hour requirements?

Accuracy of ACGME Survey Subject Demographics Age Yrs. (median) Male: Female Postgraduate year 1 2 3 4 5 Categorical: Preliminary GME Survey N=26 30 18: 8 8 6 4 4 4 20: 6

Accuracy of ACGME Survey Faculty Support Yes on ACGME (%) 12/25 (48) Yes on GME (%) 22/26 (85) Faculty spend sufficient time teaching Faculty spend sufficient time 24/25 (96) 25/26 (96) supervising Faculty participation in didactics 25/25 (100) 25/26 (96) Faculty provide feedback 22/25 (88) 23/26 (88) P-Value 0. 008 NS NS NS

Accuracy of ACGME Survey Rotation/Program Evaluation Yes on ACGME (%) Opportunity to evaluate program 25/25 (100) Opportunity to evaluate faculty 25/25 (100) Receive goals/objectives for 25/25 (100) rotation and program Opportunity to assess program 23/25 (92) for improvement Yes on GME P-Value (%) 26/26 (100) NS 25/26 (96) NS

Accuracy of ACGME Survey Work / Educational Environment Yes on ACGME (%) Yes on GME (%) P-Value Raise issues in program/institution Ability to access reference materials Interference by other trainees 23/25 (92) 25/26 (96) NS 22/25 (88) 25/26 (96) NS 6/25 (24) 21/26 (81) 0. 0005 Rotations emphasize education over other concerns 7/25 (28) 19/26 (73) 0. 001 Opportunity for research 25/25 (100) 25/26 (96) NS

Accuracy of ACGME Survey Duty Hours Compliance Education regarding sleep deprivation and fatigue Compliance with ACGME duty hour requirements Yes on P-Value ACGME (%) 22/25 (88) 23/26 (88) NS * 24/26 (92) 23/25 (92) N/A 1 day in 7 free from all educational and clinical responsibilities, averaged over a four-week period 24/25 (96) N/A 10 -hour time period provided between all daily duty periods and after in-house call In-house call must occur no more frequently than every third night, averaged over a four-week period 22/25 (88) N/A 25/25 (100) N/A Continuous on-site duty must not exceed 24 consecutive hours 25/25 (100) N/A No new patients may be accepted after 24 hours of continuous duty 23/25 (92) N/A 80 hours/week, averaged over a four week period At-home call must not be so frequent as to preclude rest and reasonable personal time

Accuracy of ACGME Survey Summary • Differences in responses found for 3 items: – Faculty time spent teaching – Interference of other trainees – Rotation emphasis of education over other concerns • Responses on ACGME survey were less favorable than on GME survey

Accuracy of ACGME Survey Conclusions • Two factors may contribute to inaccurate responses on ACGME survey: – Limited range of available responses – Confusing wording of questions • Inaccurate responses elicited on the ACGME survey may result in erroneously elevated rates of noncompliance • Further validation of the current ACGME survey is warranted