MAKING SENSE OF THE ACGME SURVEY OPDO MEETING

MAKING SENSE OF THE ACGME SURVEY OPDO MEETING 2015 STACEY T. GRAY, MD PROGRAM DIRECTOR, HARVARD MEDICAL SCHOOL

PANELISTS �STEVEN PLETCHER, MD Program Director, UCSF �VYVY YOUNG, MD Associate Program Director, University of Pittsburgh �IRAM AHMAD, MD PGY-7 Resident, University of Iowa

ACGME SURVEY What is it? � RESIDENT SURVEY Questions about the clinical and educational experience in the training program Questions about duty hours � FACULTY SURVEY Questions about the experience working in the training program Interactions with residents

ACGME SURVEY What is it? � Administered annually for residents, fellows and core faculty � NOTIFICATION Program is responsible for notifying and reminding trainees and faculty about the survey � TIMING Mid-January thru early June 5 week window to respond

ACGME SURVEY What is it? �REQUIRED RESPONSE RATE Resident/Fellow Survey ▪ 70% completion rate Faculty Survey ▪ 60% completion rate

SURVEY CATEGORIES

RESIDENT SURVEY �Duty Hours �Faculty �Evaluation Systems �Educational Content �Resources �Patient Safety/Teamwork �Overall Evaluation of Program

FACULTY SURVEY �Supervision and Teaching �Educational Content �Resources �Patient Safety �Teamwork �Overall Evaluation of Program

SURVEY QUESTIONS � Sample questions are not available Major categories remain the same Questions change each year – picked from a question bank � Not all survey takers will see the same questions

THE RESULTS

THE RESULTS

ACGME SURVEY Why is it important? � ANNUAL PROGRAM REVIEW DATA ENTRY ANNUAL ADS UPDATE ▪ Program Changes ▪ Program Attrition ▪ Scholarly Activity – faculty and residents ▪ Major changes ▪ Response to citations CASE LOGS RESIDENT SURVEY FACULTY SURVEY MILESTONES BOARD PASS RATE

HOW ARE THE RESULTS USED? “The ACGME’s Resident/Fellow and Faculty Surveys are an additional method used to monitor graduate medical clinical education and provide early warning of potential noncompliance with ACGME accreditation standards”

RED FLAGS �RESPONSE RATE Resident survey – 70% Faculty survey – 60%

RED FLAGS �RESIDENT SURVEY Duty hour violations Negative responses compared to national ORL program results in 3 major categories Negative responses compared to national ORL program results in 20% of the overall survey Negative response to final summary question

RED FLAGS �FACULTY SURVEY Negative responses compared to national ORL program results in 2 major categories Negative responses compared to national ORL program results in 20% of the overall survey

HOW DO YOU PREPARE YOUR PROGRAM?

COMPILED OTOLARYNGOLOGY DATA 2014 -2015

AGGREGATE DATA 2014 -2015 � 106 programs � Resident Response 1442/1506 96% response rate � Faculty Response 1406/1625 87% response rate

AREAS FOR IMPROVEMENT � Focus on areas of less than 95% compliance � Potential areas for improvement National level – how are we doing as a specialty? Program level – compare to national ORL standards

RESIDENT SURVEY: Duty Hours % Program Compliant % National Compliant 80 hours 96% 94% 1 day free in 7 99% 97% In-house call every 3 rd night 100% 99% Night float no more than 6 nights 100% 99% 8 hours between duty periods 96% 97% Continuous hours scheduled 98% 96%

RESIDENT SURVEY: Duty Hours % Program Compliant % National Compliant 80 hours 96% 94% 1 day free in 7 99% 97% In-house call every 3 rd night 100% 99% Night float no more than 6 nights 100% 99% 8 hours between duty periods 96% 97% Continuous hours scheduled 98% 96%

Duty Hours � Do you review duty hour rules with residents before the survey?

RESIDENT SURVEY: Faculty % Program Compliant % National Compliant Sufficient supervision 98% 92% Appropriate level of supervision 98% 96% Sufficient instruction 94% 85% Faculty and staff interested in residency education 94% 84% Faculty and staff create environment of inquiry 91% 78%

RESIDENT SURVEY: Faculty % Program Compliant % National Compliant Sufficient supervision 98% 92% Appropriate level of supervision 98% 96% Sufficient instruction 94% 85% Faculty and staff interested in residency education 94% 84% Faculty and staff create environment of inquiry 91% 78%

RESIDENT SURVEY: Faculty % Program Compliant % National Compliant Sufficient supervision 98% 92% Appropriate level of supervision 98% 96% Sufficient instruction 94% 85% Faculty and staff interested in residency education 94% 84% Faculty and staff create environment of inquiry 91% 78%

FACULTY SURVEY: Faculty supervision & teaching % Program Compliant % National Compliant Sufficient time to supervise residents/fellows 95% 94% Residents/fellows seek supervisory guidance 92% Interest of faculty and PD in education 98% 96% Rotation and educational assignment evaluation 99% Faculty performance evaluated 99% 98% Faculty satisfied with personal performance feedback 89% 86%

FACULTY SURVEY: Faculty supervision & teaching % Program Compliant % National Compliant Sufficient time to supervise residents/fellows 95% 94% Residents/fellows seek supervisory guidance 92% Interest of faculty and PD in education 98% 96% Rotation and educational assignment evaluation 99% Faculty performance evaluated 99% 98% Faculty satisfied with personal performance feedback 89% 86%

Faculty Issues � Resident Supervision vs. Independence Different perception � Learning Environment Faculty Development Mentorship Teaching techniques � Faculty Feedback How to provide it?

RESIDENT SURVEY: Evaluation % Program Compliant % National Compliant Able to access evaluations 99% Opportunity to evaluate faculty members 98% 96% Satisfied that evaluations of faculty are confidential 92% 84% Opportunity to evaluate program 99% 98% Satisfied that evaluations of program are confidential 91% 78% Satisfied that program uses evaluations to improve 88% 73% Satisfied with feedback after assignments 84% 70%

RESIDENT SURVEY: Evaluation % Program Compliant % National Compliant Able to access evaluations 99% Opportunity to evaluate faculty members 98% 96% Satisfied that evaluations of faculty are confidential 92% 84% Opportunity to evaluate program 99% 98% Satisfied that evaluations of program are confidential 91% 78% Satisfied that program uses evaluations to improve 88% 73% Satisfied with feedback after assignments 84% 70%

Evaluation issues � Resident Feedback Not just a semi-annual review In the moment � Confidentiality Reassurance for residents � Evaluations need to be meaningful Issues identified by residents should be addressed Improvements should be documented and reviewed with the residents

RESIDENT SURVEY: Educational content % Program Compliant % National Compliant Provided goals and objectives for assignments 96% 95% Instructed how to manage fatigue 97% 92% Satisfied with opportunities for scholarly activities 92% 76% Appropriate balance for education 89% 79% Education (not) compromised by service 81% 69% Supervisors delegate appropriately 100% 99% Provided data about practice habits 73% 68% See patients across variety of settings 98% 95%

RESIDENT SURVEY: Educational content % Program Compliant % National Compliant Provided goals and objectives for assignments 96% 95% Instructed how to manage fatigue 97% 92% Satisfied with opportunities for scholarly activities 92% 76% Appropriate balance for education 89% 79% Education (not) compromised by service 81% 69% Supervisors delegate appropriately 100% 99% Provided data about practice habits 73% 68% See patients across variety of settings 98% 95%

FACULTY SURVEY: Educational content % Program Compliant % National Compliant Worked on scholarly project with residents/fellows 86% 76% Residents/fellows see patients across a variety of settings 99% 98% Residents/fellows receive education to manage fatigue 99% Effectiveness of graduating residents/fellows 99% 97% Outcome achievement of graduating residents/fellows 99%

FACULTY SURVEY: Educational content % Program Compliant % National Compliant Worked on scholarly project with residents/fellows 86% 76% Residents/fellows see patients across a variety of settings 99% 98% Residents/fellows receive education to manage fatigue 99% Effectiveness of graduating residents/fellows 99% 97% Outcome achievement of graduating residents/fellows 99%

Educational Content Issues � Providing data about practice habits? Evaluations? Patient outcomes?

Educational Content Issues � Time for research and scholarship How protected is the time? How do you expose residents to all of the potential research projects and interested faculty? � Balance between education and service How protected is the time? How do you evaluate the quality of didactics? How involved are the residents in the didactics?

RESIDENT SURVEY: Resources % Program Compliant % National Compliant Access to reference materials 100% 99% Use EMR in hospital 98% 97% Use EMR in ambulatory setting 100% 97% EMR integrated across settings 89% 82% EMR effective 96% 94% Provided a way to transition care when fatigued 91% 80% Satisfied with process to deal with problems and concerns 91% 80% Education (not) compromised by other trainees 96% 91% Residents can raise concerns without fear 91% 80%

RESIDENT SURVEY: Resources % Program Compliant % National Compliant Access to reference materials 100% 99% Use EMR in hospital 98% 97% Use EMR in ambulatory setting 100% 97% EMR integrated across settings 89% 82% EMR effective 96% 94% Provided a way to transition care when fatigued 91% 80% Satisfied with process to deal with problems and concerns 91% 80% Education (not) compromised by other trainees 96% 91% Residents can raise concerns without fear 91% 80%

FACULTY SURVEY: Resources % Program Compliant % National Compliant Program provides a way for residents/fellows to transition care when fatigued 100% 99% Residents/fellows workload does not exceed capacity to do the work 98% 99% Satisfied with faculty development to supervise and educate residents/fellows 96% 95% Satisfied with process to deal with residents/fellows problems and concerns 95% 93% Prevent excessive reliance on residents/fellows to provide clinical service 98%

Resource Issues � Fatigue Management Care Transitions � Resident Concerns Confidential reporting Free of fear of intimidation or retaliation

RESIDENT SURVEY: Patient safety/teamwork % Program Compliant % National Compliant Tell patients of respective roles of faculty and residents 99% Culture reinforces patient safety responsibility 100% 99% Participated in quality improvement 88% 86% Information not lost during shift changes or patient transfers 98% 97% Work in interprofessional teams 99% 98% Effectively work in interprofessional teams 100% 99%

RESIDENT SURVEY: Patient safety/teamwork % Program Compliant % National Compliant Tell patients of respective roles of faculty and residents 99% Culture reinforces patient safety responsibility 100% 99% Participated in quality improvement 88% 86% Information not lost during shift changes or patient transfers 98% 97% Work in interprofessional teams 99% 98% Effectively work in interprofessional teams 100% 99%

RESIDENT SURVEY: Patient safety/teamwork % Program Compliant % National Compliant Tell patients of respective roles of faculty and residents 99% Culture reinforces patient safety responsibility 100% 99% Participated in quality improvement 88% 86% Information not lost during shift changes or patient transfers 98% 97% Work in interprofessional teams 99% 98% Effectively work in interprofessional teams 100% 99%

FACULTY SURVEY: Patient safety % Program Compliant % National Compliant Information not lost during shift changes or patient transfers 93% 89% Tell patients of respective roles of faculty and residents/fellows 88% 90% Culture reinforces patient safety responsibility 97% 96% Residents/fellows participate in quality improvement or patient safety activities 92% 90%

FACULTY SURVEY: Patient safety % Program Compliant % National Compliant Information not lost during shift changes or f patient transfers 93% 89% Tell patients of respective roles of faculty and residents/fellows 88% 90% Culture reinforces patient safety responsibility 97% 96% Residents/fellows participate in quality improvement or patient safety activities 92% 90%

FACULTY SURVEY: Teamwork % Program Compliant % National Compliant Residents/fellows communicate effectively when transferring clinical care 99% 98% Residents/fellows effectively work in interprofessional teams 100% 99% Program effective in teaching teamwork skills 100% 99%

Patient safety/teamwork � Patient Care Transitions � Quality Improvement � Interprofessional teams

OVERALL EXPERIENCE Resident perspective �Which of the following best summarizes your opinion of your residency program? Very negative Neutral Positive Very positive 0 20 40 60 80

OVERALL EXPERIENCE Faculty perspective �Which of the following best summarizes your opinion of your residency program? Very negative Neutral Positive Very positive 0 20 40 60 80 100

AS A SPECIALTY � Residents and Faculty generally have a positive view of our training programs � High compliance with most major categories � New questions highlight ACGME concerns with training

FINAL THOUGHTS

PREPARE YOUR PROGRAM �Remind the residents AND faculty before the survey Duty hours Confidentiality of your evaluation system Availability/accessibility of Goals & Objectives Resources & systems for communication/support ▪ Faculty mentors ▪ Chief or senior residents ▪ Institutional Sleep deprivation education Quality Improvement

USE YOUR RESULTS � Assess what the program is doing well � Assess opportunities for improvement � Topics for national discussions on training � Gain insight into what the ACGME is looking for in training programs