Accreditation Council for Graduate Medical Education Next Accreditation

Accreditation Council for Graduate Medical Education Next Accreditation System II. Annual Program Review Pamela L. Derstine, Ph. D, MHPE RRC Executive Director SNS Program Director Workshop June 8, 2013

Topics • Annual Data Submission • Screening Annual Data • Review of Annual Data © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Annual RRC Program Review • Annual ADS Update Ø Program Changes – Structure and resources Ø Program Attrition – PD / core faculty / residents Ø Scholarly Activity – Faculty and residents • Board Pass Rate – 7 year rolling average • Clinical Experience – Case logs • Resident Survey – Common and specialty elements • Faculty Survey • Semi-Annual Resident Evaluation and Feedback Ø Milestones • Omission of data © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Annual RRC Program Review • Annual ADS Update Ø Program Changes – Structure and resources v Participating Site information (e. g. , PLA on file, distance from primary site, rationale, rotation months) v Current Block Schedule: programs must follow instructions and use the block schedule template provided https: //www. acgme. org/ads/Content/Downloads/Block. Di agram. Instructions. pdf v Sponsoring institution information (e. g. , DIO, participating sites, other sponsored programs) © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Annual RRC Program Review • Annual ADS Update Ø Program Changes – Structure and resources v Major changes v Response to current citations (if resolved on annual review, will be removed and stored in program history) v Duty Hour, Patient Safety, Learning Environment (17 items) v Overall evaluation methods (6 items) © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Annual RRC Program Review • Annual ADS Update ØProgram Attrition – PD / core faculty / residents v Program director change history since 2000 v Program director CV v Program faculty basic information v Resident basic information v Change requests © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Annual RRC Program Review • Annual ADS Update Ø Scholarly Activity – Faculty and residents v CV used for program director v Scholarly activity templates used for faculty and residents v Scholarly activity is reported for the most recently completed academic year (12 month period, not a 5 year period) for all but the PD © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Program Director CV same information requested but entered online © 2013 Accreditation Council for Graduate Medical Education (ACGME)

© 2013 Accreditation Council for Graduate Medical Education (ACGME) Scholarly Activity Template

Faculty Scholarly Activity Enter Pub Med ID #’s © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Faculty Scholarly Activity Enter a number © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Faculty Scholarly Activity Enter a number © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Faculty Scholarly Activity Enter a number © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Faculty Scholarly Activity Enter a number © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Faculty Scholarly Activity Answer Yes or No © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Faculty Scholarly Activity Answer Yes or No © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Faculty Scholarly Activity © 2013 Accreditation Council for Graduate Medical Education (ACGME)

© 2013 Accreditation Council for Graduate Medical Education (ACGME) Resident Scholarly Activity

© 2013 Accreditation Council for Graduate Medical Education (ACGME) Resident Scholarly Activity

© 2013 Accreditation Council for Graduate Medical Education (ACGME) Resident Scholarly Activity

Annual RRC Program Review • Board Pass Rate: 5 year ADS self-report © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Annual RRC Program Review • Board Pass Rate – 7 year rolling average Ø ABNS has provided pass rates to the ACGME electronically for each year beginning with 2003 through 2012 for parts 1 and 2 for all programs Ø ABNS will provide an annual electronic update to the ACGME, beginning with the 2013 exam results Ø Annual ABNS reports to ACGME may preclude the need for programs to provide this information © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Annual RRC Program Review • Clinical Experience – Case logs Ø 2012 -2013 program review - minimum numbers for defined case categories reviewed and feedback provided (no citations based on min. numbers) Ø 2013 -2014; 2014 -2015 program reviews - graduates expected to comply with minimum number requirements for all categories except critical care (DC 20 -28) and endovascular (DC 3 b) - level-specific reports will be reviewed to monitor progress towards compliance Ø 2015 -2016 program review and beyond - graduates expected to comply with all categories © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Annual RRC Program Review • Resident Survey – Common and specialty elements v 7 survey question domains: duty hours; faculty; evaluation; educational content; resources; patient safety; teamwork v 70% response rate required v Aggregated non-compliant survey responses for each domain are reviewed; thresholds for noncompliance v Exception: programs with DHE and non-compliant responses for intimidation and service-overeducation will be cited © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Annual RRC Program Review • Resident Survey – Common and specialty elements v Programs with DHE and non-compliant responses to 88 -hour question - first time non-compliance: warning - second time non-compliance: action plan required - third time non-compliance: DHE canceled; program may not reapply for DHE © 2013 Accreditation Council for Graduate Medical Education (ACGME)

© 2013 Accreditation Council for Graduate Medical Education (ACGME) Annual RRC Program Review • Faculty Survey Ø 5 question domains: - supervision and teaching - educational content - resources - patient safety - teamwork Ø Intended to mirror most resident survey questions and provide opportunity to compare responses by question domain Ø First survey completed: spring 2013 Ø First RRC review of faculty survey data: spring 2014 Ø Compliance metrics not yet developed

Annual RRC Program Review • Milestones v First milestone evaluation period: July – December 2013 - Residents evaluated as usual by the program (competency-based, multiple evaluators) v First milestone reports to ACGME: Nov/Dec 2013 - Collected evaluations reviewed by the CCC - CCC determines milestone level for each resident for each milestone - Milestone reporting will be done through a link in ADS (not yet available) v Second milestone reports to ACGME: May/June 2014 v First RRC review of milestone data: spring 2015 © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Annual RRC Program Review • Annual ADS Update Ø Omission of data v If any required annual ADS update information is missing, the program will be flagged by the NAS data system v Data omission could result in an altered accreditation status © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Screening Annual Data • Thresholds for core annual data established to quickly identify programs with serious problems Ø Aids in prioritizing Review Committee work Ø Facilitates more timely identification of problems and provision of feedback to programs to correct problems Ø Goal – ensure ongoing quality resident education © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Screening for Program Changes • Three or more changes (e. g. , participating site, resident complement, block diagram, major structural change, sponsoring institution change, GMEC report of structure change) © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Screening for Program Attrition • Three or more program attrition reports (e. g. , program director change, core faculty decrease, chair change. DIO change, CEO change, resident attrition) © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Screening for Scholarly Activity • Faculty Ø < 100 % faculty with at least two points per year Ø Activity in each category counts as one point except Ø Each peer-reviewed article and grant counts as one point • Residents Ø < 80% of residents with at least one point per year Ø Activity in each category counts as one point except Ø Each peer-reviewed article counts as one point © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Screening for Board Scores • < 85% first time takers (for credit) pass primary exam averaged over last three years • < 80% first time takers pass oral exam (minimum of 10 residents) © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Screening for Case Logs • Each graduating resident: Ø below minimum number in any defined case category Ø reporting < 500 total cases (not limited to defined case categories) © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Screening for Surveys • Resident Ø Non-compliance in duty hours area Ø Non-compliance in three or more areas • Faculty Ø Not yet established © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Screening for Milestones • Not yet established © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Screening for Omissions • One or more core data elements omitted © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Policies and Procedures: 7/1/2013 http: //www. acgmenas. org/assets/pdf/Final. Master. NASPolicy. Pro cedures. pdf • Policy 17. 61 Review of Annual Data Ø Data RRC may use Ø RRC actions following annual data review • Policy 18. 00 Accreditation Actions © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Review of Annual Data • Continuous data collection/review Ø ADS annual update Ø Resident survey Ø Faculty survey Ø Milestone data Ø Certification examination performance Ø Case log data Ø CLER information Ø Hospital accreditation data Ø Faculty and resident scholarly activity / productivity Ø Other © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Review of Annual Data • Other data (episodic) Ø ACGME complaints (section 23. 10) Ø Verified public information Ø Historical accreditation decisions / citations Ø Institutional quality and safety metrics Ø Other © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Accreditation Decisions • NO proposed adverse actions • Potential Actions (if currently accredited): Ø progress report Ø focused site visit Ø full site visit Ø continued accreditation Ø accreditation with warning (may not request permanent complement increase or DHE) © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Accreditation Decisions • Potential Actions (if currently accredited): Ø probation (must be preceded by a site visit; status limit of 2 years; may not request permanent complement increase or DHE) Ø complement reduction Ø withdrawal of accreditation © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Accreditation Decisions • Potential Actions (if currently accredited): Ø Recognize exemplary performance; innovations Ø Identify opportunities for program improvement Ø Identify concerning trends Ø Issue new citations Ø Continue previous citations Ø Acknowledge correction of previous citations © 2013 Accreditation Council for Graduate Medical Education (ACGME)

Accreditation Decisions • Accreditation decisions made in the past 12 months will be updated effective July 1, 2013 Ø Short-cycled programs (1 -2 year cycle lengths) will change from continued accreditation to accreditation with warning Ø Accreditation with warning is NOT an adverse action; residents and applicants do NOT need to be informed © 2013 Accreditation Council for Graduate Medical Education (ACGME)