Competencies Milestones and EPA Why Cant They Make

Competencies, Milestones and EPA: Why Can't They Make Up Their Minds? !

Why Are We Doing This Anyway? § A brief review: why CBME § Where we are and where we need to go § What is needed to realize the “promise”

Why CBME

Individual Physician Readiness: Gaps Office-based Practice Competencies – Inter-Professional team skills – Clinical IT Meaningful Use skills – Population management skills – Reflective practice and CQI skills § § § Care Coordination Continuity of Care Leadership and management skills Systems thinking Procedural Skills Crosson Health Affairs 2011

Why CBME: System Needs Frenk J. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet. 2010 6

What Are The Outcomes? § A competent (at a minimum) practitioner aligned with: CMS Triple Aim

Where We Are and Where We Need to Go

“When You Come to a Fork in the Road, Yogi Berra Take It”

Einstein "When the Special Theory of Relativity began to germinate in me, I was visited by all sorts of nervous conflicts. . . I used to go away for weeks in a state of confusion. "

Shared Mental Model Challenge COMPETENCE MILESTONES/EPA * From Team. STEPPS/AHRQ

Milestones § By definition a milestone is a significant point in development. § Milestones should enable the trainee, program and the certification board to know an individuals trajectory of competency acquisition. § The milestones define the floor of competence but do not eliminate the need for aspirational goals!

Dreyfus & Dreyfus Development Model PGY 3 PGY 1 MS 3 MS 4 Proficient Competent Advanced Beginner Novice Time, Practice, Experience Dreyfus SE and Dreyfus HL. 1980 Carraccio CL et al. Acad Med 2008; 83: 761 -7 Expert/ Master

Internal NAS Medicine Milestones 14

Subcompetency Patient Total of. Care 22 Subcompetencies: has 5 Subcompetencies: • Break down ACGME competency domains into meaningful units • Gathers accurate data to define pt problem • Develop Separateand developmental achieve management ‘streams’ inplan which educators observe • Manages resident develop patients(or with failresponsibility to develop) competence • Procedures • Consultative Reflect prior work care of Milestones Writing Group • MK (2), SBP (4), PBLI (4), PROF (2), ICS (3) 15

Entrustable Professional Activities § EPAs represent the routine professional-life activities of physicians based on their specialty and subspecialty § The concept of “entrustable” means: – ‘‘a practitioner has demonstrated the necessary knowledge, skills and attitudes to be trusted to perform this activity [unsupervised]. ’’ 1 1 Ten Cate O, Scheele F. Competency-based postgraduate training: can we bridge the gap between theory and clinical practice? Acad Med. 2007; 82(6): 542– 547.

An Entrustable Professional Activity § § § Part of essential work for a qualified professional Requires specific knowledge, skill, attitude Acquired through training Leads to recognized output Observable and measureable, leading to a conclusion § Reflects the competencies expected § EPA’s together constitute the core of the profession ten Cate et al. Acad Med 2007

Competencies, Milestones and EPAs COMPETENCIES EPAs MILESTONES Characteristic Competencies Milestones EPAs Granularity Low Moderate to High Low to Moderate Synthetic/Integrated Moderate Low to Moderate High Practicality (application) Low Moderate High Conceptual High Low to Moderate

Norcini: Concerns about the Milestones Conceptual concerns • Reductionism • Mismatch with timebased curricula • No ongoing education Practical issues • Number of milestones • Assessment forms • Decisions • Training faculty From John Norcini; AMEE 2013. FAIMER

Reductionism & “Atomization” § Cees van der Vleuten (2008): – “The central assumption in the current model is that medical competence can be subdivided into separate measurable stable and generic traits. ” § John Norcini (2013): – Milestones are not enough: • Achieving all of the milestones does not imply overall competence – Integrated training experiences, assessments, and feedback are needed

Mismatch with Time-Based Curriculum § Milestones designed for an outcomes-based curriculum – Forced into a time/process system -based • Time is not an intervention, it is a context within which an intervention operates • Rotation system fragments training and is ill-suited to a developmental model – Accreditation standards and the nature of training need to evolve From J. Norcini; AMEE 2013; FAIMER

Medical Education Architecture 1 1 Holmboe E, Ginsburg S, Bernabeo E. The rotational approach to medical education: time to confront our assumptions. Med Educ. 2011; 45(1): 69 -80.

Lack of a System of Ongoing Education § Milestones work best in the context of ongoing professional development – Shift focus from competence at the ‘end’ of training to lifelong learning § Requires – An effective program of continuous professional development – Seamless transition between formal training and continuous professional development From J. Norcini; AMEE 2013; FAIMER

How Do We Assess “So Many” Milestones? A major concern of program directors § Assessment is about sampling from the domain of milestones – Not all milestones need to be assessed – Trainees do not need to be assessed on exactly the same • Milestones every time • Things as each other – There are sensible sampling plans that will produce reasonable results From J. Norcini; AMEE 2013; FAIMER

Why Doesn’t This Work!? Because there is NO HOLY GRAIL of Evaluation Forms: The “Form” is Only as Effective as the Individual Using It!

Which Assessment Forms Should We Use? § Forms make only a small difference in the quality of assessment – Faculty and the encounters make a big difference – Forms should comport with what is to be assessed • Forms do not need to be long • Wording and scaling have minimal impact – Shared item pools would be very useful From J. Norcini; AMEE 2013; FAIMER

What is Needed to Realize the Promise

The “System” Residents Assessments within Program: • Direct observations • Audit and performance data • Multi-source FB • Simulation • ITExam Institution and Program Accreditation: ACGME/RRC Program Aggregation Judgment and Synthesis: Committee NAS Milestones Board reporting No Aggregation Faculty, PDs. This and others is a Human Process Milestones and EPAs as Guiding Framework and Blueprint Certification: ABMS

Effective Assessment Process § Clarity on right outcomes linked to curriculum § Right combination and synthesis of assessment methods § Critical importance of shared understanding & mental models of competence – Competencies, milestones, entrustable professional activities (EPAs)

Assessment and Safe Patient Care Ø Importance of appropriate supervision Ø Entrustment Trainee performance* X Appropriate level of supervision** Must = Safe, effective patient-centered care * a function of level of competence in context **a function of attending competence in context

Assessing for the Desired Outcome Work-based assessment is mostly Does accomplished through (action) the observations of faculty, team Shows How members, peers and other co-workers (performance) Knows How (competence) Knows (knowledge) Performance in Practice/Multi-source feedback/ Direct Observation Standardized Patients/Simulation Diagnostic Reasoning using clinical vignettes or Chart Stimulated Recall Multiple choice Questions

Norcini: How do we train faculty? Faculty development § Methods of assessment will need to be based largely on observation • Faculty are the measurement instrument and they need training § Milestones make training easier but they are not a substitute for it • 2 -4 hour training exercise with periodic follow-up may be sufficient From J. Norcini; AMEE 2013; FAIMER

Assessment During Training: Components Clinical Competency Committee ·Periodic review – professional growth opportunities for all ·Early warning systems Advisor Trainee ·Review portfolio ·Reflect on contents ·Contribute to portfolio Structured Portfolio ·ITE (formative only) ·Monthly Evaluations ·Mini. CEX ·Medical record audit/QI project ·Clinical question log ·Multisource feedback ·Trainee contributions (personal portfolio) o Research project Program Summative Assessment Process Licensing and Certification · USLME ·American Boards of Medical Specialties Program Leaders ·Review portfolio periodically and systematically ·Develop early warning system ·Encourage reflection and self-assessment

Why Competency Committees?

The “System” Residents Assessments within Program: • Direct observations • Audit and performance data • Multi-source FB • Simulation • ITExam Institution and Program Accreditation: ACGME/RRC Program Aggregation Judgment and Synthesis: Committee NAS Milestones Board reporting No Aggregation Faculty, PDs and others Milestones and EPAs as Guiding Framework and Blueprint Certification: ABMS

The Wisdom of Crowds • The wisdom of many is often better than the wisdom of the few • To maximize the probability of good judgments: – Sample – “Independence” – Diversity are important…

Basic Committee Principles § Evidence-based versus verdict-based “jury” – Start and review all evidence before a decision • Do not start with a conclusion/decision – Confirmation bias § Be careful not to emphasize consensus over dissent – Minority opinions, even if “wrong”, still helpful – Be sure all voices are “heard” and watch carefully for negative effects of hierarchy

Committee Benefits § Develop group goals and shared mental models § “Real-time” faculty development § Key for dealing with difficult trainees § Share and calibrate strengths and weaknesses of multiple faculty assessments (“observations”) § Key “receptor site” for frameworks/milestones • Synthesis and integration of multiple assessments

“Wisdom of the Crowd” § Hemmer (2001) – Group conversations more likely to uncover deficiencies in professionalism among students § Schwind, Acad. Med. (2004) – • 18% of resident deficiencies requiring active remediation became apparent only via group discussion. • Average discussion 5 minutes/resident (range 1 – 30 minutes)

Narratives and Judgments § Pangaro (1999) – matching students to a “synthetic” descriptive framework (RIME) reliable and valid across multiple clerkships • Key component: good process with facilitation § Regehr (2012) – Faculty created narrative “profiles” (16 in all) found to produce consistent rankings of excellent, competent and problematic performance.

Quantitative and Qualitative Assessment Criterion Quantitative Qualitative approach Truth value Applicability Consistency Neutrality Internal validity Credibility External validity Transferability Reliability Dependability Objectivity Confirmability Used with Permission from Cees van der Vleuten (2010)

Strategy to establish trustworthiness Criteria Potential Assessment Strategy (sample) Credibility Prolonged engagement Training of examiners Triangulation Tailored volume of expert judgment based on certainty of information Peer examination Benchmarking examiners Member checking Incorporate learner view Structural coherence Scrutiny of committee inconsistencies Time sampling Judgment based on broad sample of data points Thick description Justify decisions Dependability Stepwise replication Use multiple assessors who have credibility Confirmability Audit Give learners the possibility to appeal to the assessment decision Transferability *With permission CPM van der Vleuten

Thank You and Questions