The Next Accreditation System and ACGME Milestones Learning

The Next Accreditation System and ACGME Milestones: Learning to Use Competency Assessment to Focus on Outcomes Wendy Biggs, MD Sam Cullison, MD Julie Dostal, MD Larry Mauksch, MEd Judy Pauwels, MD Stephen Wilson, MD

Introductions Wendy Biggs, MD Sam Cullison, MD Julie Dostal, MD Larry Mauksch, MEd Judy Pauwels, MD Stephen Wilson, MD

How did we get here? Sam Cullison, MD

NAS & Milestones Biggest Change in GME Since ACGME Competencies

• • • Continuous Accreditation Outcome measures (Milestones) Institution Accountability (CLER) 6 ACGME Competencies Patient Care Medical Knowledge Problem-Based Learning and Improvement Interpersonal and Communication Skills Professionalism System-based Practice 2005 ACGME Strategic Plan – Innovation Outcomes Efficiency Communication 1 20 4 2 1 20 5 0 20 2 5 9 19 Next Accreditation System Described 0 00 Competency based accreditation (NEJM 2/12)

New Accreditation System N-A-S • • • New Rules! Starts July 2014 ADS updates annually Resident/Faculty Surveys annually Board Pass Rates Biannual (every 6 months)Resident Evaluation Clinical Competency Committee (CCC) 10 year Self-Study Encourage Innovation

New Accreditation System N-A-S • • No PIF No Internal Reviews No cycle lengths Focus on attrition, scholarly activity (residents and faculty)& program structure • Annual GMEC reports • Greater GMEC oversight

ACGME ACCREDITATION Continuous RC Oversight 10 yr selfstudy Clinical Learning Environ Review & Visits Site Visits prn (prog or inst) 10 yr selfstudy visit

Milestones-What up with that?

Milestones • About progressive responsibility • Created by each specialty • Narratives about the “DOES” in Miller’s Pyramid • Less range restriction • Based on the 6 Competencies

Milestones • Observable steps on continuum to competency • Milestones for reporting, not evaluation • Existing tools used • Trained observers • Multiple observations • Clinical competency committee assesses progress

Shifting the Culture of Assessment From time based to competency From passing rotations to demonstrating performance Julie Dostal MD

What brings you here today? • What do you worry about? • What are you excited about? • What has you scratching your head?

Miller’s Assessment Pyramid Miller, G. E. The assessment of clinical skills/competence/performance. 1990 Acad Med 65(9 Suppl): S 63 -7. Adapted from Holmboe and Hawkins. Evaluation of Clinical Competence, Mosby 2008 Impact on the patient Does • Faculty observation (with real patients) Shows how Knows • Standardized patients • Matching or critical response questioning • Multiple choice questions 14

Competency Based Education Adapted from Miller, G. E. The assessment of clinical skills/competence/performance. 1990 Acad Med 65(9 Suppl): S 63 -7. Impact on Training Does • Implements and evaluates CBE in residency program Shows how Knows • Demonstrates use of direct observation feedback to learner • Can describe and generate competency assessment tools/skills • Can define formative and summative assessment 15

Building a trustworthy system of assessment • Transparency – I know who is assessing me and how • Direct Observation – Observable behaviors as often as possible • Direct Communication and Feedback – “Assessor to resident” • Clarity of Summative Assessment process – Resident involved in the summative assessment conversations about them as individuals – Daily observations feed into Summative

Pieces to the Puzzle • Faculty – Become “resident-centered educators” – Develop and practice skills in giving formative feedback – Openness to receiving feedback • Residents – Reduce reluctance to reveal gaps – Give faculty feedback • Both – Overcome Evaluation PTSD

Culture Shift Assume that your program is capable of developing competence in every resident, and the purpose of assessment is simply to document attainment as it occurs.

Strategies for Shifting to Competency Based Assessment Faculty develop principles of assessment for program and define program goals Develop a shared lexicon Increase use of direct observation Structure resident reviews to categorize comments by ACGME competencies and Milestones Design closed circuit observations by two faculty at the same time to promote shared experiences

What is your culture of assessment How valued is assessment? How reliable is assessment? Who are the change agents? Where are you in balancing formative and summative assessment?

Write-Pair-Share • Describe the current state of Assessment in your residency • Describe a target future state; • First next step to move your assessment process to the target state • Take 5 minutes and write your answers • Groups of 3 sharing themes---10 min • Pair up with 3 other groups to share group themes---10 min • Harvest themes with large group – 10 min

http: //stfm. org/rctoolkit/ 22

Trying to make small changes without fundamental changes in thinking and structure is unlikely to be successful Remember the Butterfly Effect

Best Exotic Marigold Hotel Initially you’re overwhelmed. But gradually you realize it’s like a wave. Resist, and you’ll be knocked over. Dive into it, and you’ll swim out the other side. Everything will be all right in the end…if it’s not all right, then It’s not yet the end!

Formative and Summative Assessment: Direct observation tools and skills Implementing competency assessment

Direct Observation: Methods, Faculty Time Demand Educational Pros Educational Cons Direct observation in the room High Loss of income or other activity Clear view Can teach on the fly Trainee initially self conscious Risk of upstaging relationship Video review High Loss of income or other activity Trainee self observes, strong educational options Delayed practice Requires technical expertise and expense Closed circuit Moderate Some income loss or other activity Fast practice Faculty development Distraction, time limitation, Reliability? Very Low ++Observations ++Reflection ++Practice Less depth, versatility, Reliability? Low Role modeling Observer self Faculty growth Passive trainee role Peer You (faculty) are observed

Data Collection Form design • There is NO PERFECT FORM! • Straight-forward, done frequently Need enough formative information for summative assessment

Select tools that: Are feasible to use Cover multiple competencies Educate the user Have broad faculty support Have existing psychometric properties, or Are qualitative and used by several faculty members frequently

Data Consolidation Who oversees and insures data collection Determine where data is collected • Files • Online Make data collection transparent Give residents access Data should be used regularly to promote growth (formative) rather than for summative purposes

‘Documented observations with feedback is the sine qua non of competency based education’ • M. Donoff, MD University of Alberta • Donoff MG. Field notes: assisting achievement and documenting competence. Can Fam Physician. Dec 2009; 55(12): 1260 -1262, e 1100 -1262. 30

THERE IS NO MORE DIFFICULT ART TO ACQUIRE THAN THE ART OF OBSERVATION, AND FOR SOME…. IT IS QUITE AS DIFFICULT TO RECORD AN OBSERVATION IN BRIEF AND PLAIN LANGUAGE. - SIR WILLIAM OSLER

Exercise • Field Notes

INSTUCTIONS Video 1 • Form dyads • One person plays the preceptor and the other the resident • Preceptor provides feedback (first person) in four minutes • Dyads debrief – Indentify major theme • At each table: Share theme between dyads to determine one theme and preceptor to speak for the table

VIDEO #1

Evaluation PTSD Prior noxious experience may be an unconscious barrier to providing helpful and supportive feedback Delayed, negative, summative feedback with no prior formative assessment Shame and belittlement that you experienced or observed • Roots of “pimping” Share your experiences and their impact

INSTUCTIONS Video 2 • Form dyads and reverse roles from role play #1 • Preceptor provides feedback (first person) in four minutes • Dyads debrief – Indentify major theme • At each table: Share theme between dyads to determine one theme and preceptor to speak for the table

VIDEO #2

Examine Milestone C-2 and determine if the placement of this resident has changed

Notes on Feedback Does the trainee have a goal? Balance emphasis on strengths with growth areas • Increase awareness of strengths Feedback: facilitative, not directive • Adopt a humble, non expert stance Lead with curiosity • Make an observation • Share your wonder about the thoughts or feelings • Ask if he/she want a suggestion

Giving Feedback (when requested): it… Timely About changeable behavior? Kind Not judgmental Honest Specific Provisional and curious

Creating SCRIPT Scratch your head • What is the problem? • What is important to communicate? • How would I say it? CReate a verbal example out loud Interpret your creation (Explain what you did) Practice--Have the trainee try Troubleshoot • Consider pitfalls and refinements

A Dive into the Milestones

Competency Medical Knowledge (MK) (early draft) Subcompetency MK 2. Apply critical thinking skills in patient care Beginning Resident Demonstrates basic linear analytic decision making capabilities Demonstrates the capacity to correctly interpret basic clinical tests and images Junior Resident Demonstrates synthesis of multiple information resources to make routine clinical decisions Begins to integrate social and behavioral sciences with biomedical knowledge in patient care Senior Resident Graduating Resident Demonstrates sound Integrates and clinical judgment in synthesizes non-routine situations knowledge to make decisions in complex Analyzes and clinical situations prioritizes information to make clinical Prioritizes care decisions that are based on urgency, individualized for importance and each patient prognosis. Moves from Milestone individual thinking to family and population considerations Personal Physician Integrates in-depth medical and personal knowledge of patient and family and community to decide, develop and implement treatment plans Can define the participants necessary to address important health problems for both individuals and communities

Small Group Assignment 30 Minutes • Break Into Small groups (4 -6 people) – One Sub-competency from the Milestone Document/group • Where/When do your residents show you they can do these milestones? (current learning activities) • How do you currently document their performance of this activity? (current assessment tools) • Be Prepared to share with Large group Milestones with less than desirable or No assessment being done now (gaps)

Faculty Development Stephen A. Wilson, MD, MPH UPMC St Margaret wilsons 2@upmc. edu

Barriers to Training & Implementation 4 Lacks and a Feeling Lack of Training Lack of Leadership Lack of Time Lack of Language Feeling Overwhelmed (with data collection)

The purpose of medical education is to improve patient care

Whatever we practice with good coaching we tend to improve

“Whatever we measure we tend to improve” David Leach, MD Executive Director, ACGME

http: //www. salesblog. com/want-to-startmaking-an-attitude-change/ http: //qstream. com/ Faculty Development http: //www. empowernetwork. co m/breakthroughideas/skills-youneed-on-your-resume/

Basic

y nt t l u e c Fa opm l e v e D • Common Langauge & Expectations – Competencies & Milestones For ALL • Milestones ≠ Evaluation tool = Reporting tool

http: //www. zazzle. com/worlds_best_doctor_mug-168787518832411439 http: //stacadvising. lmc. gatech. edu/? tag=house-m-d Why is CBA important?

Competency Assessment Embraces the notion that capabilities of residents have to be measured, rather than assuming the attainment of competence is an inherent part of the training process.

Competency Assessment Well functioning residency program is a - coherent - purposeful - integrated design and delivery of managed learning that enables residents to become competent and capable clinicians

Characteristics of Good Assessment • Measures actual performances • Identifies areas for improvement • Satisfies reasonable request for accountability • Is practical • Is done over time to discern growth David Leach, MD Executive Director, ACGME

http: //www. 123 rf. com/photo_13638439_attitudeconcept-in-word-tag-cloud-on-white-background. html Faculty Development – Attitude

Faculty Development – Attitude http: //brooklynjamesblogspot. com/2013/02/attitude-its-everything. html

Faculty Development – Attitude Not everything that can be measured counts and t p e c c not everything that a ’t s Le counts can be measured

Useful Concepts about Measurements • Life is not measurable • We use models to understand life • All models are limited, some are useful • Measurements are applied to models • Both measurements and models must be constantly reassessed • Need structured dialogue about Dr. David Leach, Executive Director, ACGME measurements

Faculty Development – Skill e c i t c a r P http: //www. 123 rf. com/photo_14462053_tip-acronym-for-theory-into-practicewritten-on-a-blackboard-background-with-sticky-notes. html

n o i t a bserv o t c e Dir http: //www. sciencedirect. com/science/article/pii/S 0140673600042215

• • High level Formative Informative & revealing Fly on the wall Timely Oral Written http: //www. ehow. com/info_8391456_t ypes-observation-research-study. html • • • Direct Observation to Assess Competency

http: //sat. collegeboard. org/practice/sat-practice-questions-writing/sentence-errors

http: //www. coderfriendly. com/2009/05/22/when-do-we-reach-the-expert-stage/

http: //www. learninggeneralist. com/2009/08/using-dreyfus-model-to-engage-people-in. html

http: //blog. allstate. com/rethinking-the-minimum-driving-age/

http: //elegantwordart 2. blogspot. com/ 2008/08/practice-makes-perfect. html http: //blog. ellusionist. com/practice-to-perfection/ http: //izquotes. com/quote/114103 http: //dawnmauricio. com/2013/02/19/practice-makes-permanent/

http: //www. learninggeneralist. com/2009/08/using-dreyfus-model-to-engage-people-in. html

Family Medicine Milestones Five Developmental Levels • Personal physician • Graduating resident • Senior resident • Junior resident • Beginning resident

Formative vs. Summative Roles Formative Coach (preceptor) Summative Evaluator Uses prior data to influence Using prior data to teaching determine promotion Regularly engages the learner in goal setting Measures learner’s performance against external goals Integrates learner preferences into educational design Prescribes experiences to the learner

Feedback Formative & Summative (Evaluation) • Formative Feedback – – • Emphasis on Direct Observation Rotation-Based Patient Encounter Based Multiple times during each year of training Summative Evaluation – – Annually End of Training

Substance vs. Form Substance is enduring; form is ephemeral. Preserve substance; modify form. Know the difference. -- Dee Hock

Implementation Approach • CBE emphasis will primarily be: üAssessment üFeedback üEvaluation • Honing and refining teaching skills • NOT on instituting new programs • Work in progress

Faculty Development • • This Future? Video review practice – Direct observation practice – Competency review committee – Plans for implementation – http: //www. stfm. org/rctoolkit/ – http: //www. virtualtoolbox. ca/

Clinical Competency Committee Judith Pauwels MD University of Washington jpauwels@uw. edu

CCC: what’s the intent? • Clinical Competency Committee (Resident Evaluation Committee) • Clinical Competence Committee triangulates progress of each resident • Essential for valid and reliable clinical evaluation system

CCC: what’s the intent? • Trained observers – Common understanding of the expectations – Sensitive “eye” to key elements – Consistent evaluation of levels of performance – Requires certain number of observations – Interpreter/synthesizer experts

CCC: what’s the intent? • Offers the insight and perspectives of experienced faculty in assessing resident progress • Identify and assist residents failing to progress on normal pathway • Stretch goals for residents already exceling

CCC: Who should be on it? • Decision for PD • Consider: – Representation from each major site – Dedication to education – Can NOT include residents • Size

CCC: what information is gathered? OSCE Nursing & Staff Evals Family & Patient End of Rotation Evals ITE Peer Evals Sim Lab Self Eval Unsolicited Comments Student Evals Clin. Comp. Committee Operative Performance Milestone Assessment Clinic Workplace

CCC: How does it work? • • Understand the milestones & their use Review all evaluations for each resident “Consider the source(s)” For each resident, decide for each milestone the narrative that best fits that resident

CCC: How does it work? • Review all assessments for each resident every 3 -6 months to determine current progress on the milestones by group consensus • Report on each resident (? de-identified) to the ACGME every 6 months

CCC: How does it work? • Provide feedback to each resident regarding their progress on the Milestones • Individualized educational plan (IEP) for each resident

CCC: program learnings • Faculty Development: feedback to evaluators • Program Development: where are we not meeting needs for our residents?

CCC: questions • Role of resident advisors/mentors • Management of extensive information • Reporting needs

CCC: logistics • Start July 2014 • Strongly consider pilot testing before then – Gather performance data – Test 2 -3 residents at each level of training – Complete Milestones narrative • Initial time: 1 -2 hours/resident

CCC: logistics • Faculty development! – Understanding milestones – Understanding assessment tools – Aggregating and interpreting data

Entrustable Professional Activities (aka EPAs) Judy Pauwels MD University of Washington jpauwels@uw. edu

Goal • Envision Entrustable Professional Activities and their relationship to the Milestones

What are EPAs? • You have just traveled to a small town far from home, and become ill. • Whom do you TRUST to turn for help?

Entrustable Professional Activities • EPAs are the routine professional-life activities of physicians based on their specialty.

“Entrustable” • “Entrustable”: designates that a physician has demonstrated the necessary knowledge, skills, and attitudes to be trusted to independently perform this activity.

Miller’s model • Miller’s hierarchy: EPA is “does”.

EPA Examples • Managing an adult with acute illness during an inpatient hospitalization • Conducting the well-adult visit or wellwoman examination • Managing outpatient chronic disease • Performing labor management and vaginal delivery

Why are the Milestones not enough? • They are a substrate formative feedback and a learning roadmap, but. . • This granularity does not reflect the complex integration of the competencies needed in actual day-to-day practice • Perpetuates the view that the competencies are isolated and somewhat superficial addons to training requirements.

EPAs and the Milestones • EPAs represent the core clinical activities of our specialty, appropriately scaled to level of experience, comprised of and mapping to the Milestones. • EPAs make assessment more practical and meaningful.

EPAs and the Milestones • “Do you trust this person to monitor a labor and perform a vaginal delivery without direct supervision? ” Vs. • “Is this person competent in patient care, systems-based practice, etc. ? ”

Key characteristics of EPAs • In aggregate, they represent the essential professional work that defines our discipline • Lead to a recognizable outcome • Are observable and measurable • Require integration of the competencies across domains • Maps to the milestones

How do we construct EPAs? • Process EPAs: activities constant across rotations (handoffs, discharges) • Content EPAs: variable, rotation-specific material (OB delivery, CHF mgmt)

Example process EPA • “To serve as the primary admitting physician for previously well adults suffering from common acute problems”. • To perform this, a physician must: 1) Know the signs and symptoms of these illnesses (MK) 2) Perform a physical exam to elicit findings (PC) 3) Search for outcomes associated with specific therapeutic interventions (PBL) 4) Communicate with the patient and family about the management plan (ICS) 5) Relate to families in a way that they understand (ICS) 6) Act as a liaison to the community (SBP)

Example content EPA • “To perform labor management and vaginal delivery” • To perform this, a physician must: 1) Know normal labor curves, fetal monitoring interpretation, and management of common complications (MK) 2) Perform physical examination and common procedures during labor (PC) 3) Perform vaginal delivery, including recognizing and managing common complications (PC) 4) Communicate with the patient and family about the management plan (ICS) 5) Effectively lead the health care team involved in the care of the patient (ICS, P)

How would EPAs be evolved? • Need for Family Medicine to define EPAs – and limit the number! • Map EPAs to the Milestones. • Implications for: – Teaching: how do we teach this? – Formative feedback: what specifically do we assess, and how? – Summative evaluation: what level of trust? How do we effectively remediate learners who are struggling? – Self-evaluation – Program assessment

Constructing EPAs • Behaviors paint a picture of the learner at each level of development • Vignettes created to bring those behaviors to light in authentic clinical experience • Vignette mapping used to assess the learner

Constructing EPAs • Title • Description: what is the activity? • Required knowledge, skills, attitudes: which competency domains apply? • Information to assess progress (formative feedback) • When is unsupervised practice expected, and what will be the basis for that decision (summative evaluation)?

Formative feedback • Residents must be observed performing key activities at a certain competence level before they can be entrusted to perform them independently • Performance – observable and assessable activities residents perform during specific resident-patient interactions that are manifestations of their ability • Multisource feedback

Summative Evaluation • Instead of being asked to judge on specific competencies, more global question: at what level of TRUST in this resident does the observer feel? • How many times does an EPA need to be executed proficiently? Who will judge?

Evaluating resident in EPAs • Trust scale: – Resident has knowledge but no execution, and requires full direct supervision. – Resident may act under proactive, ongoing, full supervision. – Resident may act under reactive supervision. i. e. , supervision is readily available on request – Resident may act independently. – Resident may act as a supervisor and instructor.

Are EPAs the next step? • They make sense to faculty, residents, and our patients. • They align what we assess with what we do. • They add meaning to assessment, and makes it more practical. • Meets a current need to define the content of Family Medicine as a specialty.

Questions?

Wendy Biggs, MD wbiggs@kumc. edu Sam Cullison, MD Samuel. Cullison@mhd. com Julie Dostal, MD julie. dostal@lvhn. org Larry Mauksch, Med mauksch@u. washington. edu Judy Pauwels, MD jpauwels@uw. edu Stephen Wilson, MD wilsons 2@upmc. edu