Competency Assessment in Residency Training Philosophy Tools Faculty

Competency Assessment in Residency Training: Philosophy, Tools, Faculty Development and Barriers Wendy Biggs, MD - Assistant Director, Division of Medical Education, AAFP Alan David, MD – RPS and MCW Larry Mauksch M. Ed. – Lead, Faculty Development and Dissemination Work Group – CAFM Task Force, University of Washington Family Medicine Residency Program Solutions Pre Conference March 31, 2012 Kansas City

Goal: To help participants develop a culture and structure of competency based education in residency programs Objectives: Participants will be able to: Define and describe competency based assessment and list core skills and tools needed. Design a faculty development program on competency assessment in a residency Describe common barriers and how to address them when implementing a competency bases assessment program Describe the relationship of the milestone project to competency based education

‘The biggest problem in evaluating competencies is, in our opinion, not the lack of adequate assessment instruments but, rather, the inconsistent use and interpretation of those available by unskilled faculty’ Green ML, Holmboe ES. Perspective: The ACGME Toolbox: Half Empty or Half Full? Academic Medicine 2010; 85: 787 -790. 3

Council of Academic Family Medicine (CAFM) Lead – STFM • Start January , 2011 Residency Competency Measurement Task Force • Interdisciplinary Charge: To identify and disseminate appropriate materials and training on the establishment and measurement of competency in residency education 4

Task Force Membership Resident Competency Measurement Task Force Members Chair Sam Cullison, MD, Task Force Chair - Swedish FMR Seattle Wendy Biggs, MD, AAFP Colleen Conry, MD, University of Colorado Alan David, MD, Medical College of Wisconsin Mike Donoff, MD, University of Alberta Julie Dostal, MD, Competency Assessment Work Group Lead, Lehigh Valley FMR Larry Mauksch, MEd, Faculty Development and Dissemination Work Group Lead, University of Washington FMR Tom O'Neill, Ph. D, ABFM Allen Shaughnessy, Pharm. D, Tufts University FMR Stephen Wilson, MD, UPMC St Margaret FMR Stacy Brungardt, CAE, Society of Teachers of Family Medicine 5

STFM Residency Competency Assessment Toolkit http: //www. stfm. org/rctoolkit/ Presentations ADFM RPS STFM Precon and Lecture discussion PDW “On the Road” Future conferences

Outline Introduction Establishing a Culture– Getting Started Design, Skills, Training Barriers to Training and Implementation Break Tools and Data Consolidation Faculty Development exercises Summary

Introduction

Competency Assessment Competency assessment embraces the notion that capabilities of residents have to be measured, rather than assuming the attainment of competence is an inherent part of the training process. A residency program, when functioning well, is a coherent, purposeful, integrated design and delivery of managed learning that enables residents to become competent and capable clinicians.

Why is this important? Think about your best resident. How have you helped him/her grow and prepare to be one of our leaders? Think about a resident in difficulty and the struggles the faculty have had in describing what is wrong and identifying how to help? One of the reasons we have difficulty is that we have not developed a common, articulate language among ourselves or with our residents

Current RC Requirements (p 33): V. Evaluation: Formative • 1. Provide objective assessments of competence in patient care, medical knowledge, practice based learning and improvement, interpersonal and communication skills, professionalism and systemsbased practice • 2. Use multiple evaluations (e. g. faculty, peers, patients…) • 3. Document progressive resident performance improvement appropriate to educational level • 4. Provide each resident with semi-annual evaluation of performance with feedback 11

Summative (p 34) • The PD must provide a summative evaluation for each resident upon completion of the program. This policy • a) document the resident’s performance during the final period of education, and • b) verify that the resident has demonstrated sufficient competence to enter practice without direct supervision 12

Proposed New RC Requirements – (on hold) Add under ‘Formative’ section • The program should assess individual resident competence in each of the six competency areas on entrance into the program. • Interpersonal and Communication Skills assessment must include both direct and indirect observation and multisource evaluation (including at least patients, peers and non-physician team members) 13

Proposed New RC Requirements – (on hold) Under ‘Program Evaluation and Improvement – resident performance’ • The program must assess the resident in data gathering, clinical reasoning, patient management and procedures in both the inpatient and outpatient setting. This assessment must involve direct observation of resident-patient encounters. • The program must use an objective validated formative assessment method (e. g. in-training examination, chart stimulated recall). The same formative assessment method must be administered at least annually. (? ? ? ) 14

Formative Assessment

Formative Competency Assessment A series of observations Determine trainee knowledge, attitude and skill Shaping the trainee’s ability to apply skills in diverse contexts Entrustable Professional Activities Ten Cate, Medical Teacher, 2010, 32, 669 -675

Examples of Formative Assessment Competency based: teacher provides a learner with feedback about his or her performance to support the learner’s growth • Descriptive • Related to developmentally appropriate goals Non-competency based: comments to a learner that are global, judgmental or ‘out of the blue’ not related to a specific activity or specific goal

Formative Assessment (coaching) Examples ‘You did a good job handling that patient earlier today. ’ ‘I like the way you explained diabetes to that man. ’

Formative Feedback examples cont’d ‘Your explanation of the diagnosis of diabetes was couched in terms the patient could understand so that he could ask appropriate questions. ’ Your explanation of diabetes was clear, yet not overwhelming with too much information. You asked for questions, printed an after-visit summary with a plan to see the health educator, and checked for understanding.

ACGME Competencies Patient care • compassionate, appropriate, effective Medical knowledge • biomedical, clinical, cognate sciences, and their application Practice-based learning and improvement • Life long learning; appraisal and assimilation of evidence Interpersonal and communication skills • effective interactions, teaming with patients and families Professionalism • Responsible, ethical, and sensitive Systems-based practice • Aware and responsive to larger context and system of health care, use of system resources

Exercise 2 ÜSecond video

Establishing a Culture of Competency Assessment

STFM Residency Competency Assessment Toolkit Walk you through residency competency assessment redesign

Residency Competency Assessment Toolkit Headings What is competency based education Why assess competency How do I get started Assessment methods and tools Resources Frequently asked questions Faculty development

How do I get started? From tool kit Create a culture of assessment Refine program goals Inventory your current processes Develop an improvement strategy Implement your plan Develop a competency based curriculum Balance formative and summative assessment Avoid common pitfalls

What is your culture of assessment How valued is assessment? Who are the change agents? Where are you in balancing formative and summative assessment?

STFM Residency Competency Assessment Toolkit Ühttp: //www. stfm. org/rctoolkit

Culture Shift Assume that your program is capable of developing competence in every resident, and the purpose of assessment is simply to document attainment as it occurs.

Strategies for Creating a Culture of Competency Assessment Faculty members share responsibility for defining (or approving) goals Create a culture of direct observation Develop a shared lexicon Structure resident reviews to categorize comments by ACGME competencies Design closed circuit observations by two faculty at the same time to promote shared experiences

Miller’s Assessment Pyramid Miller, G. E. The assessment of clinical skills/competence/performance. 1990 Acad Med 65(9 Suppl): S 63 -7. Adapted from Holmboe and Hawkins. Evaluation of Clinical Competence, Mosby 2008 Impact on the patient Does • Faculty observation (with real patients) Shows how Knows • Standardized patients • Matching or critical response questioning • Multiple choice questions 31

Competency Based Education Adapted from Miller, G. E. The assessment of clinical skills/competence/performance. 1990 Acad Med 65(9 Suppl): S 63 -7. Impact on Training Does • Implements and evaluates CBE in residency program Shows how Knows • Demonstrates use of direct observation feedback to learner • Can describe and generate competency assessment tools/skills • Can define formative and summative assessment 32

Trying to make small changes without fundamental changes in thinking and structure is unlikely to be successful

Break

Design Implementing competency assessment

Programmatic Competency Pathway created by Judy Pauwels, MD and Larry Mauksch, M. Ed Competencies • Patient Care • Medical Knowledge • Practice Based Learning and Improvement • Interpersonal and Communication Skills • Professionalism Milestones • Every six months for each competency • System Based Practice Data consolidation • Assessment language should be the same across rotations for each milestone within each competency Semi-annual reviews • Goal setting • Problem solving Formative Assessment • Direct Observation • Structured tools • Field Notes • Surveys • Guided Learning Experiences • Simulations Final Evaluation 36

Steps in Competency Assessment Design Dijkstra, Van Der Vleuten, Schuwirth A new framework for designing programs of assessment, Adv in Health Sci Ed 2010 15, 379 -393 What are your program goals? What methods are needed to collect data to determine resident progress towards achieving goals? How will you consolidate information from various sources How will this information be reported or shared? Who will make decisions? Who monitors quality of assessment program?

Sample Goals and Milestones

Formative Comments and the Milestones - Example Interpersonal and Communication Skills Residency Goal: Physician should effectively lead the teambased patient care in the PCMH model ACGME Level 3 Milestone: Facilitates or leads team-based patient care activities. Actively participates in meetings not directly related to patient care.

Formative Comments and the Milestones Observe third year resident leading the FPC team staff meetings “You prepared the agenda in advance and managed the flow of the meeting well” “You asked for and addressed each staff members concerns about the new patient electronic registration kiosk”

Design Devise a system for regular direct observation from which formative assessment data can be collected • When possible, select data collection tools that transcend competencies Collect information from multiple sources (360 o) including staff, patients, preceptors and peers. Devise a method of centralized collection of formative data to track progress towards residency and resident goals • Paper files • Online resources

Skills and Training Implementing competency assessment

Miller’s Assessment Pyramid Miller, G. E. The assessment of clinical skills/competence/performance. 1990 Acad Med 65(9 Suppl): S 63 -7. Adapted from Holmboe and Hawkins. Evaluation of Clinical Competence, Mosby 2008 Impact on the patient Does • Faculty observation (with real patients) Shows how Knows • Standardized patients • Matching or critical response questioning • Multiple choice questions 43

Formative VS Summative Roles Formative Summative Coach (preceptor) Evaluator Uses prior data to influence teaching Using prior data to determine promotion Regularly engages the learner in Measures the learner’s goal setting performance against external goals Integrates learner preferences into educational design Prescribes experiences to the learner

Formative Assessment Skills Elicits student learning goals Considers the learner’s developmental stage Uses skillful feedback technique Is articulate Tracks progress and contributes to tracking Establishes a relationship with the learner Self discloses

Notes on Feedback Balance emphasis on strengths with growth areas • Increase awareness of strengths Feedback: facilitative, not directive • Adopt a humble, non expert stance Lead with curiosity • Make an observation • Share your wonder about the thoughts or feelings • Ask if he/she want a suggestion

Helpful Feedback Is… Timely About changeable behavior? Kind Not judgmental Honest Specific Provisional and curious

Barriers to Competency Assessment What are your barriers?

Barriers to Training and Implementation Lack of training Lack of time Lack of leadership Lack of language Feeling overwhelmed with data collection

Lack of training How many here have had “formal” training in competency assessment?

Lack of time Clinician-educators vs educator-clinicians Initial outlay of time for faculty development Real-time feedback (formative assessment)

Direct Observation: Methods, Time Demand, Pros and Cons Faculty Time Demand Educational Pros Educational Cons Direct observation in the room High Loss of income or other activity Clear view Can teach on the fly Trainee initially self conscious Risk of upstaging relationship Video review High Loss of income or other activity Trainee self observes, strong educational options Delayed practice Requires technical expertise and expense Closed circuit Moderate Some income loss or other activity Fast practice Faculty development Distraction, time limitation, Reliability? Very Low ++Observations ++Reflection ++Practice Less depth, versatility, Reliability? Low Role modeling Observer self Faculty growth Passive trainee role Peer You (faculty) are observed

Lack of language Faculty need to describe observed behavior in the same terms • Videos • Simulations • Role play Need to create a shared set of standards Create language that goes across rotations

Lack of leadership Need buy-in from the top down • Need a frame shift – competency assessment versus time completion Milestones are coming! Always easier to “DO IT” than have it “DONE TO YOU”

Tools Implementing competency assessment

Data Collection Form design • There is NO PERFECT FORM! • Straight-forward, done frequently Need enough formative information for summative assessment

Select tools that: Are feasible to use Cover multiple competencies Educate the user Have broad faculty support Have existing psychometric properties, or Are qualitative and used by several faculty members frequently

‘Documented observations with feedback is the sine qua non of competency based education’ • M. Donoff, MD University of Alberta • Donoff MG. Field notes: assisting achievement and documenting competence. Can Fam Physician. Dec 2009; 55(12): 1260 -1262, e 11001262. 58

Programmatic Competency Pathway adapted from a model created by Judy Pauwels, MD Competencies • Patient Care • Medical Knowledge • Practice Based Learning and Improvement • Interpersonal and Communication Skills • Professionalism Milestones • Every six months for each competency • System Based Practice Data consolidation • Assessment language should be the same across rotations for each milestone within each competency Formative Assessment • Direct Observation • Structured tools • Field Notes • Surveys • Guided Learning Experiences • Simulations Semi-annual reviews • Goal setting • Problem solving Final Evaluation

Data Collection – Sample tools Exercise • Mini-CEX • Field Notes

Sample Online Tracking Tool Work in Progress “Comptrac”

Data Consolidation Who oversees and insures data collection Determine where data is collected • Files • Online Make data collection transparent Give residents access Data should be used regularly to promote growth (formative) rather than for summative purposes

Faculty development

‘There is a growing consensus that the rate-limiting step in evolution of CBME is faculty development. ’ ‘If we are to transform medical education for the good of the public, faculty must also fully embrace their role as evaluators. The role of faculty as expert “coaches” must encompass teaching, assessment and feedback. ’ • Holmboe ES et al. Faculty Development in Assessment: The Missing Link in Competency-Based Medical Education. Academic Medicine. 2011; 86: 460 -467. 64

Faculty Development Help faculty understand the philosophy of competency assessment and how it differs from what has been used Build faculty investment through shared creation of goals Practice using formative assessment tools • Observe video together • Frequent practice increases inter-rater reliability • Plan methods for preceptors to easily record impressions Plan regular discussions amongst faculty about assessment experience to helps the group become more articulate

Residency Competency Assessment Toolkit Headings What is competency based education Why assess competency How do I get started Assessment methods and tools Resources Frequently asked questions Faculty development

STFM Residency Competency Assessment Toolkit http: //www. stfm. org/rctoolkit/ ÜResources ÜFAQs

Planning Your Next Step Name one or more faculty development strategies • Be specific, consider sequence • What preparation do you need? Name one or more design elements to implement • Be specific, consider sequence • What preparation do you need? Larry Mauksch, M. Ed University of Washington Department of Family Medicine

Sharing and Wrap-up

Link to Two Videos The videos of the physician who does and does not demonstrate relationship development, agenda setting and appropriate EHR use can be found at the link below on the “Improving Communication Assessment Project”. The full videos last about 10 minutes and 30 seconds and are for training to use the Patient Centered Observation Form (PCOF). Total training takes about 40 minutes. To show these videos to a group, one must fill in the online data before advancing to the next video. Please do not press the final submit button on the last page unless you enter accurate ratings because this will ruin our data. It is fine to view the videos and enter arbitrary PCOF ratings to advance to the next video and leave the site after viewing the second video. http: //uwfamilymedicine. org/pcof

References Davidoff F. Music lessons: what musicians can teach doctors (and other health professionals). Ann Intern Med. Mar 15 2011; 154(6): 426 -429. Dijkstra J, Van der Vleuten CP, Schuwirth LW. A new framework for designing programmes of assessment. Adv Health Sci Educ Theory Pract. Aug 2009; 15(3): 379 -393. Donoff MG. Field notes: assisting achievement and documenting competence. Can Fam Physician. Dec 2009; 55(12): 1260 -1262, e 1100 -1262. Gwande A. Personal Best: Top Athletes and singers have coaches. Should you? The New Yorker. 2011. http: //www. newyorker. com/reporting/2011/10/03/111003 fa_fact_gawande. Accessed February 29, 2012. Holmboe ES, Ward DS, Reznick RK, et al. Faculty development in assessment: the missing link in competency-based medical education. Acad Med. Apr 2011; 86(4): 460 -467. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system--rationale and benefits. N Engl J Med. Mar 15 2012; 366(11): 1051 -1056. Ross S, Poth CN, Donoff M, et al. Competency-based achievement system: using formative feedback to teach and assess family medicine residents' skills. Can Fam Physician. Sep 2011; 57(9): e 323 -330. ten Cate O, Snell L, Carraccio C. Medical competence: the interplay between individual ability and the health care environment. Med Teach. 2010; 32(8): 669 -675. Wenrich MD, Jackson MB, Ajam KS, Wolfhagen IH, Ramsey PG, Scherpbier AJ. Teachers as learners: the effect of bedside teaching on the clinical skills of clinician-teachers. Acad Med. Jul 2011; 86(7): 846 -852.