Integrating Competency Assessment in Chairs Residency A Primer

Integrating Competency Assessment in. Chairs, Residency A Primer for Program Directors & Champions Training Wendy Biggs MD Sam Cullison MD Larry Mauksch MEd

Goal: To help participants develop a culture and structure of competency based education in residency programs Objectives: Participants will be able to: Describe the competency based assessment pathway. Describe the importance of faculty development for success in competency assessment Describe common barriers when implementing a competency bases assessment program Describe the relationship of the milestone project to competency based education

STFM Residency Competency Assessment Toolkit http: //www. stfm. org/rctoolkit/ Presentations ADFM RPS “On the Road” STFM Precon and Lecture discussion Future conferences PDW

Outline Introduction Formative Assessment Establishing a Culture– Getting Started Design, Skills, Training Barriers to Training and Implementation Tools and Data Consolidation Faculty Development Summary

Introduction

Your Survey Results April 2012 What is the state of competency measurement in your programs?

Training Level for Faculty 3. 3% Unfamiliar 6. 7% Fully Embraced 16. 7% Partial Implement ACGME Competencies 6. 7% No Implement 66. 7% Accept & started

Implemented Competencies?

How Impacted?

What is Level of Resistance to Change?

‘The biggest problem in evaluating competencies is, in our opinion, not the lack of adequate assessment instruments but, rather, the inconsistent use and interpretation of those available by unskilled faculty’ Green ML, Holmboe ES. Perspective: The ACGME Toolbox: Half Empty or Half Full? Academic Medicine 2010; 85: 787 -790. 11

Competency Assessment Competency assessment embraces the notion that capabilities of residents have to be measured, rather than assuming the attainment of competence is an inherent part of the training process. A residency program, when functioning well, is a coherent, purposeful, integrated design and delivery of managed learning that enables residents to become competent and capable clinicians.

Competency Assessment Future Requirements on Competency Measurement with Milestones Current RC requirements On Competency Measurement

Formative Assessment

Formative Competency Assessment A series of observations Determine trainee knowledge, attitude and skill Shaping the trainee’s ability to apply skills in diverse contexts Entrustable Professional Activities Ten Cate, Medical Teacher, 2010, 32, 669 -675

Examples of Formative Assessment Competency based: teacher provides a learner with feedback about his or her performance to support the learner’s growth • Descriptive • Related to developmentally appropriate goals Non-competency based: comments to a learner that are global, judgmental or ‘out of the blue’ not related to a specific activity or specific goal

Formative Feedback examples cont’d ‘Your explanation of the diagnosis of diabetes was couched in terms the patient could understand so that he could ask appropriate questions. ’ Your explanation of diabetes was clear, yet not overwhelming with too much information. You asked for questions, printed an after-visit summary with a plan to see the health educator, and checked for understanding.

ACGME Competencies Patient care • compassionate, appropriate, effective Medical knowledge • biomedical, clinical, cognate sciences, and their application Practice-based learning and improvement • Life long learning; appraisal and assimilation of evidence Interpersonal and communication skills • effective interactions, teaming with patients and families Professionalism • Responsible, ethical, and sensitive Systems-based practice • Aware and responsive to larger context and system of health care, use of system resources

Establishing a Culture of Competency Assessment

STFM Residency Competency Assessment Toolkit Walk you through residency competency assessment redesign

How do I get started? From tool kit Create a culture of assessment Refine program goals Inventory your current processes Develop an improvement strategy Implement your plan Develop a competency based curriculum Balance formative and summative assessment Avoid common pitfalls

STFM Residency Competency Assessment Toolkit Ühttp: //www. stfm. org/rctoolkit

Culture Shift Assume that your program is capable of developing competence in every resident, and the purpose of assessment is simply to document attainment as it occurs.

Leadership is critical! Who are your champions?

Strategies for Creating a Culture of Competency Assessment Faculty members share responsibility for defining (or approving) goals Create a culture of direct observation Develop a shared lexicon Structure resident reviews to categorize comments by ACGME competencies Design closed circuit observations by two faculty at the same time to promote shared experiences

Design Implementing competency assessment

Programmatic Competency Pathway created by Judy Pauwels, MD and Larry Mauksch, M. Ed Competencies • Patient Care • Medical Knowledge • Practice Based Learning and Milestones • For each competency, every 6 months Improvement • Interpersonal and Communication Skills • Professionalism • System Based Practice Formative Assessment • Direct Observation • Structured tools • Field Notes • Surveys • Guided Learning Experiences • Simulations Goals and Objectives for each learning experience Consolidate Learning Experience Summaries • Assessment language is the same across summaries, for each milestone and competency Summative assessments • Semi-annual reviews • Goal setting • Problem solving 27 • Final Evaluation

Sample Goals and Milestones

Formative Comments and the Milestones - Example Interpersonal and Communication Skills Residency Goal: Physician should effectively lead the teambased patient care in the PCMH model ACGME Level 3 Milestone: Facilitates or leads team-based patient care activities. Actively participates in meetings not directly related to patient care.

Formative Comments and the Milestones Observe third year resident leading the FPC team staff meetings “You prepared the agenda in advance and managed the flow of the meeting well” “You asked for and addressed each staff members concerns about the new patient electronic registration kiosk”

Design Devise a system for regular direct observation from which formative assessment data can be collected • When possible, select data collection tools that transcend competencies Collect information from multiple sources (360 o) including staff, patients, preceptors and peers. Devise a method of centralized collection of formative data to track progress towards residency and resident goals • Paper files • Online resources

Skills and Training Implementing competency assessment

Miller’s Assessment Pyramid Miller, G. E. The assessment of clinical skills/competence/performance. 1990 Acad Med 65(9 Suppl): S 63 -7. Adapted from Holmboe and Hawkins. Evaluation of Clinical Competence, Mosby 2008 Impact on the patient Does • Faculty observation (with real patients) Shows how Knows • Standardized patients • Matching or critical response questioning • Multiple choice questions 33

Formative VS Summative Roles Formative Summative Coach(preceptor) Evaluator Uses prior data to influence teaching Using prior data to determine promotion Regularly engages the learner in Measures the learner’s goal setting performance against external goals Integrates learner preferences into educational design Prescribes experiences to the learner

Formative Assessment Skills Elicits student learning goals Considers the learner’s developmental stage Uses skillful feedback technique Is articulate Tracks progress and contributes to tracking Establishes a relationship with the learner Self discloses

Helpful Feedback Is… Timely About changeable behavior? Kind Not judgmental Honest Specific Provisional and curious

Barriers to Competency Assessment What are your barriers?

Barriers to Training and Implementation Lack of training Lack of time Lack of leadership Lack of language Feeling overwhelmed with data collection

Direct Observation: Methods, Time Demand, Pros and Cons Faculty Time Demand Educational Pros Educational. Co ns Direct observation in the room High Loss of income or other activity Clear view Can teach on the fly Trainee initially self conscious Risk of upstaging relationship Video review High Loss of income or other activity Trainee self observes, strong educational options Delayed practice Requires technical expertise and expense Closed circuit Moderate Some income loss or other activity Fast practice Faculty development Distraction, time limitation, Reliability? Very Low ++Observations ++Reflection ++Practice Less depth, versatility, Reliability? Low Role modeling Observer self Faculty growth Passive trainee role Peer You (faculty) are observed

Tools Implementing competency assessment

Data Collection Form design • There is NO PERFECT FORM! • Straight-forward, done frequently Need enough formative information for summative assessment

Select tools that: Are feasible to use Cover multiple competencies Educate the user Have broad faculty support Have existing psychometric properties, or Are qualitative and used by several faculty members frequently

‘Documented observations with feedback is the sine qua non of competency based education’ • M. Donoff, MD University of Alberta • Donoff MG. Field notes: assisting achievement and documenting competence. Can Fam Physician. Dec 2009; 55(12): 1260 -1262, e 11001262. 43

Data Collection – Sample tools • Mini-CEX • Field Notes

Data Consolidation Who oversees and insures data collection Determine where data is collected • Files • Online Make data collection transparent Give residents access Data should be used regularly to promote growth (formative) rather than for summative purposes

Faculty development

‘There is a growing consensus that the rate-limiting step in evolution of CBME is faculty development. ’ ‘If we are to transform medical education for the good of the public, faculty must also fully embrace their role as evaluators. The role of faculty as expert “coaches” must encompass teaching, assessment and feedback. ’ • Holmboe ES et al. Faculty Development in Assessment: The Missing Link in Competency-Based Medical Education. Academic Medicine. 2011; 86: 460 -467. 47

Faculty Development Help faculty understand the philosophy of competency assessment and how it differs from what has been used Build faculty investment through shared creation of goals Practice using formative assessment tools • Observe video together • Frequent practice increases inter-rater reliability • Plan methods for preceptors to easily record impressions Plan regular discussions amongst faculty about assessment experience to helps the group become more articulate

Wrap-up

Link to Two Videos The videos of the physician who does and does not demonstrate relationship development, agenda setting and appropriate EHR use can be found at the link below on the “Improving Communication Assessment Project”. The full videos last about 10 minutes and 30 seconds and are for training to use the Patient Centered Observation Form (PCOF). Total training takes about 40 minutes. To show these videos to a group, one must fill in the online data before advancing to the next video. Please do not press the final submit button on the last page unless you enter accurate ratings because this will ruin our data. It is fine to view the videos and enter arbitrary PCOF ratings to advance to the next video and leave the site after viewing the second video. http: //uwfamilymedicine. org/pcof

References Davidoff F. Music lessons: what musicians can teach doctors (and other health professionals). Ann Intern Med. Mar 15 2011; 154(6): 426 -429. Dijkstra J, Van der Vleuten CP, Schuwirth LW. A new framework for designing programmes of assessment. Adv Health Sci Educ Theory Pract. Aug 2009; 15(3): 379 -393. Donoff MG. Field notes: assisting achievement and documenting competence. Can Fam Physician. Dec 2009; 55(12): 1260 -1262, e 1100 -1262. Gwande A. Personal Best: Top Athletes and singers have coaches. Should you? The New Yorker. 2011. http: //www. newyorker. com/reporting/2011/10/03/111003 fa_fact_gawande. Accessed February 29, 2012. Holmboe ES, Ward DS, Reznick RK, et al. Faculty development in assessment: the missing link in competency-based medical education. Acad Med. Apr 2011; 86(4): 460 -467. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system--rationale and benefits. N Engl J Med. Mar 15 2012; 366(11): 1051 -1056. Ross S, Poth CN, Donoff M, et al. Competency-based achievement system: using formative feedback to teach and assess family medicine residents' skills. Can Fam Physician. Sep 2011; 57(9): e 323 -330. ten Cate O, Snell L, Carraccio C. Medical competence: the interplay between individual ability and the health care environment. Med Teach. 2010; 32(8): 669 -675. Wenrich MD, Jackson MB, Ajam KS, Wolfhagen IH, Ramsey PG, Scherpbier AJ. Teachers as learners: the effect of bedside teaching on the clinical skills of clinician-teachers. Acad Med. Jul 2011; 86(7): 846 -852.