Resident to Resident Using an educational framework to

Resident to Resident: Using an educational framework to develop an ACGME competency based evaluation tool. Kimberly Painter, MD, MPH, Nicole, Kirchen, MD, MPH, Tara B. Stein, MD, Pablo Joo, MD, Edgar Figueroa, MD, MPH Carma Bylund, Ph. D New York-Presbyterian Family Medicine Program Columbia University’s Center for Family Medicine Society of Teachers in Family Medicine April 27, 2006

Context Shift to competency-based evaluation Need to develop an evaluation tool for residents to evaluate other residents. Review of the Literature: – Few Tools – Not Valid (created by attendings in other residencies and not validated) – No Educational Framework Used – Not based on the ACGME Competencies

Setting 6 -6 -6 Inner city urban residency program Dedicated inpatient Family Medicine Service – 4 residents on service every 4 weeks – Main venue where residents work with each other

The Process Preparation (3 months) Nominal Groups (6 months) Tool Development (3 months) – 7 item instrument – 6 items use the formative clinical competency scale Not evaluated added as selection – 1 open-ended section for comments Testing – currently ongoing

Stages of Clinical Competency 1 Novice – Rule governed behavior Advanced Beginner – Principles begin to formulate Competent – Actions in terms of long-range goals or plans Proficient – – Increased speed and flexibility Use maxims to appreciate the importance of nuances in a situation Expert – Intuitive grasp of situations – Applies highly skilled analytic to unfamiliar situations. 1. Brenner, P. (1984). From Novice to Expert. Menlo Park: Addison-Wesley.

Nominal Group Technique Nominal Groups (Delbecq, 1975) – efficient, structured process for problem solving and idea generation – serves to clarify, achieve consensus, and set priorities – all viewpoints are presented and all participants are involved equally Our small group brainstormed criteria to use for our tool based on ACGME competencies The criteria were presented to groups of R 1 s, R 2 s, and R 3 s who then voted on the criteria's feasibility and importance in the tool The small group then summed up the votes to determine the 6 most feasible and important criteria to be used in the tool Delbecq, A. L. , Van de Ven, A. , & Gustafson, D. H. (1975) Group Techniques for Program Planning. Glenview, IL: Scott, Foresman.

Example ICS : Works effectively with others as member or leader of the team 1. Novice – Attempts to complete assigned personal responsibilities. 2. Advanced Beginner – Effectively completes assigned personal responsibilities. 3. Competent – Assists other team members when requested. 4. Proficient – Proactively provides assistance to team members. 5. Expert Competency – Assesses team needs and facilitates team’s ability to work together effectively. Not Evaluated

Reliability Testing Tool completed twice (test-retest) – At end of rotation – Two weeks after rotation Challenges to completion – Resident time – Resident understanding of need Less difficult to overcome due to inclusive process

Preliminary Data Response rates – 83/128 (65%), representing 16/18 residents on initial test – 51/128 (48%) on retest – 50/83 (60%) with comments on initial test Means on initial test – – – PGY-1=3. 0 PGY-2=3. 7 PGY-3=4. 6 No significant difference between scores on initial test versus repeat test.

Future Directions Dissemination of data to residents Identified limitations in tool – Now being revised Integrating into formal resident evaluation process

Contact Information Pablo Joo, MD Columbia University Center for Family Medicine paj 13@columbia. edu 212 -544 -1880