ACGME Core Competency Documentation with a Webbased Evaluation
ACGME Core Competency Documentation with a Web-based Evaluation System Jennifer Welsh MD, Nancy J. Baker MD and Deanne St. George Fairview University Smiley’s Clinic Department of Family Medicine and Community Health University of Minnesota
Lecture Discussion Goal n We will demonstrate how an electronic, web-based evaluation system helps us document resident success at achieving each of the ACGME core competencies
Session Objectives n n n Review the key components of a comprehensive electronic evaluation system Demonstrate our use of behavioral anchors to assess resident progress with meeting the ACGME core competencies Propose means to use a web-based electronic evaluation system as a key component of individual resident portfolios
Lecture Discussion Outline n n Brief history of resident evaluation at Fairview University Smiley’s Residency Overview of our current, web-based evaluation system components Description of our resident portfolios Group discussion
Smiley’s Evaluation History n Pre-2001: Paper evaluations n n n poor compliance 2001: Transition to electronic evaluations n n n Rotations, FPC performance, clinic pt. care staff Paper process made electronic, web-based cumbersome 2002: Competency-based evaluations n n implemented a standardized resident evaluation form with ACGME core competency categories improved compliance
Evaluation History Cont’d. n 2002: Assessing Competency: new tools n n n 2003: Assessing Competency: new anchors n n Electronic, annual competency self assessment PDA compatible electronic procedure log Used medical school evaluation anchors as a model Gained consensus on, and added behavioral anchors Faculty agreed to “ 2” as a passing grade 2004: Better 360 degree evaluation n Trained FPC patient care staff (PCS) to do electronic evaluation of resident team members
Our Web-based Resident Evaluation Components n n n n Monthly resident evaluation by rotation faculty Semi-annual resident evaluation in the FPC by preceptors/residency faculty and patient care staff “On-the-fly” praise and concern cards Procedure log Annual competency self-assessment Conference attendance Annual video review
Smiley’s Evaluation Template: Core 13 Questions n Patient Care n n n n EBM Use of IT n n Professional conduct and communication Ethics Multiculturalism System Based Practice n n Teamwork/Work habits Interpersonal and Communication Skills n Application of basic sciences Problem Solving Practice Based Learning Professionalism n Medical Knowledge n n Interpersonal skills Physical exam and diagnostic workup n Cost effective care Overall performance Comments
Monthly Resident Evaluation by Rotation Faculty: Patient Care 1. Interpersonal Skills Cannot Evaluate 0 Poor = Interview rushed; resident rude; pt. irritated or angry with res. at times Fair = Business-like interview; res. rushed at times; minimal therapeutic relationship Good = Interview professional, appropriate, and respectful Very Good = Nurturing interview that creates a collaborative MD/pt. rel. ; pt. apprec. of MD care Excellent = Able to estab. excellent pt. rapport; time-efficient interview 1. 0 2. 0 3. 0 4. 0 5. 0 2. Physical Exam/Diagnostic Work-up Cannot Evaluate 0 Poor = Hx, PE and diagnostic studies poorly focused or inappropriate for visit Fair = Hx, PE and diagnostic studies appropriate for a single, simple issue only Good = Hx, PE and diagnostic studies appropriate for more than one issue or pertinent for complicated visit Very Good = Hx, PE and diagnostic studies consistently focused/ appropriate for all types of pt. visits Excellent = Hx, PE and diagnostic studies well focused, concise, time efficient 1. 0 2. 0 3. 0 4. 0 5. 0
Monthly Resident Evaluation by Rotation Faculty: Medical Knowledge 3. Application of Basic Sciences Cannot Evaluate 0 Poor = Lacks familiarity or knowledge of the basic sciences Fair = Superficial understanding of medical knowledge as it pertains to simple medical problems; fails to look up answers Good = Possesses a clear underst. of basic sci. as it applies to simple med. prob. and some more compl. health issues; occas. looks up questions Very Good = Is familiar with a broad range of med. knowledge and can apply it routinely to both simple and compl health issues; frequently looks up ? ’s Excellent = Has mastered the basic sciences and can apply new knowledge to both simple and complicated health issues 1. 0 2. 0 3. 0 4. 0 5. 0 4. Problem Solving Cannot Evaluate Poor = Unable to develop and prioritize potential diagnoses Fair = Able to develop and prioritize diagnoses for single prob. or simple office visit Good = Able to prioritize well for all single problems and most complicated visits Very Good = Prioritization is accurate, with emphasis on most serious and treatable options for all visits Excellent = Prioritizes potential dx. in a concise, accurate and time-efficient manner 0 1. 0 2. 0 3. 0 4. 0 5. 0
Monthly Resident Evaluation by Rotation Faculty: Practice Based Learning Practice-Based Learning 5. Evidence Based Medicine (EBM) Cannot Evaluate Poor = Lacks understanding of the principles/ tools of EBM Fair = Has a basic understanding of EBM but infreq applies to pt. care Good = Understands and occas applies EBM to answer clinical questions Very Good = Understands and applies EBM to answer simple clinical questions Excellent = Routinely uses EBM to answer both simple and complicated med. questions 0 1. 0 2. 0 3. 0 4. 0 5. 0 6. Use of Information Technology (IT) Cannot Evaluate 0 Poor = Unable to use electronic databases/ resources to assist with pt. care and to provide relevant info to pts. Fair = Infrequent use of IT tools to assist with pt. care and to provide relevant info to pts. Good = Uses PDA, internet or electronic databases to answer simple clinical questions and to provide relevant info to pts. Very Good = Regular use of IT to answer both simple and complicated clinical questions and to provide relevant info to pts. Excellent = Uses IT routinely and in a time -efficient manner to answer both simple and compl. clinical questions and provide relevant info to pts 1. 0 2. 0 3. 0 4. 0 5. 0
Monthly Resident Evaluation by Rotation Faculty: Professionalism 7. Teamwork/Work Habits Cannot Evaluate 0 Poor = Poor attention to responsibilities; frequent miscommunicatio n with team; inefficient; frequently late to clinic and behind schedule Fair = Attends to responsibilities but infrequently assists team; gaps in communication and efficiency; usually prompt and stays on time Good = Assists team when asked; notes and chart documentation usually thorough and timely; arrives promptly and stays on time Very Good = Completes work in a timely and thorough manner; is a dependable team player; arrives early to anticipate schedule and manages time well Excellent = Consistently looks for opportunities to help team function well; exhibits team leadership; efficient; teaches team; exhibits flexibility and mature problem solving skills to manage time effectively 1. 0 2. 0 3. 0 4. 0 5. 0
Monthly Resident Evaluation by Rotation Faculty: Interpersonal & Communication Skills Interpersonal and Communication Skills 8. Professional Conduct and Communication Cannot Evaluate 0 Poor = Appearance and conduct often inappropriate; awkward interactions with support staff and colleagues Fair = Appearance and conduct usually appropriate; interactions with support staff and colleagues usually smooth Good = Appearance and conduct appropriate; routinely relates well to support staff, colleagues/consultants Very Good = Appearance and conduct always professional; receives praise from colleagues and support staff Excellent = Models exemplary behavior at all times; creates “win-win” relationships with other health care professionals 1. 0 2. 0 3. 0 4. 0 5. 0 9. Ethics Cannot Evaluate 0 Poor = Lacks honesty and integrity; not trustworthy; uncomfortable with prof. boundaries; fails to recog. ethical issues Fair = Minimal awareness of prof. boundaries; infreq. recog. of ethical issues related to clin. care Good = Honest, straight-forward and trustworthy; maintains profess. boundaries; recognizes ethical issues in clin. care Very Good = Introspective and reflective; Systematically considers ethical consequences of clinical decision- making Excellent = Demonstrates exceptional commitment to pts. ; freq. raises ethical issues; routinely seeks input from others re: wholistic care and ethical concerns 1. 0 2. 0 3. 0 4. 0 5. 0 10. Multiculturalism Cannot Evaluate Poor = Appears unaware/ disinterested in issues related to cultural diversity Fair = Has limited awareness of issues related to cultural diversity Good = Able to perform Hx and PE w/ sensitivity to issues related to cultural diversity Very Good = Is able to acknowledge diversity issues in developing culturally sensitive diagnostic/ therapeutic treatment plans Excellent = Embraces cultural diversity and seeks out new resources to improve care to multiculturally diverse patients 0 1. 0 2. 0 3. 0 4. 0 5. 0
Monthly Resident Evaluation by Rotation Faculty: Systems-Based Practice Systems Based Practice 11. Cost Effective Care Cannot Evaluate 0 Poor = Poor understanding of medical costs, billing & coding issues & utilization of health care resources Fair = Aware of medical costs; has difficulty with billing & coding & utilizing health care resources Good = Uses most billing codes & health care resources appropriately Very Good = Routinely bills & codes correctly & uses health care resources appropriately Excellent = Able to handle complex billing issues & judiciously manage competing health care costs 1. 0 2. 0 3. 0 4. 0 5. 0 Poor = Not performing at the level of his/her peer class; substandard clinical evaluations Fair = Usually performs at a level consistent with his/her peer class; occasional clinical deficiencies Good = Consistently performs work in a satisfactory manner at a level commensurate with peer class Very Good = Above avg. clinical performance; exhibits learning characteristics that distinguish him/her from peer class Excellent = Outstanding performance in all clinical areas; level of mastery exceeds that of peer class 1. 0 2. 0 3. 0 4. 0 5. 0 12. Overall Performance Cannot Evaluate 0 13. Comments:
Semi-annual Resident Evaluation in the FPC by PCS - 8 of 13 Questions n Patient Care n n n n EBM Use of IT n n Professional conduct and communication Ethics Multiculturalism System Based Practice n n Teamwork/Work habits Interpersonal and Communication Skills n Application of basic sciences Problem Solving Practice Based Learning Professionalism n Medical Knowledge n n Interpersonal skills Physical exam and diagnostic workup n Cost effective care Overall performance Comments
Overall G 3 Performance Comparisons YTD Resident Number of Evals G 2 Mean Score Number of Evals G 3 Mean Score Resident 1 31 4. 39 23 4. 30 Resident 2 29 4. 07 24 4. 25 Resident 3 23 3. 91 32 4. 16 Resident 4 21 3. 62 20 3. 95 Resident 5 28 4. 14 29 4. 20
“On-the-fly” Praise and Concern Cards n n Electronic “card” outlines praise or concern, reason for comments, and comfort level in discussing with resident Provide a means for teachers to communicate immediate praise or concern about a resident to program director
“On-the-fly” Praise and Concern Cards Subject: Evaluator: Dates of Activity: Adolescent Medicine Evaluation Type: Concern Card About A Trainee Information entered on this form is considered confidential. Access to this information is available only to the program director(s) and to the E*Value system administrator(s). Reason for Concern My concerns about the performance and/or professional behavior of this physician are based on: (please check) Critical Incident Gut level reaction Series of "red flags" Concern Comments: Yes No Discussed With Physician. I have discussed my concerns with the physician. Yes No Discomfort with discussion of concern. I feel uncomfortable discussing my concerns with the physician. Yes No Call about concern. Please call me about these concerns.
Example of “On-the-fly” Praise Card n Reason for Praise: n Clinical Skills Communication Skills n Professionalism n n Praise n Resident A was post-call today. He made a special effort to do an overdue pap smear for a patient who was late for her appointment and had came for a different reason. Resident A went above and beyond and fit the pap smear into an already busy clinic. I was impressed with his selflessness and professionalism. He easily could have told the patient it had to be done at another time.
Example of “On-the-fly” Concern Card n n Reason for Concern: Series of "red flags" Concern n Medical knowledge: Resident B presented a case of a young man with prostate sx. and discharge. We did a lit search and came up with a series of articles. I asked him to review the information in a systematic review and give me the "clinical bottom line. " He was unable to find this in the article, and instead pointed to individual study findings. He was unable to develop a plan for how to treat his patient. n Problem solving: Resident B seems to have difficulty solving problems in clinic when working with nursing staff, and figuring out how things work. He frequently calls for clarifications about his schedule and wants it figured out for him. His problem solving for medicine is better, but limited to straight-forward problems. With complex problems he is unable to synthesize all the information and make a decision.
G 1 Procedure Log Summary Procedure Name Required Completed Remaining Abscess I & D 2 0 2 Endometrial Biopsy 2 0 2 Colposcopy (with biopsy and/or ECC) 5 0 5 Labor and Delivery, Uncomplicated Vaginal Delivery 5 19 Requirement Met Laceration Repair-Simple 5 2 3 IUD Placement and Removal 2 0 2 Exercise Treadmill Testing 2 0 2 Arthrocentesis/Joint injection 2 0 2 Colposcopy (without biopsy) 5 0 5
Sample Annual Competency Self-Assessment NA Observe d Managed with assistance Managed without assistance Feel compete nt to manage alone 0 1 2 3 4 5 Child Abuse Competent in assessment and management of child abuse 0 1 2 3 4 5 Myocardial Infarction Competent in assessment and management of myocardial infarction 0 1 2 3 4 5 Normal Labor and Delivery Competent in assessment and management of normal labor and delivery 0 1 2 3 4 5 Sexually Transmitted Diseases Competent in assessment and management of sexually transmitted infections 0 1 2 3 4 5 Congestive Heart Failure Competent in assessment and management of congestive heart failure 0 1 2 3 4 5 Clinical E&M Skill Able to teach Pre and postoperative management. Competent in pre and postoperative management, including wound care
PGY 3 Annual Competency Self-Assessment Clinical E&M Skills Competency Self. Assessment 1 end of year 1 Competency Self-Assessment 2 end of year 2 2002 -2003 Self-Assessment 2003 -2004 Self-Assessment 2 5 0 3 2 4 2 4 2 4 2 3 Pre and postoperative management Scale: 1 to 5 Child Abuse Scale: 1 to 5 Myocardial Infarction Scale: 1 to 5 Normal Labor and Delivery Scale: 1 to 5 Sexually Transmitted Diseases Scale: 1 to 5 Congestive Heart Failure Scale: 1 to 5 Chemical Dependency Scale: 1 to 5 Acute Head Trauma/LOC Scale: 1 to 5
Conference Attendance (PGY 2 Resident C’s attendance) Confer en ce s As si gn ed 101 Confer % Confer Confe en Ov en re ce er ce Confer nc s all s en es Pr Att ces Ex es en Ab en cu en da de se se te nc nt d d d e* * "% Overall Attendance" is calculated as follows: Percent. Attended = (Attended. Count + +29 Late. Count) - Upcoming. Count - Excused. Count) * 100. 49 Presented. Count 15 0 / (Total. Count 76. 56
Resident Portfolio Designed to be compact and simple n Easy for faculty, learner and program to use n n Information in consistent location for each resident both within portfolio and in clinic Show RRC we’re measuring core competencies n Design to match components of ABFM’s residency training module n
Components of the Resident Portfolio n n n Professional Summary Residency Training Information Competency Based Resident Performance
Resident Evaluation Summary n Evaluations types n n n n Faculty (Rotations and Clinic Evaluations) Supervising Resident Patient Care Staff “On-the-fly” Self Assessment Video Review By defined time period By defined report format
Resident Performance: ACGME Core Competencies n Patient Care Resident evaluation summary n Shadow and video review n EMR audits (pending) n Procedure log n Annual competency self assessment n Quarterly resident review by faculty n
Resident Performance: ACGME Core Competencies n Medical Knowledge In-training exam scores n Resident evaluation summary n Quarterly resident review by faculty n
Resident Performance: ACGME Core Competencies n Practice-based learning Annual competency self-assessment n In-training exam n Resident evaluation summary n Quarterly resident review by faculty n CQI practice data n
Resident Performance: ACGME Core Competencies n Interpersonal and communication skills Shadow and video review evaluations n Resident evaluation summary n Quarterly resident review by faculty n
Resident Performance: ACGME Core Competencies n Professionalism Shadow and video review checklist/evaluations n Resident evaluation summary n Primary care support staff evaluations 360 degree evaluation n Patient satisfaction surveys (pending) n Quarterly resident review by faculty n
Resident Performance: ACGME Core Competencies n Systems-based practice EMR audits (pending) n Attendance at coding education seminars n In-training exam n Practice management activities n CQI practice data n
Future Use and Benefits of the Resident Portfolio n n n Hope to convert hard copy to electronic format compatible with the ABFM residency training maintenance (RTM) system Provides portability for the resident Competency self-assessment tool could be the basis for residency postgraduate survey
Summary: Advantages of an Electronic Evaluation System n n n Improves evaluation compliance; “relatively” easy to administer and track Provides an easily accessible “snapshot” of YTD resident performance Residents are expected to complete evaluations promptly and to see that their teachers also comply Provides important reporting capabilities Helpful for resident verification upon graduation Provides better data on how residents are performing
- Slides: 35