Evaluation of Learners Effective Techniques and Strategies Jennie

Evaluation of Learners Effective Techniques and Strategies Jennie Hsu-Lumetta, M. D. March 16, 2005 1 10/29/2020

“Clinical Learning is Experience Examined…” Ende and Davidoff, 1992 2 10/29/2020

Goals and Objectives 1. Define the purpose of evaluations. 2. Identify the steps to providing effective evaluations. 3. List some potential barriers/errors to providing effective evaluation. 3 10/29/2020

What Will Be Covered. . . 1. Definition, Purposes, Types of Evaluation 2. Pre-Evaluation Activities 3. Completing the Evaluation 4. Writing Good Comments 5. Discussing Evaluation 6. RIME Model for Clinical Evaluation 4 10/29/2020

Definition of Evaluation To determine the value or worth of a Program, Product, Performance Process 5 10/29/2020

Evaluation of Learners The process by which the teacher assesses the learners’ knowledge, skills, and attitudes, based on criteria related to educational goals. 6 10/29/2020

Why Evaluate? Past: Assess performance Present: Improve performance Future: Identify future needs 7 10/29/2020

Uses of Medical Student Evaluations Use Purpose Audience Past Assessment Clerkship Director Present Improvement Student Future Dean’s Letter Dean 8 10/29/2020

Uses of Resident Evaluations Use Purpose Audience Past Assessment Program Director Present Improvement Resident Future Program Director’s Letter Fellowship, Employers 9 10/29/2020

What to Evaluate? Attitude Skills Knowledge 10 10/29/2020

ASK’s Attitude, Skills, Knowledge Competency includes the following: l Attitude: (as observed via behaviors) Prepared/Punctuality/Proactive Follow-up with patient/educational issues l Skills – context specific l Knowledge – Has the information – Able to analyze and synthesize information – Able to apply it effectively and efficiently 11 10/29/2020

ACGME Competencies 1. Patient Care 2. Medical Knowledge 3. Practice-Based Learning and Improvement 4. Interpersonal and Communication Skills 5. Professionalism 6. Systems-Based Practice 12 10/29/2020

ACGME Competencies Patient Care that is compassionate, appropriate, and effective for the treatment of health problems and the promotion of health Medical Knowledge about established and evolving biomedical, clinical, and cognate (e. g. epidemiological and social-behavioral) sciences and the application of this knowledge to patient care 13 10/29/2020

ACGME Competencies Practice-Based Learning and Improvement that involves investigation and evaluation of their own patient care, appraisal and assimilation of scientific evidence, and improvements in patient care Interpersonal and Communication Skills that result in effective information exchange and teaming with patients, their families, and other health professionals 14 10/29/2020

ACGME Competencies Professionalism, as manifested through a commitment to carrying out professional responsibilities, adherence to ethical principles, and sensitivity to a diverse patient population Systems-Based Practice, as manifested by actions that demonstrate an awareness of and responsiveness to the larger context and system of health care and the ability to effectively call on system resources to provide care that is of optimal value 15 10/29/2020

ACGME - Evaluation of Residents use of dependable measures to assess 6 areas of competency l mechanisms for providing regular and timely performance feedback to residents l a process involving use of assessment results to achieve progressive improvements in residents' competence and performance l 16 10/29/2020

Types of Evaluation 1. 2. Formative Evaluation Summative Evaluation 17 10/29/2020

Types of Evaluation 1. Formative Evaluation l Assess a learner’s progress toward educational goals l Throughout the teaching process l Basis for feedback for the learners and to make teaching adjustments as needed to learner to reach learning goals 18 10/29/2020

Types of Evaluation 1. Summative Evaluation l Final assessment of a learner l End of teaching/learning experience l Used to judge a learner’s mastery of the educational goals 19 10/29/2020

How to Evaluate? Assessment/Evaluation l Formative (ongoing) Feedback Evaluation l Summative (end of rotation) 20 10/29/2020

Elements of Effective Assessment Evaluation Prerequisites l Making Good Comments l Aligning Comments with Scaled Questions l Discussing the Evaluation with the Learner l Source: David Rubin, MD and James Woodruff, MD. Teaching Skills for the Medical Educator Workshop. University of Chicago, March 23, 2002 21 10/29/2020

Effective Assessment 1: Evaluation Prerequisites ü ü ü Understand expectations of the learner Understand the evaluation criteria Understand the survey instrument Have appropriate expectations Provide on-going feedback Provide opportunities for observation 22 10/29/2020

Key components of Evaluation 1. 2. Observation of the Learners Questioning 23 10/29/2020

Key components of Evaluation : Observation of the Learners The teacher can observe learner performance for direct evaluation of knowledge, skills, and attitudes. 24 10/29/2020

Key components of Evaluation : The One minute Observation ü ü ü ü Explain the purpose of the observation to the learner Explain how the observation will occur Select one skill for observation Inform the patient of what will take place Observe for a brief period without interrupting Leave the room and give immediate feedback Use the information gain to plan your teaching Repeat to observe learner’s skills over time 25 10/29/2020

Key components of Evaluation : Questioning of the Learners 1. 2. Types of Questions Levels of Questions 26 10/29/2020

Key components of Evaluation : Types of Questions 1. Open-ended ü What? ü How? ü Why? 2. Closed-ended ü Do? ü Can? ü Have? 27 10/29/2020

Key components of Evaluation : Level of Questions 1. 2. 3. Recall Analysis/Synthesis Application 28 10/29/2020

Effective Assessment 2: Completing the Form v Write narrative comments first v v Comments are most important!! Grades are a “blunt instrument” Dr. Rubin, U of Chicago Dept. of Medicine v v Give specific examples Keep comments to focused behaviors Connect comments to ratings Check for internal consistency 29 10/29/2020

Sources of Comments l Direct observation of learner l Written records l Responses to questions l Homework assignments l Interactions with healthcare team, patients l Learner self-evaluation l Feedback/Midpoint evaluation – Interventions with the learner – Learner’s response to interventions 30 10/29/2020

Effective Assessment 3: Writing Effective Comments l l l Strengths and weaknesses Areas of concern Suggestions for improvement Ensure internal consistency Focus on behaviors not personalities 31 10/29/2020

Effective Assessment 4: Discuss with Learner l l l l Set a time Allow enough time for discussion Have the learner perform a self-assessment Explain your rationale Emphasize changes in behavior and performance that leads to desired competency Suggest areas for future improvement Give evaluation in oral and written forms *Adding feedback element increases the usefulness of the evaluation and avoids surprises 32 10/29/2020

Avoid rating errors: l The “halo/horn” effect: – based on reports of prior performance l Restriction range: – using the same rating for all components of the evaluation l Rating nonperformance attributes: – using behavior attributes when rating nonbehavioral qualities – Evaluate performance ONLY 33 10/29/2020

RIME Model for Clinical Evaluation l Developed by Dr. Louis Pangaro l Model for student evaluation that was credible and reliable l Student performance -> Progression of 4 developmental steps l Research shows that it is predictive of future performance 34 10/29/2020

RIME Model for Clinical Evaluation 4 Stages of Clinical Development: Reporter Interpreter Manager Educator 35 10/29/2020

RIME Model for Clinical Evaluation l Reporter – collect and report data (by the M 3 year) l Interpreter – interpret history, PE findings and lab results, create & prioritize a problem list and differential diagnosis (by end of M 3 year) l Manager – select the most appropriate diagnostic and treatment options (by 2 nd & 3 rd residents) l Educator – identify and address knowledge gaps (senior residents) 36 10/29/2020

Proficient Reporter l l l Interview & History Physical examinations Demonstrates adequate medical knowledge Generates adequate problem list Demonstrates respect for patients Consistently reliable & honest Interpreter • Interprets basic test data • Integrates medical knowledge with findings • Generates several reasonable explanations • Can discuss issues & process of informed consent 37 10/29/2020

Advanced Manager l l l Develop diagnostic & therapeutic plans Makes decisions using benefit/risk analysis Uses evidence-based approach Performs as a reliable member of the team Manages relationships with patients 38 10/29/2020

Outstanding Educator l l Demonstrates leadership Demonstrates self-directed learning Exhibits critical reading skills Demonstrates teaching skills 39 10/29/2020

Summary for Effective Evaluators: Understand learner expectations l Understand the evaluation instrument: l – E-value, RIME model Include thoughtful comments l Connect comments to rating scales l Have multiple opportunities for assessing performance l Evaluate performance ONLY l Discuss evaluation with the learner l 40 10/29/2020

Special thanks: l Nancy F. Barrett l Carol Packard 41 10/29/2020

Case You sit down with your student on her last day of the rotation although you’ve never been comfortable giving evaluations. Knowing evaluations are supposed to be balanced, you try to list several positive things and then list problems. When you mention that the student’s cardiac patient who spent an extra day in the hospital because she didn’t track down a stress test result in time on Friday before the cardiology department closed, she angrily bursts out how difficult it is on a team where no one will give her any help. She also says that you haven’t spent as much time reviewing write-ups with her as other residents. 42 10/29/2020