Competency Grid Competency Improvement through objective development and

Competency Grid Competency Improvement through objective development and process improvement Todd Darian Shaffer, MD, MBA, FAAFP Professor of Medicine Program Director University of Missouri- Kansas City FMRP

Objectives • Develop a competency-based curriculum for each rotation or educational experience • Develop competency based objectives for each educational experience • Discuss the importance of linking objectives with assessment methods • Develop a 7 way Grid to bring all the integration together for future planning, improvement, and outcomes measurement.

Timeline of ACGME Competency Project Phase 2 - 7/02 -06 • Program Focus – Provide learning opportunities in all six competency domains – Improve evaluation processes as needed – Provide aggregate resident performance data for Internal Review • Phase 3 - 7/06 -6/11 Program Focus Use resident performance data as basis for improvement Begin to use external measure (patient surveys, clinical quality measure) to verify resident and program performance Accreditation Focus – Review evidence that programs are teaching and addressing the competencies – Provide constructive citations early and more constructive citations later in phase – Internal Review includes aggregate resident performance data • Accreditation Focus – Evidence that programs are making data driven improvements – Review the external program performance indicators and input from GMEC to make sure your program is achieving its educational objectives

New ACGME Changes Effective July 1, 2007 • Competency based goals and objectives for each experience at each level of training – These should be available and reviewed by the resident at the start of the rotation • Evaluation process specifically addresses competencies • Expanded language describing competencies

Common Characteristics of CBE (Competency Based Education) In CBE, teaching and learning are: 1. explicit and clearly aligned with expected competencies; 2. criteria-driven, focusing on accountability in reaching benchmarks and, ultimately, competence; 3. grounded in “real-life” experiences; 4. focused on fostering the learners’ ability to self-assess; 5. individualized, providing more opportunities for independent study.

ACGME Expectations Learning opportunities in each competency domain Provide clearning objectives and learning opportunities for each of the competencies throughout your curriculum. Evidence of multiple assessment methods Assessment of competency-based learning objectives should provide evidence of your program’s ability to prepare residents for practice. One assessment tool will not fully measure a resident’s performance so you will need to use multiple assessment methods. A portfolio involves the use of multiple assessment methods. The assessment approaches listed below provide an example of a minimum set of assessment approaches you should have in place. global evaluation tool with behavioral anchors; 360 degree evaluation (Peer and Professional Associate rating); Focused observation tool; cognitive test (in-training exam or board scores); and case logs (if applicable). Use of aggregate data to improve the educational program Aggregate data reflects how residents, as a group, are performing in specific areas. Aggregated performance data refers to summary resident assessment results, such as. percentage of residents passing the certification exam on the first attempt or the program’s percentile rank on in-training exams by PGY level; and the extent to which residents are advancing in capabilities across their years of residency. The aggregate results should reflect specific areas of resident competence that programs are targeting for improvement through educational interventions.

We must sanely accomplish: 1. 2. 3. 4. 5. 6. Develop appropriate goals Determine How we will teach them Provide the information to learn Determine How we will evaluate them Determine the quality of the evaluation method Determine and identify outcomes measurements 7. Continuously work in a changing needs analysis environment

How do we develop and track these? • Break it up into parts • Reassemble into a “Dashboard” improvements can more easily be identified across all areas of the curriculum. • Provides a roadmap (or tool) to improvement strategies

General Competencies vs Competencies for Individual Objectives • This is only a new language for taxonomy • We already have been teaching this way, it just helps to identify and classify the problem learner • Also Helps to objectively identify the best learners

ACGME Toolbox

ACGME Six Steps to Developing a Competencybased Curriculum 1. Conduct needs assessment 2. Identify competencies addressed by this rotation or experience 3. Write goals and objectives 4. Determine teaching methods 5. Determine assessment methods 6. Determine program improvement methods

Reordered Six Steps to Developing a Competency-based Curriculum 1. Conduct needs assessment 2. Write goals and objectives 3. Identify competencies addressed by each individual objective identified 4. Determine assessment methods for each objective down to the individual competency 5. Determine teaching methods for each Objective 6. Determine program improvement methods based on ACGME priority measures of both teaching methods and evaluation methods.

Writing Goals and Objectives • Provide an instructional roadmap • Provide clarity about the educational program to learners, teachers and external stakeholders • Encourage higher order learning • This should be where you and the faculty spend their time • Don’t worry about the competencies at this point

Goals (Major categories) • A statement that describes what the learner will gain from the instruction • Think of goals as broad overarching statements • Goals are not necessarily measurable

Objectives (Minor categories) • Contain behaviorally explicit statement reflecting what a resident will learn • Three Rules: – Understandable – Obtainable – Measurable Tips: Need to get away from the old classic views of Knowledge, Skills and Attitudes-these are all intermingled in the new competencies. Do not create too many objectives (<30 is best for each rotation or experience) You can still create “extra” stuff beyond the minimum they learn. Do not overwhelm the learner or they will learn nothing!

Overall Objective Plan

Objective Action Verbs Information cite count define describe draw identify indicate list name point quote read recite recognize record relate report select state summarize tabulate tell trace update Comprehension assess associate classify compare compute contrast demonstrate describe differentiate discuss distinguish experiment explain express extrapolate interpret locate predict report restate review translate Application apply calculate choose complete demonstrate develop employ examine illustrate match operate order practice prescribe restate schedule sketch solve treat use utilize Analysis analyze appraise contract criticize debate deduce detect diagram infer inspect inventory measure question separate summarize Synthesis arrange assemble collect combine compose construct create design detect document formulate generalize integrate manage organize plan prepare prescribe produce propose specify validate Evaluation appraise assess choose compare critique decide determine estimate evaluate grade judge measure rank rate recommend revise score select test Verbs that communicate knowledge (higher thinking required as you go across)

Classic Objective Skill and Attitude Verbs that impart skills (These would be hard to test for on-line) demonstrate empathize listen operate visualize diagnose hold manage project write diagram internalize measure record Verbs that convey attitudes (we would probably not use) acquire exemplify plan consider modify reflect transfer realize revise

Verbs and phrases to AVOID (When Writing Objectives) • • • • Appreciate Be acquainted with Be aware of Be familiar with Become Capable of Become Cognizant of Become Conscious of Believe Comprehend Conceptualize Enjoy Feel Fully Appreciate • • • • Grasp the Significance of Have faith in Know Learn Perceive Really Understand Realize Recognize Remember See Sympathize with Understand Value* Some of these may be useful in Goal statements *The verb “value” is found in both the affective Domain table as well as the Verbs to Avoid Table. Using the verb “value” with a qualifying behavioral outcome, ie “the learner will value the importance of confidentiality by following the acceptable chart documentation protocols. ” However, using in another context results in a goal statement- “the student will value cost effective practice” is a goal statement.

Community Medicine Goal # 1: Understand the needs of the vulnerable population in the community Objectives: By the End of the Rotation, the resident will: 1. 2. Assemble a list of vulnerable population Select and journal about a visit to a homeless shelter 2. Goal # 2: The will understand the role of public health in the community Objectives: By the end of the Rotation, the resident will: 1. 2. 3. List the top 10 communicable reportable diseases Identify the top 10 public health concerns Explain 5 functions of the public health department

Cardiology Goal # 1: The resident will effectively manage patients with CHF Objectives: By the End of the Rotation, the resident will: 1. Describe the outpatient pharmacological and Non-pharmacological management of CHF in the ambulatory setting 2. Distinguish the difference between types of CHF (Systolic, Diastolic, Combined) 3. Measure 10 of your patients compliance with #1 Number 3 is really one of your measurements, maybe should be something like “Achieve greater than 70% compliance in recommendations at Follow up from initial visit of diagnosis”

Pediatrics Goal # 1: The Resident will understand the various components of wellchild care Objectives: By the End of the Rotation, the resident will be able to: 1. 2. 3. Demonstrate the ability to find apply age appropriate immunization schedule Demonstrate the ability to complete the newborn exam Demonstrate the management of developmental abnormalities in the first 2 years of life

Emergency Medicine Goal # 1: The Resident will be able to accurately assess patients level of severity on initial presentation Objectives: By the End of the 1 st Rotation, the PGY-1 resident will be able to: 1. 2. List the criteria for definition of Sepsis Employ the criteria for Sepsis in waiting ICU admission orders ? ? ? Not for sure

Geriatrics Goal # 1: By the end of the geriatrics rotation, the resident will be able to evaluate and manage a patient presenting with memory loss Objectives: 1. 2. 3. List the Ddx of memory loss (pneumonic) Demonstrate a mini mental Exam Appraise (Assess) memory loss by cost effective testing

Neurology Goal # 1: The Resident will recognize, evaluate and manage stroke Objectives: By the End of the Rotation, the resident will be able to: 1. 2. 3. 4. Demonstrate the use of the CHAD S 2 criteria to calculate risk factors for stroke Demonstrate a focused neurologic exam for acute neurologic deficit Generate a Ddx for acute memory deficit Summarize the indications, contraindications, and risks of the use of t. PA in Stroke

Sports Medicine Goal # 1: The Resident will recognize the roles and responsibilities of the Team Physician Objectives: By the End of the Rotation, the resident will be able to: 1. 2. 3. Assess an athlete with the signs and symptoms of a concussion, and apply appropriate return to play rules Define the role of Team Physician as it relates to the Trainer, Coach, player, and institution Demonstrate an onsite knee, shoulder, and ankle exam

Obstetrics Goal # 1: By the end PGY-1, the resident will achieve skills to triage a pregnant woman presenting in L and D triage. Objectives: The resident will be able to: 1. 2. 3. Adequately be able to perform and correctly interpret a cervical exam Demonstrate ability to properly read, interpret, and assess FHT’s Define Criteria for Active Labor

How do the competencies fit in? • Complete after writing the Goals and Objectives • Each objective may involve more than one competency • Competencies are the language to pick apart all that you can evaluate about a particular learning point.

Surgery Goal # 1: By the end of the 1 st month surgery rotation the resident can perform a complete preoperative evaluation. Objectives: The resident will be able to: 1. 2. 3. Locate and Demonstrate the ACC/AHA pre-op evaluation guidelines (MK, SB) Write a succinct pre-op consult letter to a surgeon (ICS, SBP, P) Describe risk and benefits of procedure to the patient (ICS, MK, PC)

Educational Resources • List all educational resources – Books – Self directed learning modules – Slides – Videos – CD-ROMs and DVD’s – Web sites – PDF’s, etc

Teaching Methods • • Didactic conferences Clinical teaching Case based teaching Role modeling Journal Club Mentoring M&M • Simulation • Self directed learning modules • Individual or group projects • Research projects • Chart audit

Method for Program Evaluation

Important Criteria for Assessment Methods 1. Representative educational objectives 2. Educational objectives match assessment 3. Use multiple, appropriate assessment methods 4. Use multiple evaluators 5. Obtain multiple observations 6. Fair

Key Learning Points • Assessment tools should be: – Reliable – Valid – Feasible – Provide valuable information

Assessment Methods • Record the assessment methods you use to evaluate the resident’s performance – Align these with the ACGME preferred Methods Remember to use multiple methods

A Core Assessment System * Method Competency • • All competencies • Interpersonal and Communication Skills & Professionalism • Medical Knowledge • Patient Care • All 6 competencies • Global Clinical Performance Ratings Focused Evaluation/ – Observation of Patient Encounter • Multi-rater/360 -degree Evaluation • Performance on Cognitive Test (In Training Exam) Case Logs or similar data Portfolio • • * These assessment methods reflect the current thinking of the ACGME

Direct Observation (A few words) You will notice “Direct Observation” is not one of the ACGME published methods of evaluation. When starting this project, our program was mostly “directly observing” for evaluation of the residents. Now we can see where we need to improve our rotations easily in a global form by the grid. That is how the whole quality improvement comes into play. Identify what we are doing now, and then see how we can get to the preferred evaluation method for that particular goal. —Ie Process Improvement to have more objectives.

Supervision Asynchronous learning environment with modules and SAMs may come in handy

Outcome Measurement • Need to define good quality measurement tools • Some are in line with the present pay for performance models • Clear and objective tools for measuring end outcomes in patient populations and in quality of our graduates after graduation.

How do we integrate all this? • Need easy method to integrate. It is all part of the same thing • Should not think in silos but complete cross integration

ACGME Document • • • Competency Assessment Glossary Assessment Methods Competencies Educational Activities Improvements http: //www. acgme. org/outcome/e-learn/e_powerpoint. asp

Professionalism Communication Supporting Competencies Patient Care

PMBA Grid

Balint Grid

First Stab:

Where do we start?

7 Parts of the Grid 1. Objectives (number across top)

7 Parts of the Grid 1. Objective (number across top) 2. Core Competencies (along side)

7 Parts of the Grid 1. 2. Objective (number across top) Core Competencies (along side) 3. Teaching Methods (Represented by shading) Didactic conferences Clinical teaching Case based teaching Role modeling Journal Club Mentoring WIKI Quality Projects M&M Simulation Self directed learning modules Individual or group projects Research projects Chart audit

7 Parts of the Grid 1. 2. 3. Objective (number across top) Core Competencies (along side) Teaching Methods (Represented by shading) 4. ACGME Method of Evaluation (Letters In Box)

7 Parts of the Grid 1. 2. 3. 4. Objective (number across top) Core Competencies (along side) Teaching Methods (Represented by shading) ACGME Method of Evaluation (Abbreviations) 5. ACGME best methods of evaluation (Color)

7 Parts of the Grid 1. 2. 3. 4. 5. Objective (number across top) Core Competencies (along side) Teaching Methods (Represented by shading) ACGME Method of Evaluation (Abbreviations) ACGME best methods of evaluation (represented by color) 6. Hyperlink (directly to where the information can be found)

7 Parts of the Grid 1. 2. 3. 4. 5. 6. Objective (number across top) Core Competencies (along side) Teaching Methods (Represented by shading) ACGME Method of Evaluation (Abbreviations) ACGME best methods of evaluation (represented by color) Hyperlink (directly to where the information can be found) 7. Note corner triangle noting how outcomes could be measured

Putting it all together

Adding the “Seventh” Competency • AOA considers osteopathic principles a seventh competency • The grid easily allows the seventh competency to be added *

Key Points • Process Improvement • Identify broad overarching goals Strategy • Identify specific objectives the resident should learn • Align assessment method with objective • Be able to measure the outcomes • Create new Needs Assessment and start over • The cycle continues as needs and material changes over time.

Measuring Outcomes • BIG Project in process. • After Goals and objectives are written and implemented • Next setup is to identify measurable outcomes that are easy first. • Most likely will only have for a few for each rotation to begin with. • These would need to be traceable back to teaching and evaluation methods for putting into practice and showing their positive (or negative) effects on patient care and outcome.

Remediation and Probation • IEP’s Individual Education Plans can then be developed in line with where the deficiencies are occur. • Probationary letters specifically address the “IN”-competency and improvement plans will directly measure these deficiencies

Example A) Gynecologic Screening Goal: The resident should have a basic understanding of screening for gynecologic malignancies. Objectives: By the end of this rotation, the resident will be able to demonstrate: 1. Adequate skill at Pap smear performance and management of results (inflammation, ASCUS, AGUS and SIL). 2. Adequate skill at bimanual examination performance and understanding of tests and limitations thereof in screening for ovarian cancer. 3. Ability to state common presenting complaints for vulvar malignancies and their appearance. 4. List risk factors for various gyn malignancies. (HPV infection and the impact of the new HPV Vaccine) 5. Give indications for colposcopy, know basic procedure and recognize common findings (per ASCCP guidelines. )

As Changes Occur • Easily Change the interconnected grids, assessment methods, outcome measurements while you progress through time as Needs Assessments and RRC requirements change • I. e. Effective July 1, 2007, there were several eliminations and additions to the ACGME subcompetencies. These can easily be changed on your working grids.

Resources • ACGME Web site – www. acgme. org – Modules 1 -4 – ACGME Best Methods • Rockhurst MBA Faculty • RAP Presentations: – Ted Epperly, MD- G and O’s – UT FMRP R. Viken, MD-Competency Integration – Swedish FMRP Web Site

Blackboard