Applying the Performance Framework Data Model July 3

Applying the Performance Framework Data Model July 3, 2013

Performance Framework Metadata: lom, Effective. Date, Retired. Date, Replaces, Is. Replaced. By, Supporting. Information Performance. Scale: id, Least. Competent, Most. Competent 1+ Component • • id Title Abbreviation Competency (reference to • • • a competency object) Author Reviewer Additional. Information Component. Reference: id of a nested Component 0+ or 0 or 1 Performance. Level. Set Performance. Scale (reference to a Level. Scale id) 1 Performance. Level 2+ • Display. Order • Label • Additional. Information Score: Single. Value OR Range (Min. Score, Max. Score) 1 Indicator: id, Description, Additional. Information 1+

Internal Medicine Example • Title: The Internal Medicine Milestone Project • Identifier: – Catalogue: URI – Entry: http: //www. acgme-nas. org/assets/Internal. Medicine. Milestones • Description: The Milestones are designed only for use in evaluation of resident physicians in the context of their participation in ACGME accredited residency or fellowship programs… • Contributions: – – Role: Author Entity: William Iobst, M. D. Role: Author Entity: Eva Aagaard, M. D. • Effective date: 2013 -01 -30

Internal Medicine Continued • Supporting Information – This document presents milestones designed for programs to use … • Performance Scale – 1 to 5 (1 least competent, 5 most competent) – 1 to 3 (1 least competent, 3 most competent)

Internal Medicine Component • ID: 12345 • Competency: Description: http: //www. example. org/im_milestones/PC 1. xml Type: http: //ns. medbiq. org/competencyobject/v 1/ • Competency: Description: http: //www. example. org/im_framework. xml Type: http: //ns. medbiq. org/competencyframework/v 1/ • Title: Gathers and synthesizes essential and accurate information to define each patient’s clinical problem(s).

Performance level set • Performance Scale Reference: 1 to 5 Position: 1 Score: Single Value: 1 Label: Critical Deficiencies Indicator (id = i 1_1): Does not collect accurate historical data Indicator (id = i 1_2): Does not use physical exam to confirm history Indicator (id = i 1_3): Relies exclusively on documentation of others to generate own database or differential diagnosis Indicator (id = i 1_4): Fails to recognize patient’s central clinical problems Indicator (id = i 1_5): Fails to recognize potentially life threatening problems This would allow indicator to be tracked as on/off.

Position: 2 Score: Single Value: 2 Label: Indicator (id = i 2_1): Inconsistently able to acquire accurate historical information in an organized fashion Indicator (id = i 2_2): Does not perform an appropriately thorough physical exam or misses key physical exam findings Indicator (id = i 2_3): Does not seek or is overly reliant on secondary data Indicator (id = i 2_4): Inconsistently recognizes patients’ central clinical problem or develops limited differential diagnoses Position: 3 Score Single Value: 3 Label: Indicator (id = i 3_1): Consistently acquires accurate and relevant histories from patients Indicator (id = i 3_2): Seeks and obtains data from secondary sources when needed Indicator (id = i 3_3): Consistently performs accurate and appropriately thorough physical exams Indicator (id = i 3_4): Uses collected data to define a patient’s central clinical problem(s)

Position: 4 Score: Single Value: 4 Label: Ready for unsupervised practice Indicator (id = i 4_1): Acquires accurate histories from patients in an efficient, prioritized, and hypothesis-driven fashion Indicator (id = i 4_2): Performs accurate physical exams that are targeted to the patient’s complaints Indicator (id = i 4_3): Synthesizes data to generate a prioritized differential diagnosis and problem list Indicator (id = i 4_4): Effectively uses history and physical examination skills to minimize the need for further diagnostic testing Position: 5 Score Single Value: 5 Label: Aspirational Indicator (id = i 5_1): Obtains relevant historical subtleties, including sensitive information that informs the differential diagnosis; Indicator (id = i 5_2): Identifies subtle or unusual physical exam findings Indicator (id = i 5_3): Efficiently utilizes all sources of secondary data to inform differential diagnosis Indicator (id = i 5_4): Role models and teaches the effective use of history and physical examination skills to minimize the need for further diagnostic testing

Internal Medicine Component • Competency: Description: http: //www. example. org/im_milestones/PC. xml Type: http: //ns. medbiq. org/competencyobject/v 1/ • Competency: Description: http: //www. example. org/im_framework. xml Type: http: //ns. medbiq. org/competencyframework/v 1/ • ID: 23456 • Title: Patient Care • Additional Information (label = background): The resident is demonstrating satisfactory development of the knowledge, skill…

Performance levels • Performance Scale Reference: 1 to 3 Position: 1 Score: Single Value: 3 Label: Yes Indicator (id = pc 1): The resident is demonstrating satisfactory development of the knowledge, skill, and attitudes/behaviors needed to advance in training. He/she is demonstrating a learning trajectory that anticipates the achievement of competency for unsupervised practice that includes the delivery of safe, timely, equitable, effective and patient-centered care. Position: 2 Score: Single Value: 1 Label: No Indicator (id = pc 2): The resident is demonstrating unsatisfactory development of the knowledge, skill, and attitudes/behaviors needed to advance in training. He/she is not demonstrating a learning trajectory that anticipates the achievement of competency for unsupervised practice that includes the delivery of safe, timely, equitable, effective and patient-centered care.

Performance levels Position: 3 Score: Single Value: 2 Label: Marginal Indicator (id = pc 3): The resident is demonstrating marginal development of the knowledge, skill, and attitudes/behaviors needed to advance in training. He/she is marginally demonstrating a learning trajectory that anticipates the achievement of competency for unsupervised practice that includes the delivery of safe, timely, equitable, effective and patient-centered care.

Pediatrics Example • Title: The Pediatrics Milestone Project • Identifier: – Catalog: URI – Entry: http: //www. acgmenas. org/assets/Pediatrics. Milestones • Contributions: – Role: Author – Entity: Carol Carraccio, M. D. – Role: Reviewer – Entity: Richard Antonelli, MD, MS • Effective date: 2013 -01 -30

Pediatrics Continued • Supporting Information – http: //www. acgme. org/acgmeweb/Portals/0/PFAssets/Program. Resou rces/320_Peds. Milestones. Project. pdf • Performance Scale – 1 to 4 (1 least competent, 4 most competent) – 1 to 5 (1 least competent, 5 most competent) – 1 to 6 (1 least competent, 6 most competent)

Pediatrics Component • ID: 12345 • Title: Gather essential and accurate information about the patient • Competency: Description: http: //www. example. org/peds_milestones/PC 1. xml Type: http: //ns. medbiq. org/competencyobject/v 1/ • Competency Description: http: //www. example. org/peds_milestones. xml Type: http: //ns. medbiq. org/competencyframework/v 1/ • Author: Daniel Schumacher, MD • Additional Information (label = background): Early Development of Information-Gathering Skills In the early stages of clinical reasoning, learners must rely upon their knowledge of basic pathophysiology and …

Pediatrics Component Continued • Additional Information (label = reference, position = 1): Schmidt HG, Norman GR, Boshuizen HPA. A cognitive perspective on medical expertise: theory and implications. Academic Medicine. 1990; 65: 611 -621. • Additional Information (label = reference, position = 2): Carraccio CL, Benson BJ, Nixon LJ, Derstine PL. From the educational bench to the clinical bedside: translating the Dreyfus Developmental Model to the learning of clinical skills. Academic Medicine. 2008; 83: 761 -767. • Additional Information (label = reference, position = 3): Eva K. What every teacher needs to know about clinical reasoning. Medical Education. 2004; 39: 98 -106. • Additional Information (label = reference, position = 3): Schmidt HG, Boshuizen HPA. On acquiring expertise in medicine. Educational Psychology Review. 1993; 5: 205 -221. • Etc…

Performance level set • Performance Scale: 1 to 5 Position: 1 Score: Single Value: 1 Free text reference Label: Indicator (id = pc_i 1): Either gathers too little information or exhaustively gathers information following a template regardless of the patient’s chief complaint, with each piece of information gathered seeming as important as the next. Recalls clinical information in the order elicited, 7 with the ability to gather, filter, prioritize, and connect pieces of information being limited by and dependent upon analytic reasoning through basic pathophysiology alone. No formatting Position: 2 Indicator (id = pc_i 2): Score Single Value: 2 Label: Clinical experience allows linkage of signs and symptoms of a current patient to those encountered in previous patients. Still relies primarily on analytic reasoning through basic pathophysiology to gather information, but the ability to link current findings to prior clinical encounters allows information to be filtered, prioritized, and synthesized into pertinent positives and negatives as well as broad diagnostic categories.

Position: 3 Score: Single Value: 3 Label: Indicator (id = pc_i 3): Advanced development of pattern recognition leads to the creation of illness scripts, which allow information to be gathered while it is simultaneously filtered, prioritized, and synthesized into specific diagnostic considerations. Data gathering is driven by real-time development of a differential diagnosis early in the information-gathering process. 8 Position: 4 Score: Single Value: 4 Label: Indicator (id = pc_i 4): Well-developed illness scripts allow essential and accurate information to be gathered and precise diagnoses to be reached with ease and efficiency when presented with most pediatric problems, but still relies on analytic reasoning through basic pathophysiology to gather information when presented with complex or uncommon problems. Position: 5 Indicator (id = pc_i 5): Score: Single Value: 5 Label: Robust illness scripts and instance scripts (where the specific features of individual patients are remembered and used in future clinical reasoning) lead to unconscious gathering of essential and accurate information in a targeted and efficient manner when presented with all but the most complex or rare clinical problems. These illness and instance scripts are robust enough to enable discrimination among diagnoses with subtle distinguishing features.

Pediatrics Component • ID: 12345 • Title: Prescribe and perform all medical procedures • Competency: Description: http: //www. example. org/peds_milestones/PC 8. xml Type: http: //ns. medbiq. org/competencyobject/v 1/ • Competency: Description: http: //www. example. org/peds_milestones. xml Type: http: //ns. medbiq. org/competencyframework/v 1/ • Author: Patricia Hicks, MD

Additional Information (label = background) • All of the competencies are involved in prescribing and performing medical procedures. In an integrated …

Additional Information Continued The component KSA of each procedure are numerous and complex. They include: • Anatomy and Physiology • Indications and Benefits • Contra-indications and Risks • Informed Consent • Pain Management, Patient Psychological Preparation • Specimen Handling • Interpretation of Results or Outcomes • Procedural Technique (multiple elements unique to procedure; common elements to all [e. g. , sterile technique, situational awareness, course correction]) • Post-procedure Management This approach to assessment makes some assumptions: • Performance level is specific to each procedure based on the relevant components and level of responsibility of the physician. • Given the variability of required components, measures of competence are based on all of the relevant components for that procedure. • Performance level for a given procedure, therefore, requires reaching the desired performance level for each of the individual components.

Additional Information (label = reference, position = 1): Wigton R, Nicolas J, Blank L. Procedural skills of the general internist: a survey of 2500 physicians. Annals of Internal Medicine. 1989; 111: 1023 -1034. Additional Information (label = reference, position = 2): Wigton R, Blank L, Nicolas J, Tape T. Procedural skills training in internal medicine residencies. Annals of Internal Medicine. 1989; 111: 932 -938. Additional Information (label = reference, position = 3): Wigton R. Training internists in procedural skills. Annals of Internal Medicine. 1992; 116: 1091 -1093. Additional Information (label = reference, position = 4): Hicks C, Gonzales R, Morton M, et al. Procedural experience and comfort level in internal medicine trainees. General Internal Medicine. 2000; 15: 716 -722. Additional Information (label = reference, position = 5): Kirkpatrick DL. Evaluating Training Programs: The Four Levels. San Francisco, CA: Berrett-Koehler Publishers; 1998. Additional Information (label = reference, position = 6): Kirkpatrick L, Kirkpatrick JD. The four levels: an overview. In: Kirkpatrick DL, ed. Evaluating Training Programs: The Four Levels. San Francisco, CA: Berrett. Koehler Publishers; 2006: 21 -26. Additional Information (label = reference, position = 7): Davis D. Accuracy of physician self-assessment compared with observed measures of competence. Journal of the American Medical Association. 2006: 296: 1094 -1102. Additional Information (label = reference, position = 8): Carbine D, Finer N, Knodel E, Rich W. Video recording as a means of evaluating neonatal resuscitation performance. Pediatrics. 2000; 106: 654 -658. Additional Information (label = reference, position = 9): Adams K, Scott R, Perkin R, Langga L. Comparison of intubation skills between interfacility transport team members. Pediatric Emergency Care. 2000; 16: 5 -8. Additional Information (label = reference, position = 10): Falck A, Escobedo M, Baillargeon J, et al. Proficiency of pediatric residents in performing neonatal endotracheal intubation. Pediatrics. 2003; 112: 1242 -1247. Etc.

Nested Component • ID: 98765 • Competency: Description: http: //www. example. org/peds_milestones/anatomy_physiology. xml Type: http: //ns. medbiq. org/competencyobject/v 1/ • Competency: Description: http: //www. example. org/peds_milestones. xml Type: http: //ns. medbiq. org/competencyframework/v 1/ • Title: Anatomy and Physiology

Performance levels • Performance Scale Reference: 1 to 4 Position: 1 Score: Single Value: 1 Label: Beginning of Spectrum Indicator (id = pcap_i 1): 2 SD below mean on knowledge test Position: 2 Score: Single Value: 2 Label: Indicator (id = pcap_i 2): 1 SD below mean on knowledge test Position: 3 Score: Single Value: 3 Label: Indicator (id = pcap_i 3): 1 SD above mean on knowledge test Position: 4 Score: Single Value: 4 Label: Indicator (id = pcap_i 4): 2 SD above mean on knowledge test

Pediatrics Component from http: //www. acgme-nas. org/assets/pdf/Milestones/Pediatrics. Milestones. pdf • ID: 90876 • Title: Identify strengths, deficiencies, and limits in one’s knowledge and expertise • Competency: Description: http: //www. example. org/peds_milestones/PBLI 1. xml Type: http: //ns. medbiq. org/competencyobject/v 1/ • Competency: Description: http: //www. example. org/peds_milestones. xml Type: http: //ns. medbiq. org/competencyframework/v 1/

Performance Levels • Performance Scale Reference: 1 to 4 Position: 1 Score: Single Value: 1 Label: Level 1 Indicator (id = pbli_i 1): The learner acknowledges external assessments, but understanding of his performance is superficial and limited to the overall grade or bottom line; has little understanding of how the performance measure relates in a meaningful way to his specific level of Knowledge, Skills and Attitudes (KSA) Additional Information (label=example): During a semiannual review, a learner is unable to describe in any specific terms how he has performed when asked to do so by his mentor. In response, the mentor reviews and interprets the learner’s evaluations and then asks the learner to reflect on the discussion. The learner repeats the language used and recites the overall score/grade without interpretation of further meaning or inference regarding the reported performance assessment.

Nursing (DNP Eval) Example • Title: University of San Diego Hahn School of Nursing and Health Science DNPC 630 Residency DNP NP Student Evaluation • Identifier: – Catalog: URI – Entry: http: //www. sandiego. edu/nursing/DNPC_630_Eval • Performance Scale – 1 to 3 (1 least competent, 3 most competent)

DNP Component • Competency: Description: http: //www. example. org/dnp 630_3. xml Type: http: //ns. medbiq. org/competencyobject/v 1/ • Competency: Description: http: //www. example. org/dnp 630. xml Type: http: //ns. medbiq. org/competencyframework/v 1/ • ID: 5678 • Title: Prepared to practice independently managing previously diagnosed and undiagnosed patients.

Performance Levels • Performance Scale Reference: 1 to 3 Position: 1 Score Single Value: 3 Label: Score Single Value: 2 Label: Indicator (id = pi_i 1): Met Position: 2 Indicator (id = pi_i 2): In progress Position: 3 Score Single Value: 1 Indicator (id = pi_i 3): Not Met Label:

National University of Singapore Example • Title: Standards of Achievement • Identifier: – Catalog: URI – Entry: http: //www. nus. edu. sg/standards 2013 • Performance Scale – 1 to 5 (1 least competent, 5 most competent)

NUS Component • ID: 2013_1 • Title: Standards of Achievement

Performance Levels • Performance Scale Reference: 1 to 5 Position: 1 Score: Single Value: 1 Label: Indicator (id = i 1): Unable to achieve outcome. Position: 2 Score: Single Value: 2 Label: Indicator (id = i 2): Requires a lot of guidance to achieve outcome. Position: 3 Score: Single Value: 3 Label: Indicator (id = i 3): Requires moderate amount of guidance to achieve outcome.

Performance Levels Position: 4 Score: Single Value: 4 Label: Indicator (id = i 4): Able to achieve outcome with little or no guidance (entrustment) Position: 5 Score: Single Value: 5 Indicator (id = i 5): Has the ability to guide/teach others. Label:

Diagnostic Radiology Example (see http: //www. acgme-nas. org/assets/pdf/Milestones/Diagnostic. Radiology. Milestones. pdf ) • Title: The Diagnostic Radiology Milestone Project • Identifier: – Catalogue: URI – Entry: http: //www. acgmenas. org/assets/Diagnostic. Radiology. Milestones • Description: The Milestones are designed only for use in evaluation of resident physicians in the context of their participation in ACGME accredited residency or fellowship programs… • Contributions: – – Role: Author Entity: Kay Vydareny, MD Role: Author Entity: E. Stephen Amis Jr. , MD • Effective date: 2013 -01 -30

Diagnostic Radiology Continued • Supporting Information – This document presents milestones designed for programs to use… • Performance Scale – 1 to 5 (1 least competent, 5 most competent) – 1 to 3 (1 least competent, 3 most competent)

Diagnostic Radiology Component • • ID: 546756 Competency: • Competency: • • Title: Professional Values and Ethics Additional Information (label = Assessment method): End-of-Rotation Global Assessment Additional Information (label = Assessment method): 360 Evaluation/Multirater/Peer Additional Information (label = Assessment method): Simulation/OSCE Additional Information (label = Assessment method): Direct observation and feedback Additional Information (label = Assessment method): Conference attendance logs Additional Information (label = Assessment method): Timeliness in completing institutional and program requirements • • • Description: http: //www. example. org/dr_milestones/PROF 1. xml Type: http: //ns. medbiq. org/competencyobject/v 1/ Description: http: //www. example. org/dr_milestones. xml Type: http: //ns. medbiq. org/competencyframework/v 1/

Diagnostic Radiology Component continued These are suggested educational tools. • Additional Information (label = resource): Teaching and Assessing Professionalism: A Program Director’s Guide by the ABP and APPD. See Chapter 8: Measuring Professionalism, Critical incidents, Peer assessments, Multi-source assessments, Professionalism Mini. Evaluation Exercise (P-MEX) • Additional Information (label = resource): The Professionalism Mini-Evaluation Exercise: A Preliminary Investigation. Richard Cruess, Jodi Herold Mc. Ilroy, Sylvia Cruess, Shiphra Ginsburg, and Yvonne Steinert Acad Med. 2006 Oct; 81(10 Suppl): S 74 -8

Diagnostic Radiology Component continued • Additional Information (label = resource): Text: ABRF Online Modules on Ethics and Professionalism Dcterms: references: Description: https: //www. abronline. org/asp/abrf/

Performance level set • Performance Scale Reference: 1 to 5 Position: 1 Score: Single Value: 1 Label: Level 1 Indicator (id = p 1_1): recognizes the importance and priority of patient care and advocates for patient interests Indicator (id = p 1_2): fulfills work-related responsibilities Indicator (id = p 1_3): is truthful Indicator (id = p 1_4): recognizes personal limitations and seeks help when appropriate Indicator (id = i 1_5): recognizes personal impairment and seeks help when needed Etc…

Position: 2 Score: Single Value: 2 Label: Level 2 Indicator (id = p 2_1): Is an effective health care team member Indicator (id = p 2_2): Demonstrates professional behaviors listed in the second column Position: 3 Score: Single Value: 3 Label: Level 3 Indicator (id = p 3_1): Is an effective health care team leader, promoting primacy of patient welfare, patient autonomy, and social justice Indicator (id = p 3_2): Demonstrates professional behaviors listed in the second column Position: 4 Score: Single Value: 4 Label: Level 4 Indicator (id = p 4_1): Serves as a role model for professional behavior Indicator (id = p 4_2): Demonstrates professional behaviors listed in the second column

Position: 5 Score: Single Value: 5 Label: Level 5 Indicator (id = p 5_1): Participates in local and national organizations to advance professionalism in radiology Indicator (id = p 5_2): Mentors others regarding professionalism and ethics

Psychiatry Example (see http: //www. acgme-nas. org/assets/pdf/Milestones/Psychiatry. Milestones. pdf ) • Title: The Psychiatry Milestone Project • Identifier: – Catalogue: URI – Entry: http: //www. acgme-nas. org/assets/Psychiatry. Milestones • Description: The Milestones are designed only for use in evaluation of resident physicians in the context of their participation in ACGME accredited residency or fellowship programs… • Contributions: – – Role: Author Entity: Sheldon Benjamin, M. D. Role: Author Entity: Adrienne L. Bentman, M. D • Effective date: 2013 -01 -30

Psychiatry Continued • Supporting Information – This document presents milestones designed for programs to use… • Performance Scale – 1 to 5 (1 least competent, 5 most competent) – 1 to 3 (1 least competent, 3 most competent)

Psychiatry Component • ID: 546756 • Competency: Description: http: //www. example. org/psy_milestones/pc 4. xml Type: http: //ns. medbiq. org/competencyobject/v 1/ • Competency: Description: http: //www. example. org/psy_milestones. xml Type: http: //ns. medbiq. org/competencyframework/v 1/ • Title: Psychotherapy Refers to 1) the practice and delivery of psychotherapies, including but not limited to psychodynamic (1), cognitive -behavioral (2), and supportive therapies (3); 2) exposure to couples, family, and group therapies; and 3) integrating psychotherapy with psychopharmacology • Additional Information (label = note, position = 1): Includes the capacity to generate a case formulation, to demonstrate techniques of intervention, and to understand the concepts of resistance/defenses, transference/countertransference

Psychiatry Component (continued) • Additional Information (label = note, position = 2): Includes the capacity to generate a case formulation, to demonstrate techniques of intervention, including behavior change, skills acquisition, and addressing cognitive distortions • Additional Information (label = note, position = 3): Includes the capacity to generate a case formulation, to demonstrate techniques of intervention, and to strengthen the patient’s adaptive defenses, resilience, and social supports

Performance level set • Performance Scale Reference: 1 to 5 Position: 1 Score: Single Value: 1 Label: Level 1 Indicator (id = pc 4_1_1): 1. 1 Accurately identifies patient emotions, particularly sadness, anger, and fear (4) Additional Infomration (label =note): This thread, consisting of the first item at each level, regarding the development of empathy across residency is adapted from the AADPRT Psychotherapy Workgroup’s document “Benchmarks for Psychotherapy Training. ” Indicator (id = pc 4_1_2): 1. 2 Demonstrates interest and curiosity in patient’s story

Position: 2 Score: Single Value: 12 Label: Level 2 Indicator (id = pc 4_2_1): 1. 1 Accurately identifies patient emotions, particularly sadness, anger, and fear (4) Indicator (id = pc 4_2_2): 1. 2 Demonstrates interest and curiosity in patient’s story Etc.

How are use cases addressed? • Map performance frameworks to competencies, including Entrustable Professional Activities – YES (Competency Reference) • Publish a performance framework for use in curriculum planning and assessment – YES (Requires IDs for Components so that assessments may be tied to performance levels) • Import descriptions of performance levels for use in assessment – YES • Reference a Performance Framework for use in assessment – YES (Requires IDs for Components so that assessments may be tied to performance levels) • Describe an individual’s current level of performance for purposes of formative or summative assessment – YES (a score and a reference to a Component should do the trick) • Describe an individual’s level of performance over a longitudinal period for purposes of formative or summative assessment – YES (provided the framework is authored to support that) • Associate assessment evidence with a particular level of performance in a portfolio – Change: assessment evidence to have a score that is interpreted using a particular group of performance levels defined for the relevant competency.

How are use cases addressed? • Define where in a curriculum students are expected to achieve certain levels of performance – YES (will require updates to curriculum inventory, may require id on performance level) • Define what level of performance is required to progress to the next phase or block within a curriculum – YES (will require updates to curriculum inventory, may require id on performance level) • Use in a system that is capable of showing a learner changes in performance over time – YES • View data regarding the performance levels of learners in a program for purposes of program evaluation (external) – YES (would require a spec that allows for the exchange of aggregate data) • View data regarding the performance levels of learners in a program for purposes of program evaluation (internal) – YES • Publish program data for public viewing – YES (would require a spec that allows for the exchange of aggregate data)