Eric Holmboe MD 56 th ASH Annual Meeting

Eric Holmboe, MD 56 th ASH Annual Meeting Disclosure Statement • Employment: ACGME Discussion of off-label drug use: not applicable

Accreditation Council for Graduate Medical Education What Constitutes “Good” Assessment and Milestones-based Assessment © 2014 Accreditation Council for Graduate Medical Education

The Assessment “System” Unit of Analysis: Program Residents FB Assessments within Program: • Direct observations Qual/Quant • Audit and “Data” D performance data Synthesis: • Multi-source FB Committee • Simulation • ITExam Faculty, PDs and others J U D G E M E N T FB Milestones and EPAs as Guiding Framework and Blueprint Accreditation D FB Certification and Credentialing Unit of Analysis: Individual P U B L I C

Process vs. Outcome Approach Educational Program Variable Structure/Process Competency-based Driving force: process Teacher Learner Path of learning Hierarchical (Teacher→student) Non-hierarchical (Teacher↔student) Responsibility: content Teacher Student and Teacher Goal of educ. encounter Knowledge acquisition Knowledge application Typical assessment tool Single subject measure Multiple objective measures Assessment tool Proxy Authentic (mimics real tasks of profession) Setting for evaluation Removed (gestalt) Direct observation Evaluation Norm-referenced Criterion-referenced Timing of assessment Emphasis on summative Emphasis on formative Program completion Fixed time Variable time Carracchio, et al. 2002.

Measurement Tools: Criteria Cees van der Vleuten’s utility index: • Utility = V x R x A x EI x CE/Context* • Where: V = validity R = reliability A = acceptability E = educational impact C = cost effectiveness *Context = ∑ Clinical Microsystems

Criteria for “Good” Assessment 1 • • • Validity or Coherence Reproducibility or Consistency Equivalence Feasibility Educational effect • Learning that occurs in preparation for an assessment (e. g. certification exam; MRCP) • Catalytic effect • Assessment resulting in feedback that “drives future learning forward. ” • Acceptability 1 Ottawa Conference Working Group 2010

Assessing for the Desired Outcome Work-based assessment is mostly accomplished Does through the (action) observations and questions of faculty, Shows How team members, peers and other co-workers (performance) Knows How (competence) Knows (knowledge) Performance in Practice//Multi-source feedback/ Direct Observation Standardized Patients/Simulation Diagnostic Reasoning using clinical vignettes or CSR Multiple choice Questions

Milestones and Work-based Assessment § The predominant approach in CBME is workbased assessment § Milestones can serve as a framework for shared mental models and to inform observation and assessment by faculty © 2014 Accreditation Council for Graduate Medical Education