Optimal Assessment for Skill Improvement in Surgery OASIS

Optimal Assessment for Skill Improvement in Surgery (OASIS) A Strategic Model Howard W. Francis, M. D. , MBA Associate Professor Dept. Otolaryngology-Head and Neck Surgery and The Dr. Mohan Swami Institute of International Medical Education

Objectives § Trends in Surgical Education § Identified Challenges: Needs § Strategic framework: OASIS § Innovative Solution: LCMA

Foundations

Apprenticeship in Surgical Training • Declarative Knowledge • Procedural Knowledge • Systems Knowledge The Miller Triangle (1990)

Impetus for Change

Johns Hopkins Otolaryngology-Head & Neck Surgery Objective Assessment and Outcomes-based Surgical Education

. Johns Hopkins Otolaryngology Competency Assessment Tools Otology Mastoidectomy Tympanoplasty Head and Neck Surgery Sinus/Plastics Pediatric/General Parotidectomy Endoscopic Sinus Surgery I&D Peritonsillar Abscess Thyroidectomy Rhinoplasty Tracheostomy At Johns Hopkins, we have already developed and implemented competency assessment tools for evaluation of commonly performed otolaryngology procedures Neck Node Excision Pediatric Tracheostomy Rigid Bronchoscopy Submandibular Gland Excision

Pilot-Testing of 2 Assessment Tools • Assessment tools for Mastoidectomy and Endoscopic Sinus Surgery (ESS) were pilot-tested for feasibility, validity, and reliability over 3 years during annual dissection courses.

Objective Structured Assessment of Technical Skills (OSATS) Mastoidectomy Task-Based Checklist Laeeq et al, 2009; Francis et al, 2010

Skill Development Curves Francis et al, 2010

Milestones of Procedural Development Mastoidectomy Francis et al, 2010

Citations • • • • • 1: Laeeq K, Waseem R, Weatherly RA, Reh DD, Lin SY, Lane AP, Ishii M, Cummings CW, Bhatti NI. In-training assessment and predictors of competency in endoscopic sinus surgery. Laryngoscope. 2010 Dec; 120(12): 2540 -5. Pub. Med PMID: 21082748. 2: Ishman SL, Brown DJ, Boss EF, Skinner ML, Tunkel DE, Stavinoha R, Lin SY. Development and pilot testing of an operative competency assessment tool for pediatric direct laryngoscopy and rigid bronchoscopy. Laryngoscope. 2010 Nov; 120(11): 2294 -300. Pub. Med PMID: 20939072. 3: Stewart CM, Masood H, Pandian V, Laeeq K, Akst L, Francis HW, Bhatti NI. Development and pilot testing of an objective structured clinical examination (OSCE) on hoarseness. Laryngoscope. 2010 Nov; 120(11): 2177 -82. Pub. Med PMID: 20824785. 4: Laeeq K, Pandian V, Skinner M, Masood H, Stewart CM, Weatherly R, Cummings CW, Bhatti NI. Learning curve for competency in flexible laryngoscopy. Laryngoscope. 2010 Oct; 120(10): 1950 -3. Pub. Med PMID: 20824777. 5: Lonze BE, Parsikia A, Feyssa EL, Khanmoradi K, Araya VR, Zaki RF, Segev DL, Ortiz JA. Operative start times and complications after liver transplantation. Am J Transplant. 2010 Aug; 10(8): 1842 -9. Pub. Med PMID: 20659090. 6: La. Porte DM, Marker DR, Seyler TM, Mont MA, Frassica FJ. Educational resources for the Orthopedic In-Training Examination. J Surg Educ. 2010 May-Jun; 67(3): 135 -8. Pub. Med PMID: 20630421. 7: Francis HW, Masood H, Laeeq K, Bhatti NI. Defining milestones toward competency in mastoidectomy using a skills assessment paradigm. Laryngoscope. 2010 Jul; 120(7): 1417 -21. Pub. Med PMID: 20578231. 8: Ro K, Cameron JL, Yeh MW. The Hunterian Laboratory of Experimental Medicine. Ann Surg. 2011 May; 253(5): 1042 -8. Pub. Med PMID: 21451394. 9: Francis HW, Masood H, Chaudhry KN, Laeeq K, Carey JP, Della Santina CC, Limb CJ, Niparko JK, Bhatti NI. Objective assessment of mastoidectomy skills in the operating room. Otol Neurotol. 2010 Jul; 31(5): 759 -65. Pub. Med PMID: 20517169. 10: Laeeq K, Infusino S, Lin SY, Reh DD, Ishii M, Kim J, Lane AP, Bhatti NI. Video-based assessment of operative competency in endoscopic sinus surgery. Am J Rhinol Allergy. 2010 May; 24(3): 234 -7. Epub 2009 Dec 16. Pub. Med PMID: 20021742. 11: Basafa E, Sheikholeslami M, Mirbagheri A, Farahmand F, Vossoughi GR. Design and implementation of series elastic actuators for a haptic laparoscopic device. Conf Proc IEEE Eng Med Biol Soc. 2009; 2009: 6054 -7. Pub. Med PMID: 19963665. 12: Galvagno SM Jr, Segal BS. Critical action procedures testing: a novel method for test-enhanced learning. Med Educ. 2009 Dec; 43(12): 1182 -7. Pub. Med PMID: 19930509. 13: Cummings CW. The legacy and obligations of the head and neck surgeon: the 2009 Hayes Martin Lecture. Arch Otolaryngol Head Neck Surg. 2009 Nov; 135(11): 1077 -81. Pub. Med PMID: 19917917. 14: Laeeq K, Bhatti NI, Carey JP, Della Santina CC, Limb CJ, Niparko JK, Minor LB, Francis HW. Pilot testing of an assessment tool for competency in mastoidectomy. Laryngoscope. 2009 Dec; 119(12): 2402 -10. Pub. Med PMID: 19885831. 15: Berkow LC, Greenberg RS, Kan KH, Colantuoni E, Mark LJ, Flint PW, Corridore M, Bhatti N, Heitmiller ES. Need for emergency surgical airway reduced by a comprehensive difficult airway program. Anesth Analg. 2009 Dec; 109(6): 1860 -9. Epub 2009 Aug 27. Pub. Med PMID: 19713264. 16: Haut ER, Chang DC, Hayanga AJ, Efron DT, Haider AH, Cornwell EE 3 rd. Surgeon- and system-based influences on trauma mortality. Arch Surg. 2009 Aug; 144(8): 759 -64. Pub. Med PMID: 19687381. 17: Lin SY, Laeeq K, Ishii M, Kim J, Lane AP, Reh D, Bhatti NI. Development and pilot-testing of a feasible, reliable, and valid operative competency assessment tool for endoscopic sinus surgery. Am J Rhinol Allergy. 2009 May-Jun; 23(3): 354 -9. Pub. Med PMID: 19490815. 18: Pierorazio PM, Allaf ME. Minimally invasive surgical training: challenges and solutions. Urol Oncol. 2009 Mar-Apr; 27(2): 208 -13. Pub. Med PMID: 19285236. 19: Shippey S, Handa VL, Chen TL, Chou B, Bowen CW. Validation of an instrument for evaluation of subcuticular suturing using a plastic tissue model. J Surg Educ. 2009 Jan-Feb; 66(1): 31 -4. Pub. Med PMID: 19215895. 20: Reiley CE, Hager GD. Task versus subtask surgical skill evaluation of robotic minimally invasive surgery. Med Image Comput Assist Interv. 2009; 12(Pt 1): 435 -42. Pub. Med PMID: 20426017.

Major Benefits of Skills Assessment • Formative feedback – Benefits to resident – Benefits to faculty • Refinement of resident training – Guided/mentored learning and practice – Early remediation • Program development

The JH Swami Institute of International Medical Education (SIIME)

Resource Center for Surgical Skills Evaluation and Training SIIME Dr. David Nichols (Director) Dr. Howard Francis Dr. Eun. Mi Park (Ed. D. ) Elizabeth Goodspeed Christopher Matroniano JHI Sanford Wu Charles Kim PUGSOM Dr. Charles Wiener and faculty

Needs Assessment § Challenges in Five Dimensions § Based on literature review and expert opinion § Exploratory & confirmatory interviews with surgical leadership in Malaysia (June ~ October 2011) § Ongoing with visiting international scholars at JHM

Challenge-1. Resident § Ready for independent practice?

Surgery, 146(4): 533, 2009 Concerns: • Operative skill is learned, not innate • Current level of operative experience inadequate • Teaching and learning in the OR is poorly understood and assessed

Challenge-2. Faculty § No formal preparation for teaching § Too busy for effective instruction § Retention of talented faculty in academics

• Faculty members require basic training in educational theory and teaching skills • Longer term interventions, e. g. seminar series, longitudinal programs, fellowships – sustainable change in learning, behavior and organizational culture

Challenge-3. Healthcare Sector. Horizontal § Variability in standards of patient care and patient safety within and between institutions and countries

BMJ, 2012 • Differences found between countries • Greater differences between units than between hospitals

Challenge-4. Healthcare Sector. Vertical § Limited opportunities to advance surgical skills locally- e. g. training abroad needed § Limited local opportunity for leadership development and innovation

Challenge-5. Global § Needed emphasis on public health dimensions of surgical disease

Challenge-5. Global • Surgical Disease: 11% of global Disability-Adjusted Life-Years (DALYs) [Farmer et al, 2008; Debas et al, 2006] • 2015 millennium development goals (MDGs) attention to surgical capacity through improvements in infrastructure and training [Kushner et al, 2010; PLo. S Editorw, 2008] • WHO’s Global Initiative in Emergency and Essential Surgical Care [WHO, 2010]

OASIS Framework (Optimal Assessment for Skill Improvement in Surgery) Facilitates the development of a learning environment that promotes continuous improvement of operative skills within a culture of patient safety and service excellence Residents: Promotion of life-long learning and skill improvement Faculty: Development as teachers, education innovators Professional Boards, MOE, MOH, public: System of accountability in teaching, learning and outcomes

Outcomes / Impact OASIS Roll-out Ensuring Standardiz ation 1. Expansio n In Network Program Pilot 3. Full program 2. Resources Impact 5. Residents 6. Faculty 7. Clinics 8. Upstream 9. Financial

OASIS Faculty Development • Participation: Train the trainer model/CME -OASIS curriculum -OASIS validated skills assessment tools -OASIS validated methods • Implementation: Longitudinal engagement -Faculty engagement and training -Integration into clinical work flow -Quality control • OASIS certification: OASIS Network • Research: Best practices

OASIS Faculty Development Engagement Participation / Actions • OASIS curriculum • OASIS tools • Integrated clinical work flow • Remediation & Coaching Certified OASIS Network Members R E S E A R C H

OASIS Networ k Resources Ongoing Learning Programs (e. g. faculty de v. Programs, CME)

Surgery, 146(4): 533, 2009 Recommendations: • Make operative competency a requirement for board certification • Study and improve teaching in the OR • Develop pedagogic scheme for teaching operations • Develop valid assessment tools for resident performance in the OR • Increase use of simulations in surgical teaching

March, 2012 • Effective July 2012 • Education outcomes reported semiannually • Innovation rewarded

New Foundational Endeavor at JHU RAISE Team (Research & Advancement in Surgical Education) • A grass-roots leadership group: Neurosurgery, Orthopedics, Otolaryngology, Urology • Addressing ACGME requirements for Next Accreditation System • Developed a new framework for residency education

LCMA© Framework Learner Centered Milestone Achievement (LCMA): A Strategic Model for Measuring, Interacting, Monitoring, and Reporting Milestone Achievement Outcomes Eun. Mi Park, Ed. D. Howard Francis, MD, MBA

Learner-Centered Teaching Models A B Learner-centered Faculty-centered

A Staged Model of Development Self-Directed Severe Learner Mismatch Near match Match Involved Learner Mismatch Near match Match Near match Interested Learner Near match Match Near match Mismatch Severe Mismatch Authority, Expert Salesman, Motivator Team Leader Delegator Dependent Learner Grow (1991)

A Model of Milestones Achievement Efficiency / Refinement Error Reduction FACULTY ----- RESIDENT Clarification Identify Discuss Establish Assess Knows Shows (Practice) Does

LCMA Strategic Principle - 1 1. Milestones guide path to proficiency § Transparent expectations § Formative assessment guides progress § Learning needs and style change along milestone continuum

LCMA-1: A Model of Milestones Intentional Assurance For Patient Safety Culture

Operative Competence Technical/ motor skills competence Cognitive competence Other competency domains: - experience - opportunity - clinical judgment - confidence - communication skills - professionalism

LCMA Strategic Principle - 2 2. LCMA is focused on bi-directional interactions § LCMA models progressive learner autonomy with transition from faculty- to learner-initiated exchange (modeled after the skillful teacher and mentor) § Faculty - Learner interactions drive milestone achievement through shared responsibility

LCMA-2. Interactive Responsibilities Faculty Resident 1 3 6 8 10 2 4 5 7 9

LCMA Strategic Principle - 3 3. LCMA focuses on systematic reflective interactions § I: Inform the expected standard & current data § D: Discuss the learner’s next goal and timeline § E: Establish customized instructional approaches § A: Assess next milestone achievement status

LCMA-3: Systematic Interactions

e. g. H 1: Increased Efficiency with Structured Intervention?

Thank you! Howard W. Francis, M. D. , MBA Eun. Mi Park, Ed. D. The Johns Hopkins Dr. Mohan Swami Institute for International Medical Education