From Program Theory to Systems Theory Using Logic

From Program Theory to Systems Theory: Using Logic Analysis to Reconceptualize an Evaluation Lori L. Bakken, Ph. D; Jonathan M. Ross, MD; Curtis A. Olson, Ph. D Presented at the 28 th Annual Conference of the American Evaluation Association, Denver, CO, October 16, 2014

Purpose & overview Purpose • To demonstrate how we used a logic analysis to expand a program theory to a systems theory of change Overview • Description of MM&I • Logic Analysis • Systems Model 1

Learning objectives • Critically examine an educational program theory through a systems lens • Use logic analysis as a tool to convert program theory to systems theory • Think about ways that programs contribute to change within a larger system 2

MM&I purpose • Educate for quality • Generate cross-disciplinary dialogue to improve clinical practice and the quality of care 3

MM&I goals • Create an organizational culture for learning that promotes constructive dialogue in a safe environment • Identify areas for practice improvement to be acted upon by work teams • Identify and describe outcomes that include – Programmatic initiatives – Team based improvement – Individual behavior change – Patient-centered care 4

MM&I Facts • • Weekly conference Multidisciplinary Inter-professional Multigenerational Varied learner levels Well-attended (100/week) Participants are there to learn 5

Audience • • Residents Attendings/faculty Medical students Community physicians Retired physicians Nurses Trainees from various programs (eg, physician assistants) 6

How is MM&I planned? • Case-based • Explicit learning objectives • Planners reflect together -What do we want the audience members to learn, feel, change? -What could have been done better? -What systems issues need to be addressed? 7

How is MM&I delivered? • Large conference room (100 -120 participants) • Progressive presentation of case • Interaction with audience, direct and through audience response system – What are you thinking now? – What would you do at this point? – What issues come up for you? • Summary of case • Learning objectives 8

How often do you conduct a Goals of Care conversation as part of Code Status discussion? 45% A. B. C. D. Most of the time Often Rarely These are two different conversations E. I’m not sure 20% 15% e ur co n I’m ve no rs at ts io re Ra to nt os Th es e ar e tw o di ffe re M 9% ns ly te n Of ft he tim e 11% 9

Going forward, my efforts to include a Goals of Care conversation as part of a Code Status discussion are: 43% A. Likely to improve B. Likely to stay the same C. Going to require some study on my part D. A and C 34% 17% Go C an in gt o re qu ire so m Lik el e st yt o ud st a yo n A m he yt d ar t sa pr im yt o Lik el yp m ov e e 6% 10

How was MM&I being evaluated? • Single evaluation form – Check marks, Likert scale • Resident presenter – Matrix of evaluations • ACGME competencies + – Narrative reflection • Faculty planning “committee” 11

But what I really wanted to know. . • Are we making a difference? – In physician behavior – In our learning environment – In our cultural environment – In patient outcomes 12

A confluence of interests • Associate Dean for Continuing Medical Education • Director of the Center for Continuing Education in the Health Sciences • Aligning Education for Quality—American Association of Medical Colleges • Dartmouth-Hitchcock Value Institute for Quality and Safety • Evaluation expertise • Engaged Vice Chair for Education 13

Purposes of the evaluation The overall purposes of this evaluation are to: • Program improvement (including evaluation and assessment) • Assess the impact of MM&I on patient care 14

Use of findings • improve medical education at DHMC, • improve the conference’s effects on clinical practices that can be directly or indirectly related to patient outcomes; and • inform the discipline about continuing medical education’s role and contributions to patient care. 15

Steps in logic analysis Brousselle & Champagne, 2011 1. Create a logic model of the program “in action” 2. Consult the literature and experts to fill gaps and build a conceptual framework 3. Compare the logic model with the information gleaned from Step 2 to the logic model created in Step 1 to design a more accurate theory of change. 16

Case Activities Input Output/Evide nce Short-term Outcomes Case Selected by ACGME criteria Power. Point Slides of Case Extent to which case meets ACGME criteria is high Selects Case Prepares Case Presentation w/guidance of faciliator Resident Delivers PP Presentation Dialogues w/participating HCPs Identify Areas for practice improvement that would be acted upon by work teams Presentation Delivered with sufficient detail, an adequate differential diagnosis, etc. Presentation explored controversial areas, was clinically relevant, etc. Collaborate in a safe educational environment Poses discussion questions Facilitator Selects participants Tracks dialogues among participants Other Resources, e. g. room, support staff, ARS, etc. Responses on ARS Encourages dialogue Tracked Dialogue (# elements present) White board of ideas/thoughts Participate in Discussion Judge Resident's presentation Health Care Providers Judge Case Interactions Among HCPs Completed Evaluation Forms Create an organizational culture for learning that promotes constructive dialogue among HCPs Seek areas for improvement across the ACGGME competencies Attendance Records Self-assess learning Evaluate extent to which conference explored contorversial areas, was clinically relevant, etc. Collaborate in team-based care of medically ill patients Freely express his/her ideas opinions and/or feelings View Presentation Long-term Outcomes Improve Clinical Practice Appreciate the presentation, diagnosis, . . . (i. e. specific learning objectives for today's presentation) Improve Patient Care Are good role models of effective communication 17

Activity • Please take a few minutes to reflect on this logic model from an evaluator’s perspective – What are your observations about the current program theory? – What theories/bodies of literature might you consider to fill gaps and develop a richer conceptual model for the conference? 18

Targeted literature review Studies of MM&I Conferences (7) Diagnostic Reasoning (1) Inter-professional Education (2) Human Factors Sciences and Systems Thinking (6) • Organizational Learning (2) • • 19

Meetings with local experts • Subset of Key Stakeholders • Dartmouth Aligning Education for Quality (Dae 4 Q) Committee • Department of Medicine’s Advisory Council for Education 20

Conceptual framework • • • Formal mechanisms Systems thinking and approaches for MM&I Critical thinking Interprofessional teamwork Environmental elements and processes Training as a “fix” vs a “trigger” Psychologically safe environment Local mechanisms existed Multiple QI initiatives at DHMC 21

Systems model for MM&I 22

Conclusions & Insights • The MM&I Conference is a complex educational intervention situated in a complex environment • Both “operative” and “aspirational” elements in the program theory • Aspects of the process unclear 23

The evaluation design Phase III • Developmental Learning Outcomes & QI Issues and Actions • Formative/Processes of QI Activities that lead to Practice Changes • Impact MM&I contributions to improving pt care 24

Design/systems model alignment 25

Reflections on the process • Value to – Program Director – Evaluators – Other Stakeholders • Lessons learned