Can an Electronic Health Record EHR Promote Implementation

Can an Electronic Health Record (EHR) Promote Implementation Quality and Fidelity in Children’s Behavioral Health? OR: The Potential for Positive Impact of a Tool Designed to Promote Implementation Quality Probably Depends at Least Somewhat on the Usability and Quality of Implementation of that Tool Eric J. Bruns, Ph. D Alyssa N. Hook, BS Elizabeth Parker, Ph. D Aaron R. Lyon, Ph. D University of Washington / Wraparound Evaluation & Research Team Kelly L. Hyde, Ph. D Fidelity. EHR 4 th Biennial Society for Implementation Research Collaboration September 8, 2017 Wraparound Evaluation & Research Team 2815 Eastlake Avenue East Suite 200 Seattle, WA 98102 www. depts. washington. edu/wrapeval Fidelity. EHR 2100 Calle de la Vuelta, C-202 Santa Fe, NM 87505 www. fidelityehr. com

Acknowledgments • Fidelity. EHR – Founder & CEO Kelly L. Hyde, Ph. D – Mission: • “To support empowerment, engagement and healthy outcomes through innovations in technology for families and communities. ” • This study funded by the National Institute of Mental Health (R 42 -MH 95516; PI Bruns)

Health Information Technologies (HIT) increasingly Function as Implementation Strategies (Lyon & Lewis, 2016) • Some subtypes of HIT support service quality monitoring and can be classified within ‘‘quality management’’ implementation strategies (Powell et al. , 2012, 2015) • HIT simultaneously function as practitioner-facing implementation strategies and client-facing intervention components.

Health Information Technologies (HIT) increasingly Function as Implementation Strategies (Lyon & Lewis, 2016) • Despite their potential, HIT often require their • own strategic implementation supports to be successfully applied (Cohen, 2015; Ruud, 2015) Evaluation of implementation outcomes (e. g. , acceptability, feasibility) for digital implementation strategies is critical to ensure their potential for impact – Can be conceptualized using “hybrid trial” approaches that simultaneously consider effectiveness and implementation (Curran et al. , 2012)

Research Hypothesis: Health Information Technology (HIT) can facilitate efficiency, fidelity, positive outcomes

NIMH Small Business Technology Transfer (STTR) Study Three phases: Phase 1: Program elements of Fidelity. EHR Phase 2: User Experience Testing: Determine if Fidelity. EHR is feasible and usable Phase 3: Determine if transitioning from paper to Fidelity. EHR impacts Wraparound implementation by providers and outcomes for youth and families

Theory of Positive Impact EHR Components • Information management: e. g. , family, team, plan, providers, services, billing • Fidelity support: e. g. , Workflow pane, reminders, alerts, supervisor reports • Standardized assessment: clinical alerts, treatment recommendations • Feedback of information via dashboard reports on fidelity, services, progress, outcomes • Supervisor, manager, administrative reports: e. g. , services, costs, satisfaction, fidelity, outcomes, placements Impact on Staff/Teams • Availability of information • Transparency and efficiency • Better collaboration and teamwork • Adherence to elements of highfidelity Wraparound • More frequent progress review • Decision-making based on objective data • More focused, directive, datainformed supervision • Staff more satisfied and self-efficacious • Admin/managerlevel accountability Paths to Family Outcomes • Goal clarity • Team communication and consensus • Better problemsolving • Greater treatment alliance • Family and team better engaged, hopeful, and satisfied • Shorter selfcorrection cycles • More effective treatment • Reduced staff turnover Outcomes • Families retained in services • Greater social support • Greater progress and reduction in top problems • Reduced youth emotional and behavioral problems • Improved youth functioning • Reduced out of home/ community placement • Reduced costs to systems

Fidelity. EHR Highlighted Features • • Secure, web-based login User friendly interface Contact/Progress Notes, Critical Incident Tracking Secure Messaging and Scheduling Report Builder for program and system decision support High-Fidelity Wraparound-based Plan of Care – Including family vision, team mission, individualized needs statements, strategies linked to needs, etc Workflow pane customized to Wraparound Care Coordination Idiographic Progress Monitoring plus Standardized Assessment Builder

Fidelity. EHR Record Navigation and Workflow

Fidelity. EHR Plan of Care

Fidelity. EHR Core Assessments

Research Aims • • Is Fidelity. EHR feasible, acceptable, and contextually appropriate in the “real world” of wraparound implementation? Comparing care coordinators randomly assigned to EHR vs. wraparound service as usual (SAU), how does Fidelity. EHR affect: – Wraparound supervision? – Wraparound practice? – Teamwork and Alliance? – Wraparound Fidelity? – Parent Satisfaction?

Study Flow (CONSORT Diagram) Randomization at the Care Coordinator (Facilitator) level

Facilitator Demographics EHR n = 18 (%) SAU n = 13 (%) Male 9 (39%) 2 (15%) Female 11 (61%) 11 (85%) White 12 (67%) 10 (77%) African American 5 (28%) 2 (15%) 0 1 (8%) 1 (6%) 0 Hispanic Other No significant differences at baseline

RESULTS: User Experience and EHR Acceptability and Appropriateness

EHR usability ratings in marginal range but slowly increased over time • • • The System Usability Scale (SUS) provides a quick and easy understanding of a user’s subjective rating of a product’s usability 12 facilitators completed the SUS over the course of one year (Site 1) 3 facilitators completed the SUS at 6 months only (Site 2) Acceptable usability Site 2 (n=3; 1 UX assessment) 100 90 80 Marginal usability Low: 50 -62 High: 63 -70 70 60 58. 3 50 40 30 Unacceptable usability 63. 9 20 Site 1 (n=12; 4 waves of UX data) 48. 1 54. 6 50. 6 10 0 Field-Based Testing Sept. 2015 (n=7) Implementation Wave 1 Feb. 2016 (n=12) SOURCE: Bangor, A. , Kortum, P. , & Miller, J. (2009). An empirical evaluation of the system usability scale. Implementation Wave 2 June 2016 (n=12) Implementation Wave 3 Jan. 2017 (n=12)

The distribution of scores indicate a range of opinions on usability • The distribution indicates more than half of the users (61%) rated Fidelity. EHR with Marginal or Acceptable usability after 6 months of use Distribution of SUS Scores for both agencies 7 6 Number of Users 6 5 4 4 4 3 2 2 1 1 0 0 -20 0 21 -30 31 -40 41 -50 51 -60 Marginal usability Low: 50 -62 High: 63 -70 Unacceptable usability SUS Score (n=18) 61 -70 71 -80 81 -100 Acceptable usability

Facilitators newly hired and trained on system report higher usability ratings • Facilitators trained on Fidelity. EHR as part of their onboarding process report higher ratings for usability than facilitators in the research study Acceptable usability 100 90 80 Marginal usability Low: 50 -62 High: 63 -70 70 62. 2 60 56. 8 52. 2 50 40 30 Unacceptable usability 20 10 0 Newly-Hired Facilitators (n=43) EHR Group Facilitators (n=14) SAU Group Facilitators (n=8)

Staff report EHR aligns well with Wraparound service setting • System Acceptability & Appropriateness scale (SAAS) gauges satisfaction, utility, and fit with service context of technology Appropriateness Staff (n=18) rate the degree to which they agree with each item at 6 months 72% Fits with approach to service delivery 67% Fits with treatment modality 39% Comfort interacting with system Acceptability 33% 61% Compatible with service setting Satisfied with content of system 28% 6% 61% 78% Satisfied with ease of use 17% Satisfied with current version 17% 0% 72% 11% 67% 20% Not at all 17% Moderately 40% Extremely 17% 60% 80% 100%

Qualitative feedback: Strengths of the system • • • “Can quickly pull reports” “Can more easily make changes on the fly” “Better direction of where to go in supervision” “More aware of looking at needs and progress” “Great to be able to work remotely” “Families are better at understanding their outcomes” “Overall, love the system compared to the old one… Keeps us focused on particular needs & outcomes, more organized with monitoring” “Tasks flow from strategies which link to needs” “System is overall good… just need to work out kinks”

Qualitative feedback: Needs for system improvement • • • “Contact logs take a lot of clicks… and we use it the most” “Team meeting reminders aren’t consistent” “Core assessments don’t all display in supervision” “Plan of Care is too long – can’t just print one page (e. g. , assessments) … need POC report builder” “Tedious to add and delete strategies” “Can’t sort contact logs by dates”

Qualitative feedback: Change is hard, and transition to EHR must be done strategically • • • “First weeks were hard – challenging to have conflicting answers from supervisors… hard because things weren’t sorted out” “Hard to learn all at once – had a lot of workarounds” “Would have been better to have earlier trainings, and a better user’s manual” “Took a long time to transition… couldn’t breathe til March” “EHR was added to the CAFAS, Suicidal Ideation/BX assessment, assessments asking families at EVERY team meeting how they do and how they feel, Protective Risk Factors Survey, etc. there is too much… we are overwhelmed with requirements” “Starting to get the hang of it but study data will be impacted because we weren’t using the system to its maximum capacity … just trying to get by”

RESULTS: Changes in Practice

Supervisors report small differences in supervision activities by group • • After six months of Fidelity. EHR use, Wraparound Supervisors report how much time they spent on certain activities in supervision with Facilitators Reviewing Plans of Care and Skills Coaching & Training take up approximately one-third of supervision 15. 7% 13. 8% Reviewing Plans of Care 12. 3% Skills Coaching & Training 15. 8% 10. 9% 9. 6% Youth & Family Engagement 10. 1% 9. 6% Natural Support Engagement Reviewing Progress Toward Needs 6. 9% Facilitator Personal Support 5. 8% 10. 1% 8. 2% 7. 7% Administrative Tasks Crisis Assessment/Management 7. 2% 8. 1% Supervisory Relationship 6. 6% 8. 1% 11. 5% 6. 1% 5. 8% Case Conceptualization 5. 0% Facilitator's Professional Role 0% EHR 5% SAU 10% 15% 20%

Supervisors report more time reviewing progress toward needs for EHR staff (p<. 01) • EHR group spends more time reviewing progress toward needs compared to the SAU group Percentage of time spent in supervision 12% 10. 1% 9% 6% 3% 0% Reviewing Progress Toward Needs EHR SAU

Facilitators report shifts in practice throughout the course of EHR use • The Current Assessment Practice Evaluation – Revised (CAPER) was administered to facilitators on a biweekly basis for eight months to assess the degree to which their practice was influenced by reviewing assessment data

Attitudes toward standardized measures improved at 6 months – but only for SAU • Positive Attitudes At 6 -month follow-up, SAU facilitators reported more positive attitudes about using standardized measures and opinions about their reliability and validity compared to EHR group 5 4 3 2. 8 3. 0 3. 4 3. 3 3. 2 PQ PC 3. 4 3. 6 2. 9 2. 8 3. 1 3. 3 2 1 Negative Attitudes 0 BCJ EHR (n = 18) SAU (n = 13) Baseline * Indicates item is significantly different; p<. 05 PQ* 6 Months PC*

Fidelity to Wraparound • • • Caregivers completed the WFI-EZ after four months of Wraparound services No difference found for total fidelity Sig difference in favor of EHR found for Strength/Family Driven 100 90 Fidelity Score 80 70 69. 7 66. 1 67. 8 57. 9 60 62. 4 65. 6 71. 7 73. 6 73. 8 74. 4 74. 7 75. 3 79. 7 * 71. 4 77. 6 69. 4 68. 8 72. 0 50 40 30 20 10 0 Effective Teamwork Natural Supports Needs-Based Outcomes-Based EHR (n=42) SAU (n=23) National Mean Strength & Family Driven Total Fidelity

Facilitator satisfaction with Wraparound practice is high • • Adapted “Clinician Satisfaction Index” measured facilitators’ feelings about usefulness and effectiveness of Wraparound with enrolled families Both groups report high satisfaction; no significant within- or betweengroup differences over time High Satisfaction 70 60 58. 29 62. 15 58. 29 59. 08 50 40 30 20 Low Satisfaction 10 0 EHR (n=17) SAU (n=13) Baseline 6 Months

RESULTS: Impact on Youth & Family Experiences

No significant differences in Caregiver Satisfaction • Caregiver satisfaction with services at approximately the mean for WFI-EZ sites nationally 2. 00 High Satisfaction 1. 2 1. 4 0. 9 1. 00 0. 9 1. 1 1. 0 1. 2 1. 1 1. 2 0. 00 C 1. Satisfied with Wraparound C 2. Satisfied with youth's progress C 3. Family made progress toward needs -1. 00 Low Satisfaction -2. 00 EHR SAU National Mean C 4. More confident about ability to care for youth

Caregivers in both groups report positive working alliance with facilitators • • High Alliance Working Alliance Inventory (WAI) quantifies the degree to which team members work collaboratively and connect emotionally Caregiver ratings very high; no between- or within-group differences 28 24 23. 8 24. 4 26. 0 23. 6 25. 4 26. 6 24. 2 25. 7 20 16 12 8 Low Alliance 4 0 Goal Task EHR (n=42) Bond SAU (n=23) SOURCE: Hanson, W. E. , Curry, K. T. , & Bandalos, D. L. (2002). Reliability generalization of working alliance inventory scale scores. Total Working Alliance

Caregivers report a positive team climate for both groups • • Positive Team Climate The Team Climate Inventory (TCI) assesses team interactions and performance with items such as, “We have a ‘we are in it together’ attitude” Both groups report positive team climate; no between- or within-group differences 20 Max score: 20 17. 1 17. 6 17. 3 18. 0 Max score: 15 15 13. 0 13. 1 12. 9 13. 0 10 5 Negative Team Climate 0 Vision Participative Safety EHR (n=42) Task Orientation SAU (n=23) Support for Innovation

DISCUSSION & IMPLICATIONS

Successful EHR Implementation is becoming a science in and of itself • Studies of successful EHR implementation have consistently recognized the importance of thoughtful planning and training in the implementation process: – Timing training to coincide with implementation – Targeting training to users’ needs – Providing knowledgeable on-site support

Limitations and Study Challenges • Grant timeline required rapid training and • implementation cycles Staff-level randomization within supervisors/ organizations caused disruptions to routines – Supervisors having to supervise differently depending on staff – Staff not able to all support one another • System continued to be improved throughout study based on feedback

Discussion: User Experiences • Staff report EHR aligns with Wraparound • • service setting Marginal usability reported overall User opinions ranged from low to high – Typical patter of “eager adopters” vs “laggers” – Staff saw strengths of the EHR, but also experienced multiple “kinks” during study to be addressed by development team

Discussion: User Experiences • User experiences affected by study enrollment – Facilitators trained on EHR as “part of their job” and/or trained during onboarding more satisfied – Those who had to “change practice” and/or do different things from their colleagues less satisfied • Usability scores increasing over time – System improvements in response to feedback – “On the job learning” made things easier

User experience over time during EHR adoption parallels typical “Big Change” trends Confidence and Usability starts high, typically declines, then increases again Adapted from Domenz, Bob, (2014) Leading Organizations Through ‘Big Change’, ABF Journal, 12(6)

User experience over time during EHR adoption parallels typical “Big Change” trends Confidence and Usability starts high, typically declines, then increases again 100 80 60 40 20 0 63. 9 48. 1 Field-Based Testing Implementation Wave 1 50. 6 Implementation Wave 2 54. 6 Implementation Wave 3

Discussion: Impact on Practice & Implementation • Few significant findings: – EHR group spends more supervision time reviewing progress toward needs – EHR group had marginally better fidelity in one area (Strengths and Family Driven) – Both groups demonstrated significantly improved use of assessment and feedback • Side effect of investment in EHR agency-wide in these sites? – SAU facilitators report more positive opinions about using standardized measures at 6 months

Discussion: Impact on Practice & Implementation • • No differences on caregiver satisfaction, working alliance, team climate Still analyzing youth emotional/behavioral outcomes and fidelity as assessed by document reviews

Implications • • • Rigorous study provided opportunity for substantial improvements Fidelity. EHR System – Staff viewed system as appropriate to wraparound context, but change was hard and improvements were needed Modest but positive shifts in some proximal outcomes (supervision, use of data, fidelity) and lack of negative impact on satisfaction, teamwork, staff job satisfaction could be viewed favorably given the challenges Wraparound-specific EHR in wraparound worthy of continued development and research

Discussion: Next Steps • Complete analysis on youth and family • outcomes More rigorous study with: – Updated Fidelity. EHR system featuring revamped “responsive design” – More time / resources for implementation support – Longer follow-up

For more information: wrapeval@uw. edu www. wrapinfo. org Wraparound Evaluation & Research Team 2815 Eastlake Avenue East Suite 200 Seattle, WA 98102 www. depts. washington. edu/wrapeval info@Fidelity. EHR. com www. Fidelity. EHR. com Fidelity. EHR 2100 Calle de la Vuelta, C-202 Santa Fe, NM 87505 www. fidelityehr. com