National Center for Injury Prevention and Control Learning

National Center for Injury Prevention and Control Learning from Evaluation Data: Interpreting and Tailoring Findings for Multiple Stakeholders Kari Cruz, MPH Centers for Disease Control and Prevention Kathryn Lowerre, Ph. D, MPH New Mexico Department of Health Kerry Zaleski, MPH UW-Madison School of Medicine and Public Health Wisconsin Department of Health Services Kelly K. Gurka, MPH, Ph. D University of Florida American Evaluation Association Annual Conference November 10, 2017

Rapid Increase in Drug Overdose Death 2000 SOURCE: NCHS Data Visualization Gallery 2015 2

Overdose Prevention in States (OPIS) WA MT ME ND OR MN NV CA WI SD ID NY MI WY IA NE UT AZ IL CO KS OK NM WV KY AR VA NC TN SC MS TX IN MO PA OH VT NH MA CT RI NJ DE MD DC AL GA LA FL AK HI Prevention for States (Pf. S) Data-Driven Prevention Initiative (DDPI) Enhanced Morbidity-Mortality Surveillance (ESOOS) Unfunded State 3

Five Components of OPIS 4

Creating Feedback Loops to Link Evaluation Findings to Policy and Ongoing Program Development Kerry Zaleski, MPH, Evaluation Consultant for Opioid Harm Prevention Program UW-Madison/WI Department of Health Services Wisconsin Department of Health Services Division of Public Health

Sense-making Sessions • Reflection and progress (R&P) reports • Face-to-face meetings to discuss highlights, lessons learned and implications of findings • Develop action plan for sharing feedback with others Wisconsin Department of Health Services Division of Public Health 6

Evaluation of Opioid Pathway Toolkit for Emergency Department (ED) Doctors Purpose: To understand changes in ED doctor attitudes and practices related opioid prescribing guidelines in Wisconsin’s Southeast Region as a result of exposure to toolkit. Wisconsin Department of Health Services Division of Public Health 7

Systemic Barrier Not enough affordable alternatives to manage pain Wisconsin Department of Health Services Division of Public Health 8

CDC and Wisconsin Medical Examining Board (MEB) Guidelines Recommendations for pain management include: • Non-pharmacologic therapies • Non-opioid pharmacologic therapies Wisconsin Department of Health Services Division of Public Health 9

Implications for Work Wisconsin Department of Health Services Division of Public Health 10

System-level Change: Policy Efforts DATA Lack of affordable alternatives makes safe prescribing challenging ACTIONS Health insurance companies and programs begin to reimburse for effective, nonopioid alternatives for pain management OUTCOMES Use of non-opioid alternatives pain stress Wisconsin Department of Health Services Division of Public Health Use of opioid prescriptions Opioid poisoning Opioid use disorder Overdose deaths Other opioid related harms Quality of life 11

Sharing Feedback • R&P meetings • Reports stored in central location • Evaluation briefs • Phone calls • Emails • Newsletters • Presentations • Workshops • Communities of Practice • Data sharing sites (e. g. Share. Point) • Other? Wisconsin Department of Health Services Division of Public Health 12

Lessons Learned • Start with written report, then meet face-toface • Reflection meetings help “connect the dots” • Action plans helps ensure feedback is shared with relevant stakeholders • Quarterly meetings seem to work well • R&P reports are a useful data source for multiple processes and purposes • Tailor communications to meet the needs and interests of your audience Wisconsin Department of Health Services Division of Public Health 13

Steps & Strategies: Working with NM Stakeholders on Pf. S-Funded Programs & Policies Kathryn Lowerre, Ph. D, MPH New Mexico Department of Health Epidemiology and Response Division Prescription Drug Overdose Prevention Program

Barriers & Challenges • Range of stakeholders, including those not traditionally involved in public health • Level of public & often political attention • Values/Frameworks in conflict • Staffing/Funding/Authority • Historical scarcity & silos • Multiple grant organizations, overlapping goals & strategies • Urgency [crisis mode], desire for quick answers Note: These are not new or unique to drug overdose prevention, but often intense variations on familiar themes

Stratification (not just for data analysis…) Evaluation Planning • Which partners/stakeholders are going to need or want what kinds of evaluation information? How are they prepared to contribute to its creation? • When will they need it? And when are they most likely to use it? • In what format(s) will it be accessible and usable by them and their partners/ stakeholders?

New Mexico Pf. S Grant: Evaluation Highlights Prescription Monitoring Program (PMP) Mandate • Jan. 1, 2017: new legislation (SB 263) took effect requiring all NM prescribers of controlled substances to check the PMP according to new licensing board rules Prescriber Feedback Reports (PFRs) • Summer 2016: NM Board of Pharmacy (Bo. P) starts issuing PFRs each quarter to healthcare providers (HCPs) with 20 or more controlled substance patients • Data include how often PMP was checked, % of patients with multiple doctors/pharmacies/Rxs

New Mexico Pf. S Grant: Evaluation Dissemination • Use bi-monthly Advisory Council* meetings to present findings from PFR survey pilot (2 Q 2016), quarterly survey results (3 Q-4 Q 2016) • Slides & meeting minutes; e-mail • Factsheets • Evaluation Brief (pilot, 4 Q), Report (3 Q) • Factsheets to be available on the NM PMP (Bo. P) & DOH websites, & sent to all HCPs receiving PFRs • Policy: Informal interviews, + survey of all PMP users scheduled for 2018 • Work with Bo. P to manage survey, reminders to licensing boards & council members (especially HCPs) • Coordination with contract evaluators for SAMHSA’s Opioid STR grant *The NM Prescription Drug Misuse and Overdose Prevention and Pain Management Advisory Council, with council members appointed by the Governor.

Stakeholder & Partner Map

Map Detail (Advisory Council Current Voting Members)

Map Detail (Advisory Council Non. Voting Participants)

Map Detail (PDOPP & Community Groups)

Lessons (Still Being) Learned • You can’t be all things to all people (+ the corollary: you can’t be everywhere at once) • May feel like Alice in Wonderland with the Red Queen, BUT… • We have a broad range of tools, some low-tech & some high-tech: be flexible! • Make triangulation part of dissemination thinking • Mixed methods are often needed to reach those who need to know about & will use your results

Thanks! Kathryn. lowerre@state. nm. us 505 -476 -3798

Rapidly Informing Quality Improvements of an Insurance-Based Overdose Prevention Program through Evaluation Kelly K. Gurka, MPH, Ph. D 1, 2; Herbert I. Linn, MS 2; Mark K. Garofoli, Pharm. D, MBA; Emily Vasile, MPAff 1 Department of Epidemiology, University of Florida; 2 WVU ICRC; 3 West Virginia University School of Pharmacy

Funding & Disclaimer The evaluation reported in this presentation is supported, in part, by the National Center for Injury Prevention and Control of the Centers for Disease Control and Prevention under CE 15 -1501 to the West Virginia Department of Health and Human Resources (WV DHHR). The content is solely the responsibility of the authors and does not represent the official views of the US Department of Health and Human Services or the WV DHHR.

Program

OPIS Components

Safe and Effective Management of Pain Program § The Safe & Effective Management of Pain Program (SEMPP) is a coordinated care program for providers prescribing chronic opioid therapy. SEMPP provides patient case reviews and authorizes prescriptions on behalf of PEIA and WV Medicaid. The goal of SEMPP is to transform best practices into the standard of care in West Virginia. Urine Drug Screening or Testing PDMP Review Opioid Risk Screening Drug Interaction Review Medication Selection & Dosage SEMPP Patient Case Reviews Patient & Provider Agreements Naloxone Education & Supply Medication Storage & Disposal Improved Function & Reduced Pain Goal

Evaluation Approach

Engagement: SEMPP Meetings Use and Share Lessons Learned § Primary goals § Relationship building § Build rapport § Earn trust § Learning Engage Stakeholders Standards Utility Feasibility Propriety Accuracy Justify Conclusions § Program features § Data sources Gather Credible Evidence Describe the Program Focus Evaluation Design

Engagement: SEMPP Meetings Use and Share Lessons Learned § Primary goals § Relationship building § Build rapport § Earn trust § Learning Standards Utility Feasibility Propriety Accuracy Justify Conclusions § Program features § Data sources § Provide feedback Engage Stakeholders Gather Credible Evidence Describe the Program Focus Evaluation Design

Engagement: SEMPP Meetings Use and Share Lessons Learned § Primary goals § Foster positive relationship § Learning § Ongoing “data” collection § Provide feedback Engage Stakeholders Standards Utility Feasibility Propriety Accuracy Justify Conclusions Gather Credible Evidence Describe the Program Focus Evaluation Design

Engagement: SEMPP Meetings § Scheduled at regular intervals § Agenda § § § Review previous action items Staffing changes Document development Current Activities Evaluation § Data collection § Reporting § Other

Sharing and Follow-Up Use and Share Lessons Learned § Recommendations § Made in real-time during meetings when possible § Documented in “mini” evaluation reports § Meeting minutes § Action items § Tools § Semi-annual reports Engage Stakeholders Standards Utility Feasibility Propriety Accuracy Justify Conclusions Gather Credible Evidence Describe the Program Focus Evaluation Design

Examples § Documentation § Logs § SEMPP Standard Operating Procedures § Training Documents § Indicator § Proposed indicator § Leveraged relationship with CSMP to facilitate data acquisition for indicator

Sharing and Follow-Up § Feedback shared with larger Pf. S team § Creates positive buy-in for evaluation across project strategies Use and Share Lessons Learned Engage Stakeholders Standards Utility Feasibility Propriety Accuracy Justify Conclusions Gather Credible Evidence Describe the Program Focus Evaluation Design

Lessons Learned

Lessons § § Listen (and read between the lines) Ask lots of questions (and stay in the loop) Trust your instincts (and do not make assumptions) Demonstrate value of evaluation (and delay asks)

Contact information for presenters Kari Cruz, MPH Centers for Disease Control and Prevention 770 - 488 -1573 kcruz@cdc. gov Kathryn Lowerre, Ph. D, MPH New Mexico Department of Health 505 -476 -3798 Kathryn. Lowerre@state. nm. us Kerry Zaleski, MPH UW-Madison School of Medicine and Public Health/ Wisconsin Department of Health Services 608 -266 -3411 Kerry. Zaleski@dhs. wisconsin. gov Kelly K. Gurka, MPH, Ph. D Department of Epidemiology University of Florida 352. 273. 5362 kgurka@ufl. edu 40