Build Your Own Forecaster Immunization Project Nathan Bunker

Build Your Own Forecaster Immunization Project Nathan Bunker 03/29/2011

Build Your Own Forecaster 1) 2) 3) 4) 5) 6) 7) 8) Design model Write software Populate meta data Integrate software Identify test cases Test forecaster Make changes Repeat steps 5 – 7

Step 1: Design Model Break vaccines down into antigens Break schedule down to dose states that indicate which vaccination dose is due next Vaccinations given on schedule transition a Dose 2 patient through the states Born Measles MMR Mumps Rubella Protected Dose 3 Dose 1

Initial Design for MMR in 2009 MMR No doses given yet!

Initial Design for PCV-7 in 2009 PCV-7

MMR

Step 2: Write Software Create core forecasting software Should support these activities: forecaster recommendation report reminder/recall activities vaccination rate assessment Forecast should: Evaluate vaccines given as invalid/valid Set dates for next vaccine dose due Identify missed vaccination opportunities

Raw Forecast Output

Step 3: Populate Meta Data Establish vaccination expert team Train expert team on the model Technical team must be involved Expert team should consult: Pink Book ACIP CDC (MMWR, NIP Info) Immunization Action Coalition Expect a Long Process!

Now Put It All Together

Step 4: Integrate Software Immunization Information System DB Core Forecaster Meta Data Test System Core Forecaster Meta Data Post Processing

Not Done Yet

Step 5: Identify Test Cases Assemble test cases Patient DOB Vaccination history Contraindications Expected results Considering rotavirus and PCV Forecaster changes with patient age Possible date types: relative and specific Specific dates must "aged" periodically

Step 6: Test Forecaster Create testing system that can: Create forecast recommendation Provide forecast result explanation Compare expected and actual results Track pass-fail status Have space for notes and comments Support regression testing

Forecast Test System

Step 7: Make Changes Create new test cases as needed Fix problems found in testing Most problems fixed in meta data Some problems fixed in forecaster Deploy changes in testing system Verify problems were fixed Perform regression testing Deploy changes to IIS Step 8: Lather, Rinse, Repeat

Integrating Forecaster with EHR There is no standard way to integrate so we explored several options: Write integrated EHR application module Create integrated EHR application module that supports a web services interface and create forecasting web service engine Create an HL 7 query interface to allow EHR to query forecast Create simple web link in EHR, display forecast results in web browser

Lessons Learned

Lessons Learned Creating a forecasting algorithm itself is relatively simple Creation and maintenance of meta data is a much bigger effort The technical guidance from ACIP covers expected scenarios; the forecaster must cover all potential scenarios

Lessons Learned Consistent process should be developed and applied to deal with gaps in recommendations Some recommendations were difficult to operationalize

Lessons Learned A robust testing system is critical: Ensures accuracy of forecaster Retains shared expert knowledge Facilitates collaboration Helps answer questions from users Helps you sleep at night!

What We Need

What We Need National expert forecast guidance to translate clinical recommendations into implementable computer algorithms National forecaster test case repository More collaboration between other forecaster development projects Common web service interface for getting forecasts from a forecaster