Detection and Analysis Perspectives of Both Data Monitors

Detection and Analysis Perspectives of Both Data Monitors and Algorithm Developers

Data Monitors • Turbulence – Turnover – Training • • Varied skill sets Limited understanding of algorithms Overworked and few Feedback to providers critical

Developers • Interaction and feedback from users is important – Iteration • Critical for developers to know users problems • Limited understanding of public health operations – Expense of false positive to users

Available Detection Methods • Bio. Sense, Essence, RODS, Red Bat… – Cusum, Smart Scores, RLS, EWMA, Wavelet, … • Spatial scan statistics, Sat. Scan, zipcode • Multiple detection algorithms – how many are flagging? • Multiple detections corroborate problem • Real-time vs. batch?

Real-time Detection • Data is unsettled when it arrives – Pressure for real-time may exacerbate existing problems • • Is it sustainable? Who will monitor it? How valuable is it? If not everyday – can we do it in a crisis?

Weaknesses of Detection Algorithms • Cusum, EWMA – most widely used – Control chart has many assumptions • Normally distributed • Stationary assumption – Are assumptions being met? • Method must match data – Getting false alarms that exceed rate that would be expected may signal disconnect between data and algorithm

Challenges (to name a few) • Syndrome categorization – More? – Fewer? – Subsetting? • Statistical significance does not equal public health significance – “False” alarms

Issues for Users • • False positives Disconnect between developers and users Difficulty evaluating what is a real alarm Data quality issues What user needs are being supported? What are other uses for data? Overall evaluation of syndromic surveillance utility

Possible Approaches to Improving Detection • Explanation to user of why alarming – Pop-up? • Text-strings from physicians • Phased alerting system – Yelling, Anomalies, Alert • Improving alert qualification (RODS in Ohio)

Possible Approaches to Improving Detection (continued) • Better pre-processing of data • Better post-processing of data – De-duplication of data, etc. • Failure analysis of false alarms – What are major contributors?

Needs • Refined data • Improved algorithms • Human expert required to make determination – Domain knowledge – Local knowledge – Statistical analysis is only a tool • Good relationships among users – Federal, state, local, facility levels

Summary • Systems: Alarm too often • Users: May become discouraged • Statistical challenges – Detection methods up to it? – Data problems? • Solutions: – Better data, better processing, more money