Generalization of Machine Learning Approaches to Identify Notifiable

Generalization of Machine Learning Approaches to Identify Notifiable Conditions from a Statewide Health Information Exchange Machine Learning and Predictive Modeling – Clinical Research Informatics Track VS 02 Gregory Dexter Regenstrief Institute Twitter: @Regenstrief #IS 20

Disclosure My coauthors and I have no relevant relationships with commercial interests to disclose. 2020 Informatics Summit | amia. org 2

Healthcare Needs Generalizable ML • Machine learning has demonstrated potential across healthcare • Prognosis: Heart attack and stroke • Diagnosis: Autism • Treatment: ICU ventilation management • No ready methods to widely deploy ML in healthcare • Technical and legal barriers exist to sharing health data • Models built at one institution not effective at others • This issue has been recognized in the literature • Most current studies do not formally test for generalizability 2020 Informatics Summit | amia. org 3

Overview of Methods • Utilized health information exchange data • Classified lab test results using ML • Detect positive laboratory results for Syphilis, Salmonella, and Histoplasmosis • Determined how well model architecture generalizes • Train overall models (baseline) • Train holdout models • Determined what features of the dataset affect generalizability 2019 Informatics Summit | amia. org 4

Learning Task – ML Use Case • Notifiable Condition Detection (NCD) • Syphilis • Salmonella • Syphilis • Classification of plaintext laboratory test results • Positive vs Not-Positive 2019 Informatics Summit | amia. org 5

Data Source • INPC health exchange contains >100 Hospitals, 32 Health systems • Laboratory extracted from HL 7 messages • Collected form 2016 -2017 • Relevant messages were isolated using ICD-10 codes and keywords • Manually sorted as Positive vs Not-Positive 2019 Informatics Summit | amia. org 6

Dataset Size 2019 Informatics Summit | amia. org 7

Training Overall Models Gives baseline performance of ML architecture 1. Vectorized data using bag of words 2. Applied random forest algorithm 3. Evaluated architecture by 80 -20% train test split 2019 Informatics Summit | amia. org 8

Performance of Overall Models • ML models performed well on all diseases • F 1 -Scores >= 0. 90 • Good performance with simplistic model architecture 2019 Informatics Summit | amia. org 9

Holdout Method • Emulates situation where ML model is applied to non-HIE hospital • Take one lab system as test, Use all others as train • To ensure significant results, only look at lab systems with >= 28 associated datapoints 2019 Informatics Summit | amia. org 10

Performance of Holdout Models • Table shows summary of holdout model performances • Low minimum F 1 -Score for all three diseases • Reduced median F 1 -Score for Syphilis and Salmonella 2019 Informatics Summit | amia. org 11

Causes: Disease Prevalence vs F 1 -Score • No strong relationship between prevalence with F 1 -score • Size vs F 1 -score also does not show trend 2019 Informatics Summit | amia. org 12

Causes: Clustering Experiment • Hypothesis: Poor generalizability caused by different feature vector distributions • Test by looking at distribution of clusters • Check pairs of laboratory systems • Use Χ 2 to test for difference in clusters 2019 Informatics Summit | amia. org 13

Clustering Results • Few lab pairs with non-significant difference in clusters • Cluster sizes were relatively balanced • Correctness of results supported by fewer unique non-significant labs 2019 Informatics Summit | amia. org 14

Discussion • Standard methods of evaluating ML models do not predict generalizability • Large reductions in model performance seen when generalizing • The ability of a model to generalize can not be predicted from summary statistics • Vector representation is likely important for generalizability • More work is needed to determine relationship between feature vectors and generalizability • Association between clustering and holdout model performance • Relationship can guide development of methods to improve generalizability 2019 Informatics Summit | amia. org 15

Thank you! Email me at: gdexter@purdue. edu