Fitting Propensity Models on Electronic Health Records Data

Fitting Propensity Models on Electronic Health Records Data: Lessons Learned

P( treatment | covariates ) outcome treatment study population Covariate Matching Propensity Matching 3

en tre at m s at ur e fe tre at m en t t Traditional vs. EHR data Propensity Scores features ~ ~ ~10 k ~10 ~5 k ~100 => Linear model => Machine learning

Propensity Model Evaluation Framework data propensity scores models fit parameters estimate fit metrics matched data match balance metrics

Post-Match Std. Diff. of Means Evaluating Propensity Score Models model and parameters one covariate Improvement in balance Pre-Match Std. Diff. of Means acceptable bounds on balance

Elastic Net Propensity Models 1 -SE Rule Min CV Error Increasing Regularization (λ) No Regularization

overfitting propensity score overcorrection of imbalances bias in effect estimate?

Problem: propensity score models are evaluated on the same data they are trained on… machine learning tools have enough flexibility to “memorize” the data cross-validating to select hyperparameters does not eliminate this kind of overfitting

A Strategy to Combat Overfitting

Experimenting with these methods in six related observational studies of organ failure during sepsis What is the “causal effect” of a particular organ failure during sepsis care (SOFA organ sub-score in the first 48 hrs) on long-term survival of patients who survive sepsis?

Data 30, 168 inpatient sepsis hospitalizations from Kaiser N. California 2009 -2013 Classified patients as having each type of acute organ dysfunction if they had a Sepsis Organ Failure Assessment (SOFA) subscore ≥ 1 during the first 48 hours of hospitalization. Included only patients who survived their hospitalization in mortality analyses. Mortality based on EHR and state mortality data through April 2015.

Comparing propensity score estimates between the traditional method and employing correction strategies

Propensity models have face validity in terms of important features

Covariate balance after matching on traditionallyestimated propensity vs. on propensity calculated with strategies to mitigate overfitting

Results

Conclusions Nervous system failure is robustly associated with sepsis post-survival mortality Out-of-fold propensity score fitting is a viable approach that improves balance, but we still do not know if it improves estimates

Thanks to: Kaiser Permanete Division of Research NIH K 23 GM 112018