OHDSI updates Martijn Schuemie Method Evaluation Method Evaluation
- Slides: 13
OHDSI updates Martijn Schuemie
Method Evaluation
Method Evaluation Time line • OHDSI Method Evaluation Task Force • Two posters at OHDSI Symposium 2017 • Just submitted paper to Harvard Data Science Review
Evaluation Gold standard • 200 real negative controls • 600 synthetic positive controls Methods • Cohort. Method • Self. Controlled. Cohort • Case. Control • Case. Crossover • Self. Controlled. Case. Series Metrics • Area under the ROC • Coverage of the 95% confidence interval • Mean precision • Mean Squared Error • Type I and II errors • Fraction non-estimable Databases • IBM Market. Scan CCAE • IBM Market. Scan MDCR • JMDC • Optum Pan. Ther
Results http: //data. ohdsi. org/Method. Eval. Viewer/ Key findings: • All have poor coverage / type I error (without calibration) • IPTW almost as bad as no adjustment • Pan. Ther biased for most methods due to observation time issues • Best methods (highest AUC + highest precision after calibration) are self-controlled methods (specifically: MSCCS)
Methods Library
Methods Library • • A core set of OHDSI R packages For population-level estimation and patient-level prediction Moving deployment to CRAN Production-level – Unit tested – Evaluated in the Methods Evaluation – Well-documented https: //ohdsi. github. io/Methods. Library/
Book of OHDSI
Book of OHDSI • Central knowledge repository – – – OHDSI community CDM + Vocabulary + conventions (THEMIS) ATLAS + R packages Characterization + Population-level estimation + Patient-level prediction Network studies • Can be used in classroom • Community developed through Git. Hub + bookdown • Could print in paper later https: //github. com/OHDSI/The. Book. Of. Ohdsi
Network studies How to share study code
Sharing study code What is shared? • R package per study (with dependencies) How is it shared? • Previously: Study. Protocol. Sandbox + Study. Protocols • Now: Ohdsi. Studies using submodules https: //github. com/OHDSI/Ohdsi. Studies
Topics for next meeting • ?
Next workgroup meeting Western hemisphere: January 17 • 6 pm Central European time • 12 pm New York • 9 am Los Angeles / Stanford Eastern hemisphere: January 9 • 3 pm Hong Kong / Taiwan • 4 pm South Korea • 5: 30 pm Adelaide • 8 am Central European time • 7 am UK time http: //www. ohdsi. org/web/wiki/doku. php? id=projects: workgroups: est-methods
