Flexibility in f MRI data analysis and standardization
Flexibility in f. MRI data analysis and standardization Chuan-Peng Hu, Ph. D Neuroimaging Center (NIC), Mainz
Why should we care Sources of Flexibility Available solutions
Why do you take this SPM Course? Or Why do you choose to do research?
• Reliability • Validity Source of this picture: Barfod, 2014
Reliability of f. MRI/MRI studies?
More relevant here … How about the reproducibility of data analysis? Computational reproducibility: Same data, different analyzers Nichols et al. , (2017). Best practices in data analysis and sharing in neuroimaging using MRI, doi: 10. 1038/nn. 4500 Peng, (2011). Science. doi: 10. 1126/science. 1213847
Why results from different analyzers differ? (i. e. , low computational reproducibility) Flexibility Reproducibility
Sources of flexibility Station/ Operating system Software/version for data analysis Pipeline of data analysis
Station/ Operating system Software/version for data analysis Pipeline of data analysis
Working station: • PC • MAC • High performance clusters (HPC) Operating Systems: • OSX: 10. 1, 10. 2, …. • Linux: Ubuntu, Cent. OS, Fedora, …. • Windows: win XP; win 7; win 10;
• Working station: different • Operating system: same Glatard, T. , et al. (2015). Frontiers in Neuroinformatics, 9(12). doi: 10. 3389/fninf. 2015. 00012
Sources of Flexibility Station/ Operating system Software (version) Pipeline of data analysis
• Multiple software with multiple versions: • SPM: SPM 96/99, SPM 2, SPM 5, SPM 8, SPM 12; • FSL: 1. 0 (beta, Jun. 2000) ~ 6. 0 (Oct. 2018); • AFNI: 1. 0 (Feb. 1995) ~ 19. 1. 00 ( Apr. 2019); • Freesurfer: v 3. 0 (Mar. 2006) ~ v 6. 0 (Jan. 2017).
Do different versions matter? Yes! “Differences between Free. Surfer v 5. 0. 0 and previous versions were on average 8. 87% (range 1. 3– 64. 0%) (volume) and 2. 86% (1. 1– 7. 7%) (cortical thickness). ” Gronenschild, et al. (2012). PLo. S One, 7(6), e 38234. doi: 10. 1371/journal. pone. 0038234
• Do different software matter? • Yes,
• Do different software matter? • Yes, Fig 1. from: Bowring, A. , Maumet, C. , & Nichols, T. (2018). Bio. Rxiv, 285585. doi: 10. 1101/285585
Sources of Flexibility Station/ Operating system Software (version) Pipeline of data analysis
There are MANY choices in preprocessing 69, 120 Poldrack et al. (2017) doi: 10. 1038/nrn. 2016. 167
In practice, almost every study has its own pipeline Carp, J. (2012). reviewed 241 studies, and found 207 unique combination of analysis procedures. Carp, J. (2012). Neuro. Image, 63(1), 289 -300. doi: 10. 1016/j. neuroimage. 2012. 07. 004
How does the results vary across pipelines? Carp, J. (2012). Frontiers in Neuroscience, 6. doi: 10. 3389/fnins. 2012. 00149
Summary of the flexibility • There is a great degree of flexibility in data analysis; • Flexibility can have (strong) impact on the results and conclusion (false positive!). Station/ Operating system Software (version) Pipeline of data analysis
So, what’s the solution (available)?
Station/ Operating system Docker Software (version) Pipeline of data analysis
Docker • Docker is an open platform for developing, shipping, and running applications. • With Docker, you can manage your infrastructure in the same ways you manage your applications.
Virtue machine:
Docker
Virtue Machine vs. Docker https: //nickjanetakis. com/blog/comparing-virtual-machines-vs-docker-containers
Station/ Operating system Software (version) Docker fmriprep Pipeline of data analysis
What is fmriprep?
Features: • Flexible, adaptive workflow
Esteban et al. , 2019, Nature Methods. Doi: 10. 1038/s 41592 -018 -0235 -4
Features: • Flexible, adaptive workflow • Rich visual reports
• Example of fmriprep: • https: //fmriprep. readthedocs. io/en/stable/_static/sample_report. ht ml
Station/ Operating system Software (version) Pipeline of data analysis Docker fmriprep Open science
Gorgolewski & Poldrack (2016) PLOS Biology. Doi: 10. 1371/journal. pbio. 1002506
• Open Data: https: //openneuro. org/ (Sci. Data) • Open code: Github. com • Papers: Open access
Flexibility in data analysis is just a tip of the iceberg of the reliability and validity problem of f. MRI study Source of this picture: link
Poldrack, et al. , 2017, Nat. Rev. Neurosci. doi: 10. 1038/nrn. 2016. 167
Reflections in f. MRI • Flexibility in data analysis • Low statistical power • Multiple comparisons • Software errors • Insufficient reporting (publication bias) • Lack of direct replication • Software errors • … Poldrack, et al. , 2017, Nat. Rev. Neurosci. doi: 10. 1038/nrn. 2016. 167
Take home message: • Be skeptical when you’re reading papers • Be rigorous when you’re analyzing data, and if possible, be transparent and open.
Thank You! Any questions?
- Slides: 41