Improving Rigor and Reproducibility of Scientific Research Tim
Improving Rigor and Reproducibility of Scientific Research Tim Errington Center for Open Science http: //cos. io/
Mission: Improve openness, integrity, and reproducibility of scientific research
Evidence to encourage Metascience Incentives to embrace Community Training to enact Technology to enable Infrastructure
Improving the alignment between scientific values and scientific practices
Norms Counternorms Communality Secrecy Open sharing Closed Universalism Particularlism Evaluate research on own merit Evaluate research by reputation Disinterestedness Self-interestedness Motivated by knowledge and discovery Treat science as a competition Organized skepticism Organized dogmatism Consider all new evidence, even against one’s prior work Invest career promoting one’s own theories, findings Quality Quantity Merton, 1942;
Anderson, Martinson, & De. Vries, 2007
Barriers 1. Perceived norms (Anderson, Martinson, & De. Vries, 2007) 2. Motivated reasoning (Kunda, 1990) 3. Minimal accountability (Lerner & Tetlock, 1999) 4. Concrete rewards beat abstract principles (Trope & Liberman, 2010) 5. I am busy (Me & You, 2017)
Scientific Ideals ● Innovative ideas ● Reproducible results ● Accumulation of knowledge Central Features of Science ● Transparency ● Reproducibility
What is reproducibility? • Computation Reproducibility: – If we took your data and code/analysis scripts and reran it, we can reproduce the numbers/graphs in your paper • Empirical Reproducibility: – We have enough information to rerun the experiment or survey the way it was originally conducted • Replicability: – We use your exact methods and analyses, but collect new data, and we get the same results
Direct Replication ● Same procedure on new samples ● Tests the current beliefs to produce a finding ● Establish that a finding is reproducible ● Does not guarantee validity Conceptual Replication ● Different procedure ● Test the same hypothesis ● Evidence to converge on an explanation for a finding ● Does not guarantee reproducibility
Why should you care? • To increase the efficiency of your own work – Hard to build off our own work, or work of others • We may not have the knowledge we think we have – Hard to even check this if reproducibility low
Academic Life Science Research Process GBSI, 2013
Read more: osf. io/8 mpji
Problems Flexibility in analysis Selective reporting Ignoring nulls Lack of replication Sterling, 1959; Cohen, 1962; Lykken, 1968; Tukey, 1969; Greenwald, 1975; Meehl, 1978; Rosenthal, 1979
Researcher Degrees of Freedom All data processing and analytical choices made after seeing and interacting with your data “Does X affect Y? ” Should I collect more data? Exclude outliers? Control for expression? Median or mean? Jorge Luis Borges; Gelman and Loken
Figure created by 538 Silberzahn et al. , 2015
http: //compare-trials. org
Challenges Franco, 2014
Button et al. , 2013, Nature Reviews Neuroscience
Fanelli, 2010,
1) Were experiments performed blinded? 2) Were basic experiments repeated? 3) Were all the results presented? 4) Were there positive and negative controls? 5) Were reagents validated? 6) Were statistical tests appropriate?
Problems Unrecognized experimental variables Disorganization Poor documentation of methodology Misinterpretation of noise as an indication of a positive finding Loss of Materials and Data Infrequent Sharing
How quality control could save your science Baker, 2016
How quality control could save your science Baker, 2016
Challenges in sharing Vines, 2014
Unique identification of research resources in the biomedical literature Vasilevsky, 2013
Resource Identification Initiative
RRIDs
Bio. Sharing. org
Why you might want to share • Journal/Funder mandates • Increase impact of work – Other can easily replicate/understand your work – Others can reuse/build on your data/analysis/etc • Recognition of good research practices
Sólymos, P. & Fehér, Z. (2008): http: //biogeography. blogspot. com. au/2008/04/ mefa-package-tool-for-reproducible-data. html Peng, R. (2011): http: //sciencemag. org/content/334/6060/1226
Two Modes of Research Context of Discovery Exploration Data contingent Hypothesis generating Context of Justification Confirmation Data independent Hypothesis testing
A reader quick, keen, and leery Did wonder, ponder, and query When results clean and tight Fit predictions just right If the data preceded theory Anonymous, quoted from Kerr (1998)
Preregistration Purposes 1. Discoverability: Study exists 2. Interpretability: Distinguish exploratory and confirmatory approaches Why needed? Mistaking exploratory as confirmatory increases publishability and decreases credibility of results
Solution: Pre-registration • Before data is collected, specify – The what of the study/experiment • • Research question Population Primary outcome General design – Pre-analysis plan • • • Information on exact analysis that will be conducted Sample size Data processing and cleaning procedures Exclusion criterion Statistical Analyses ● Registered in a read-only format and time-stamped
https: //cos. io/prereg
Discrepancies in drug sensitivity Weinstein & Lorenzi, 2013
Science is knowledge obtained by repeated experiment or observation Vaux, et al. , 2012
Experimental design is the art of varying one factor at a time while controlling others Vaux, et al. , 2012
A conclusion can only apply to the population from which you took the random sample of independent measurements Vaux, et al. , 2012
Although cannot support inference, replicates do provide important quality controls of the experiments Vaux, et al. , 2012
Next class: • Training on the going through documentation and transparency • Use Open Science Framework (https: //osf. io) – Watch intro video – Create an account/log-in with UVA account • Come with an experiment about to start, or just started.
Find this presentation at https: //osf. io/ Questions: tim@cos. io
Some additional reading: • • • http: //jpet. aspetjournals. org/content/351/1/200 https: //elifesciences. org/content/3/e 04333 https: //elifesciences. org/content/6/e 23383 https: //peerj. com/articles/148/ http: //www. nature. com/nrn/journal/v 14/n 5/full/nrn 3475. html http: //circres. ahajournals. org/content/116/1/116 http: //journals. sagepub. com/doi/full/10. 1177/1745691612463078 http: //www. nature. com/nrclinonc/journal/v 10/n 12/abs/nrclinonc. 2013. 1 71. html http: //jamanetwork. com/journals/jama/fullarticle/2603417 http: //sciencemag. org/content/355/6325/584. full http: //www. nature. com/news/reproducibility-1. 17552 http: //www. nature. com/collections/qghhqm/
- Slides: 45