Reproducibility of Research Data OLA HAWAII Mentoring Boot

Reproducibility of Research Data OLA HAWAII Mentoring Boot Camp Jun Panee, Ph. D. September 12, 2018

Advances in science, engineering and all fields of research depend on the reliability of the research record. . . Sustained public trust in the research enterprise requires confidence in the research record and in the processes involved in its ongoing development. - Office of Science and Technology Policy (OSTP)

Reproducibility Project: Cancer Biology https: //osf. io/e 81 xl/

First Reproducibility Project: Cancer Biology Publication in e. LIFE Meta-analyses of each effect. Effect size and 95% confidence interval are presented for Sugahara et al. (2010), this replication attempt (RP: CB), and a random effects metaanalysis to combine the two effects. Sample sizes used in Sugahara et al. (2010) and this replication attempt are reported under the study name. (A) Dox accumulation in tumor tissue of mice treated with 10 mg/kg DOX alone or 10 mg/kg DOX and 4 µmol/kg i. RGD (meta-analysis p=0. 237). (B) Tumor weight of mice treated with 1 mg/kg DOX or 1 mg/kg DOX and 4 µmol/kg i. RGD (meta-analysis p=0. 195). (C) Body weight shift of mice treated with PBS, 1 mg/kg DOX and PBS, or 1 mg/kg DOX and 4 µmol/kg i. RGD (meta -analysis p=0. 229). (D) TUNEL staining from mice treated with 1 mg/kg DOX or 1 mg/kg DOX and 4 µmol/kg i. RGD (meta-analysis p=0. 211). Published: 19 January 2017, ELIFE September 2017, 9 citations 3, 017 views September 2018: 20 citations, Replication Study: Coadministration of a tumor-penetrating peptide enhances the efficacy of cancer drugs Christine Mantis Irawati Kandela Fraser Aird Reproducibility Project: Cancer Biology Is a corresponding author Northwestern University, United States REPLICATION STUDY Jan 19, 2017 Original study: Cells, animals Coadministration of a tumor-penetrating peptide enhances the efficacy of cancer drugs KN Sugahara T Teesalu PP Karmali VR Kotamraju L Agemy DR Greenwald E Ruoslahti (2010) Science 328: 1031– 1035. https: //doi. org/10. 1126/science. 1183057 Pub. Med Google Scholar

Reproducibility Project: Cancer Biology By the end of September 2017, 7 studies published in e. Life. Total reproducible outcomes, n=7 original effect sizes were in the 95% confidence interval of the replication effect size Total non-reproducible outcomes, n=15 Reproducibly rate: 32%

Possible reasons for irreproducibility Microbiome of recipient mice Housing temperature in mouse facilities Cell line drift cell line genetic drift subclonal drift in heterogeneous stable cells Circadian biological responses to therapy Differing compound potency resulting from different stock solutions or differences in peptide synthesis Mouse strain stocks Heterogeneity in individual tumor microenvironments ….

“… the complex system for ensuring the reproducibility of biomedical research is failing and is in need of restructure. ” “With rare exceptions, we have no evidence to suggest that irreproducibility is caused by scientific misconduct. … fraudulent papers are vastly outnumbered by the hundreds of thousands published each year in good faith. ”

Factors contributing to the lack of reproducibility: Poor training in experimental design Emphasis on making provocative statements rather than presenting technical details Do not report basic elements of experimental design Frequently ignored: blinding, randomization, replication, sample-size calculation, effect of sex differences… Withhold details from publication

Preclinical research, especially work that uses animal models, seems to be the area that is currently most susceptible to reproducibility issues. Different animal strains Different lab environments Subtle changes in protocol Coincidental findings Publication bias Over-interpretation

1, 541 citation by September 2016 2, 138 citations by September 2017 2, 892 citations by September 2018 Despite empirical psychologists’ nominal endorsement of a low rate of false-positive (<=0. 05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. A researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. Current standards for disclosing details of data collection and analyses make false positives vastly more likely. It is unacceptably easy to publish “statistically significant” evidence consistent with any hypothesis.

Researcher degrees of freedom: Should more data be collected? Should some observations be excluded? Which conditions should be combined and which ones compared? Which control variables should be considered? Flexibility in reporting subsets of experimental conditions.

3, 928 citations by Sept. 2016 4, 978 citations by Sept. 2017 6, 159 citations by Sept 2018 … in modern research, false findings may be the majority or even the vast majority of published research claims. …the high rate of nonreplication (lack of confirmation) of research discoveries is a consequence of the convenient, yet ill-founded strategy of claiming conclusive research findings solely on the basis of a single study assessed by formal statistical significance, typically for a p-value less than 0. 05. “Negative” research is also very useful. “Negative” is actually a misnomer, and the misinterpretation is widespread.

Corollary 1: The smaller the studies conducted in a scientific field, the less likely the research findings are to be true. Corollary 2: The smaller the effect sizes in a scientific field, the less likely the research findings are to be true. Corollary 3: The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true. Corollary 4: The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true. Corollary 5: The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true. Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.

Reproducibility of experiments using an ALS murine mode September 2018: 376 citations, google scholar September 2018, journal website: Amyotrophic lateral sclerosis (ALS) is a paralytic neurodegenerative disorder ~3% of cases are caused by single point mutation in the gene encoding SOD 1 23 -copy human SOD 1 -G 93 A transgenic mouse is the most widely used murine model >50 publication described therapeutic agents that extend the lifespan of this SOD 1 G 93 A mouse. To date, no therapy has demonstrated a significant impact on the course of human disease.

In the past 5 years we have screened more than 70 drugs in 18, 000 mice across 221 studies, using rigorous and appropriate statistical methodologies. … we observed no statistically significant positive or negative effects for any of the 70 compounds tested, including several previously reported as efficacious.

4 mice per group

Most published studies do not discuss any type of exclusion criteria, gender, or litter issues, and usually rely on standard PCR genotyping that cannot discriminate low- from high copy transgenics.

Conclusion: The majority of published effects are most likely measurements of noise in the distribution of survival means as opposed to actual drug effect. Recommend a minimum study design for this mouse model to best address and manage this inherent noise and to facilitate more significant and reproducible results among all laboratories employing the SOD 1 G 93 A mouse.

THERE IS NO SUCH THING AS A C 57 BL/6 MOUSE! • C. C. Little (the founder of The Jackson Laboratory) initially generated the C 57 BL inbred strain in the 1920’s-1930’s. • Every time a new C 57 BL/6 colony is maintained separately from an existing colony for 20 or more generations, it becomes a new C 57 BL/6 substrain. • The generations are cumulative, so if each of the two separate colonies breed for 10 generations (~2 -3 years), they are 20 generations apart, and different substrains with potentially different phenotypes. acetaminophen-induced liver injury https: //www. jax. org/news-and-insights/jax-blog/2016/june/there-is-no-such-thing-as-a-b 6 -mouse

Change of micronutrients Pyrophosphate (PPi) in laboratory animal chow and drinking water. The chow used in Hawaii (HI) in 2016 (2920 diet) had significantly higher PPi content than the chow used before 2015 (5053 diet). The PPi measurements in Hawaii (HI) were confirmed by a laboratory in Budapest (BP). The standard rodent chow in BP also had significantly lower PPi content than the 2920 diet. The PPi content of the drinking water was negligible.

Summary Low reproducibility of research data is a widespread concern. What can we do to increase data reproducibility: Proper training in experimental design and statistics Full disclosure of materials and methods Proper management of conflict of interest Reduce bias In-house check of data reproducibility before publishing

Discussion?