Reproducibility Project Cancer Biology Barriers to Replicability in

Reproducibility Project: Cancer Biology Barriers to Replicability in the Process of Research Tim Errington Center for Open Science

Begley & Ellis, 2012

RP: P Key Features ● Systematic selection ● Preregistration ● Completely open ● Materials and feedback from original authors Open Science Collaboration, 2015

Reproducibility Project: Cancer Biology Systematic sampling of 51 high-impact preclinical cancer biology papers Completely open prereg, process, data, materials: https: //osf. io/e 81 xl/ Registered Reports with e. Life Stage 1 review of protocols: Precommitment by researchers, reviewers, editor Stage 2 review of results: Confirming adherence to precommitments Primary Outcomes Meta-analytic summary of statistical outcomes (not shown today)

Sampling Design 1 st Review Data Collection 2 nd Review Published Eligible if published 2010 -2012 and had “cancer biology” keywords Excluded genomics, proteomics, high-throughput assays Selected highest impact by citations and readership 51 papers included 33 Nature family, 11 Cell family, 4 Science family, 3 others

Sampling Design 1 st Review Data Collection 2 nd Review Top-Level Outcome Started Design Phase: 51 papers, 197 experiments Mean = 3. 9, STD = 1. 9; Range 1 to 11 Finished Design Phase: 36 papers, 114 experiments Mean = 3. 2, STD = 1. 6; Range 1 to 8 Published

Sampling Design 1 st Review Data Collection 2 nd Review Published Question: For how many papers/experiments was all data available in the paper or a repository? 0 of 51 papers (0%) 3 of 197 experiments (2%)

Sampling Design 1 st Review Data Collection 2 nd Review Published Question: After contacting authors, how much original data was shared? Papers (liberal) Experiments No data sharing 28 (55%) 139 (71%) Summary data (N, std) 5 (10%) 24 (12%) Raw data 15 (29%) 31 (16%) 3 (2%) Open data or data in paper

Sampling Design 1 st Review Data Collection 2 nd Review Published Question: For how many papers/experiments could we design a full protocol based on the paper and no input from the original author(s)? 0 of 51 papers (0%) 0 of 197 experiments (0%)

Sampling Design Clarifications Needed 1 st Review Data Collection 2 nd Review Published Papers (Avg) Experiments No clarifications 0 (0%) Few clarifications 5 (10%) 15 (8%) Some clarifications 14 (27%) 78 (40%) Moderate clarifications 20 (39%) 62 (31%) Strong clarifications 10 (20%) 33 (17%) 2 (4%) 9 (5%) Extreme clarifications

Sampling Design 1 st Review Original Author Helpfulness Data Collection 2 nd Review Published Papers (Avg) Experiments Not at all helpful 19 (37%) 75 (38%) Minimally helpful 4 (8%) 15 (8%) Some helpful 2 (4%) 17 (9%) Moderately helpful 4 (8%) 16 (8%) Very helpful 8 (16%) 26 (13%) Extremely helpful 14 (27%) 48 (24%)

Sampling Design 1 st Review Data Collection 2 nd Review Published

Sampling Design 1 st Review Data Collection 2 nd Review Resource Challenges in Assessing Replicability ● Accessing materials ● Recreating materials ● Design adjustments ● Unexpected events in data collection Illustration: Estimated cost per paper over time 1. Start: $25, 000 2. Stage 1 review: $36, 429 3. Data collection: $42, 016 4. Final: $54, 442 Published

Sampling Design 1 st Review Data Collection 2 nd Review Published Question: How did the 32 Registered Reports fare in the Stage 1 review process? Accepted with no revisions 0 Accepted after minor revisions 9 Accepted after major revisions 20 Not resubmitted based on resource projections 1 Rejected 2

Sampling Design 1 st Review Data Collection 2 nd Review Published Top-Level Outcome Started Collection Phase: 29 papers, 89 experiments Finished Collection Phase: 23 papers, 52 experiments ● 6 papers = no experiments ● 5 papers = some experiments ● 18 papers = all experiments

Sampling Design Modifications needed 1 st Review Data Collection 2 nd Review Published Papers (Avg) Experiments No modifications 3 (11%) 23 (30%) Few modifications 5 (18%) 9 (12%) Some modifications 10 (36%) 18 (24%) Moderate modifications 6 (21%) 10 (13%) Strong modifications 2 (7%) 6 (8%) Extreme modifications 2 (7%) 10 (13%)

Sampling Design 1 st Review Modifications Implemented Data Collection 2 nd Review Published Papers (Avg) Experiments Not implemented 3 (11%) 12 (16%) Little implemented 2 (7%) 4 (5%) Some implemented 1 (4%) 4 (5%) Moderately implemented 2 (7%) 3 (4%) Mostly implemented 2 (7%) 2 (3%) 18 (64%) 51 (67%) Completely implemented

Sampling Design 1 st Review Data Collection 2 nd Review Case studies: 1) Replication of Delmore et al. , 2011 (Cell). Published replication: Aird et al. , 2017 (e. Life) 1) Replication of Kang et al. , 2011(Nature). Published replication: TBD Published

Sampling Design 1 st Review Data Collection 2 nd Review Published Question: How did the 18 Final Reports fare in the Stage 2 review process? Accepted with no revisions 0 Accepted after minor revisions 4 Accepted after major revisions 10 Still in revisions 3 Rejected 1

Sampling Design 1 st Review Data Collection 2 nd Review Published Final summary Started: 51 papers, 197 experiments Finished Papers: 23 (some) / 18 (all) / 17 (accepted*) Experiments: 52 total Coming next: ● Full report on process outcomes ● Full report on meta-analysis of statistical outcomes

Conclusion Testing replicability is impeded by process factors… ● Lack of transparency and accessibility of methodology ● Lack of sharing of original data ● Lack of communication for obtaining needed information ● Unexpected challenges with protocols during data collection ● Resource challenges (cost, time, uncertainty) inflated by all the above Questions? tim@cos. io or nosek@cos. io