Research reliability The problems and the way forward

Research reliability: The problems and the way forward Stephanie Wykstra March 7 -8, 2016 Research Transparency Workshop Maanzoni Lodge, Athi River, Kenya

Part I: Problems

Kick-off discussion Take a few minutes to write down your answer to these two questions. When you read or hear about a new piece of research (i. e. in a publication or a talk), what are things that would make you trust the result? What would make you skeptical? 2) Are there any particular cases where you tried to delve into the research result to investigate it further? What did you do, and how did it go? 1)

Reliable research Research results that we can rely on are incredibly important. Policies, programs and products (such as pharmaceutical drugs) based on research affect many millions of lives. … So the question is: can we confidently rely on research results? What problems undermine our ability to rely on scientific research? And how can we do better?

Research problems: � Publication bias � P-hacking � Non-disclosure � Selective reporting � Failure to replicate � Lack of transparency

Publication Bias “File drawer problem”

Publication Bias Statistically significant results more likely to be published, while null results are buried in the file drawer.

Very few studies report null results Ed Yong, Nature 2012

Publication Bias �Example: Franco et al. (Science 2014) found that of a group of NSF-funded studies run through TESS (time-sharing experiments in the social sciences), 60% of those with positive, statistically significant results had been published versus only 20% of those finding null results.

In social sciences… Franco et al. (Science 2014)

Pop quiz • How would you respond to this person? “I don’t see the problem if we don’t see the null results. We’re interested in finding out what works, and reading about studies where the researchers didn’t find anything significant to report won’t help us much! …. So what’s the problem? ”

P-Hacking • Also called “data fishing, ” “data mining. ” • “If you torture your data long enough, they will confess to anything. ” -Ronald Coase

P-Hacking �Researchers test hypotheses i. e. look for relationships among variables (e. g. schooling, test scores). In particular, a result which shows a result which is statistically significant at p<. 05 is often considered noteworthy and thus more publishable. �This leads to pressure to tinker with specifications (how variables are coded, which outliers are excluded, which subgroups are considered, etc) to find p<. 05.

P-Hacking • “Why Most Research Findings Are False” Ioannidis (PLOS 2005) “The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true. ”

In economics… Brodeur et al (AEJ 2016). Data: 50, 000 tests published in AER, JPE, QJE (2005 -2011) Turner et al. [2008]

In sociology… Gerber and Malhotra 2008 doi: 10. 1177/0049124108318973

In political science… Gerber and Malhotra 2008 doi: 10. 1561/100. 00008024

Selective Reporting �Cherry-picking results for reporting

Selective Reporting XKCD http: //xkcd. com/882/

Selective Reporting �Malhotra, Franco, Simonovits (2015) find that of the studies run through TESS, roughly 60% of papers report fewer outcome variables than are listed in the questionnaire. �If many relevant outcomes aren’t mentioned in the paper (and aren’t listed elsewhere), how can we be confident that the reported results weren’t just noise, rather than true effects?

Replications “Replication” is often used to mean different things. Here are three different activities which replication refers to: (1) Verification and re-analysis: Checking that original data/code can produce published results, as well as going further to check the robustness of the results. (2) Reproduction: Testing whether the results hold up when the study conducted in a very similar way. (3) Extension: Investigating whether the results hold up when the study is conducted in another place, under different conditions, etc. , to test external validity.

Replication Terminology from Michael Clemens 2014

Well-known replications: �Reinhart-Rogoff (2013) Spreadsheet errors found by grad student (Herndon et al 2013). �Deworming debate (2015) Miguel/Kremer (2004) and Aiken/Davey et al. (2015) Big debate within development econ/epidemiology �Reproducibility project in psychology (2015) ~40% of studies successfully reproduced. But big debate this past week! What does it mean for a replication to “fail”? �Begley et al. (2012): Attempt to reproduce “landmark” pre-clinical cancer lab studies at Amgen (6 out of 53 studies reproduced).

Replication incentives • Journals are often reluctant to publish “unoriginal” work, and replications are often considered “unoriginal. ” • Yet simply sharing results of replications on a website (e. g. Political Science Replication Initiative; Psych file drawer) is not ideal either: some argue strongly that replications should be peer-reviewed.

Transparency Peng 2011, Science 334(6060) pp. 1226 -1227

Transparency �Sharing data, code and surveys (along with clear documentation) can allow others to check one’s work.

Transparency �Yet relatively few researchers sharing data: Alsheichk-Ali et al. 2011: Review of 10 first research papers of 2009 published in top 50 journals by impact factor. Of the 500 papers, 351 were subject to a data availability policy of some kind: 59% did not adhere to the policy, most commonly by not publicly depositing the data (73%). Overall, only 47 papers (9%) deposited the full primary raw data online. � Research funders (e. g. government funders such as NSF, NIH, ERSC in the UK, private foundations) are adopting datasharing policies at greater rates. Yet they rarely enforce their policies.

Transparency �And relatively few journals have data-sharing policies: �In 2013, only 18 out of 120 political science journals inspected had data-sharing policies (Gherghinaa and Katsanidoua 2013) and another review found that only 29 out of 141 journals reviewed in economics had policies (Vlaeminck 2013). �Even though having a mandatory policy makes it much more likely that researchers will share data: The rate of data archiving compliance for surveyed journals with the strictest policies, which required data archiving along with data accessibility statements in manuscripts, was nearly 1000 x higher than having no policy. (Vines 2014)

Discussion Discuss in groups of 3 -4: �What changes in how research is practiced and shared do you think would improve the reliability of the research the most? Why?

Part II: Solutions

Meta-research As we have discussed…big problems include: § § Publication bias Data-mining Selective reporting Lack of transparency and inability to replicate

How do we improve? Research transparency! 1. 2. 3. 4. 5. Study registration and pre-analysis plans “Results-neutral publishing” Replications Publish / share all study results Data-sharing: sharing data, code, surveys, readme files

Registering studies • Creating a public record of a study and basic information about the study (e. g. , intervention, outcomes, location, dates). • All IPA required and J-PAL studies strongly encouraged to pre-register studies on the AEA registry (www. socialscienceregistry. org). • We recommend registering prior beginning the intervention.

What is the AEA registry? • A website for registering RCTs in the social sciences, established by the American Economics Association • Established 2013 • ~580 trials registered.

What is the AEA registry?

Other registries:

Reward for registration

Why register a trial? • Combats publication bias by providing public record of the study. • Pre-registration is a long-standing requirement in the medical community (e. g. , for FDA approval, for publication in medical journals) and is increasingly a focus in the social sciences as well. • Clinicaltrials. gov (185 K+ studies registered).

How and when to register? • How? • • Register by making a user account on socialscienceregistry. org. 18 required fields: basic information such as trial title, country, intervention start/end dates, outcomes, etc. Project staff (e. g. research managers) may register a trial for Pis, but PIs must log in to approve and submit the registration. More detailed “how to” guide and video in Box [link] / Sharepoint.

Pre-analysis plans • Pre-analysis plans are more detailed write-ups about the study hypotheses, outcomes, and planned analysis. • • The goal is to combat data-mining by tying the hands of the researcher. AEA registry does NOT require pre-analysis plans (about 1/6 of registered studies have them – you can search by “preanalysis plan” field if interested).

Pre-analysis plans Why have a pre-analysis plan? We identified all large NHLBI supported RCTs between 1970 and 2012 evaluating drugs or dietary supplements for the treatment or prevention of cardiovascular disease. 17 of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after 2000. (Kaplan and Irvin, 2015, PLOS).

Pre-analysis plans Kaplan and Irvin, 2015, PLOS

Share all results (including null results!) • Publication bias leads to a skew towards positive (exciting) results. • Problem: how do we disseminate null results, if journals tend to accept positive results at greater rates? • One solution: reporting all results in a public registry. • For example: the FDA in the US requires that all results be reported on clinicaltrials. gov within one year after trial completion. For social sciences studies, the results could be reported in the place they are registered (e. g. AEA registry). • Alltrials is a group that campaigns to push for all clinical trials to report results.

Results-neutral publishing • Submitting a protocol with details on methodology of study and planned analysis, before carrying out the study. The article is then accepted based on the design, before any data are collected. • “Because the study is accepted in advance, the incentives for authors change from producing the most beautiful story to producing the most accurate one. ” –Chris Chambers, editor of Cortex (OSF citation)

Results-neutral publishing 20 journals have adopted the “registered reports” model so far (list). Examples: • • • • AIMS Neuroscience Attention, Perception & Psychophysics Comparative Political Studies Comprehensive Results in Social Psychology Cortex Drug and Alcohol Dependence e. Life Experimental Psychology Frontiers in Cognition Journal of Business and Psychology Nutrition and Food Science Journal Perspectives on Psychological Science Social Psychology Working, Aging, and Retirement

Replications As valuable as they are, replications are relatively rare. What can the research community do to increase replications being done and shared?

Replication initiatives: • • 3 ie’s replication program Reproducibility projects in psychology and cancer biology Many labs project in psychology Replication wiki in economics

Data-sharing • What? • Making data, code, and other materials publicly available. • "An article. . . in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete set of instructions [and data] which generated the figures. ” -Dr. David Donoho, Stanford stats professor

Data-sharing: why? • Facilitates replication of published results • Re-use of data for further studies and meta-analysis • Promote better quality data/code/metadata

The data-sharing movement • Funders: • • Holdren memo, February 2013 Foundations – Gates, Wellcome Trust (and 15 other funders who signed joint statement), many others • Journals and associations • • Top economics journals e. g. AER, Econometrica, AEJ American Political Science Association - new ethics guidelines for data-sharing • Open science groups advocating for open data • • Berkeley Initiative for Transparency in the Social Sciences Center for Open Science Meta-research institute at Stanford (Ioannidis) YODA at Yale (clinical trials data-sharing portal)

Research transparency at IPA • IPA has adopted new data publication guidelines and policy: sharing data by 3 years following endline data collection. • Data repository created for data from RCTs (including IPA/J-PAL and other collaborating organizations).

Data from RCTs in the social sciences

Data from RCTs in the social sciences

IPA guidelines IPA data publication guidelines [link]

What to share? 1. Datasets • • • Recommended: cleaned, study dataset (PII removed!) Minimally: “publication” dataset – code/data underlying publication. Ideal: start to finish reproducibility (more on this in a minute!) 2. Readme files explaining relation between data and code, as well as any further data documentation. 3. Surveys 4. Study-level metadata

Where to share? Data repositories: • Dataverse (used by IPA and J-PAL) • Can also create a personal dataverse (free!) such as Ted Miguel’s Dataverse. • Figshare • Dryad • ICPSR (not free to deposit data, but curation services are offered). • Many others for specific types of data, see Re 3 data. org

Steps before sharing data/code: 1. 2. • • • 3. 4. 5. Checking usability of data e. g. , that there are variable labels/value codes Ensuring no PII is publicly released: Checking that all direct identifies (names, IDs, etc) are removed. Indirect identifiers such as geospatial data removed. Thinking carefully about whether variables can be recombined to identify participants. Replicating the study, in the sense of checking that data/code produce the published results Checking that Read. Me files provide good instructions for replication Ensuring variable/file/study-level metadata is shared.

What is “fully reproducible research”? • The ideal: sharing “start to finish” data/code to permit replication from initial raw dataset to final tables. • Includes all code (variable construction and cleaning as well). • Key point: sharing isn’t all or nothing: sharing some data/code e. g. underlying the published results … is better than sharing nothing!

Getting data/code into shape early on Preparing data/code early on is crucial, since there are serious limitations in what can be shared and understood later on, if good practices aren’t followed.

Getting data/code into shape e. g. , • Variable naming and labeling • Commenting code • Master do files • Headers/footers • Read. Me files …More detail in IPA’s best practice manual for managing data and code.

Tools for reproducible research • Version control for code using git (session later today) • Dynamic documents using Markdoc (Stata) or Rmarkdown (R) which generate tables in papers directly from the code (session tomorrow) • Open science framework for collaborative workflow (session tomorrow)

Incentives to share data? Badges to reward open practices https: //osf. io/tvyxz

Incentives to share data? Data citation and shifting tenure/promotion committees to rewarding transparency, rather than just publication.

Incentives to share data? Transparency and openness promotion (TOP) guidelines (Science, 2015)

Discussion We have talked about ways of improving the reliability of research (sharing data/code, replicating work, preregistering studies and reporting all results): • What do you think the main obstacles are, which would prevent researchers from doing these things? • What are the most promising ways to improve the reliability of research, in your view? Which are the least promising?