REPRODUCIBLE RESEARCH PRACTICES How openness can increase the

REPRODUCIBLE RESEARCH PRACTICES How openness can increase the validity of published results

Scientific Ideals

Scientific Ideals ● Innovative ideas

Scientific Ideals ● Innovative ideas ● Reproducible results

Scientific Ideals ● Innovative ideas ● Reproducible results ● Accumulation of knowledge

What’s the Problem? ● ● ● Low power Questionable research practices Overabundance of positive results Ignoring null results Lack of replication Limitations of NHST Sterling, 1959; Cohen, 1962; Lykken, 1968; Tukey, 1969; Greenwald, 1975; Meehl, 1978; Rosenthal, 1979

How Can We Fix the Problem? ● ● Pre-register studies Increase documentation Open data and materials sharing Change norms and incentive structures Increase openness and transparency in research

Researcher Degrees of Freedom

Researcher Degrees of Freedom All data processing and analytical choices made after seeing and interacting with your data

Researcher Degrees of Freedom All data processing and analytical choices made after seeing and interacting with your data Should more data be collected? Should some observations be excluded? Which conditions should be compared? Which variables should I control for? Which variable should I use as my main DV? What is statistical effect of interest to test my hypothesis?

Researcher Degrees of Freedom ● ● ● All data processing and analytical choices made after seeing and interacting with your data It inflates false positive rates and p-values become uninformative Often feel very reasonable and logical in the moment

Do People Really Do This? Admission rate Defensibility rate

Do People Really Do This? Admission rate Defensibility rate

Why do people do this? ● Perceived norms (Anderson, Martinson & De. Vries, 2007) ● Motivated reasoning(Kunda, 1990) ● Minimal accountability(Lerner & Tetlock, 1999) ● Incentive Structure ● I am busy (Everyone)

Study Pre-registration Specify a priori (before data collection): Research question Hypotheses Study design Materials Data analysis plan

Pre-analysis Plans Target sample: size, population, sampling

Pre-analysis Plans Target sample: size, population, sampling Data cleaning and processing

Pre-analysis Plans Target sample: size, population, sampling Data cleaning and processing Exclusion criterion Specific analyses to be conducted

Preregister Studies Decreases researcher degrees of freedom Holds you accountable to yourself and others If registration made public: Helps replication attempts Decreases the file drawer effect

Confirmatory vs. Exploratory Does this mean we can’t/shouldn’t do exploratory tests?

Confirmatory vs. Exploratory Does this mean we can’t/shouldn’t do exploratory tests? Exploratory and Confirmatory analyses have different purposes

Confirmatory vs. Exploratory Does this mean we can’t/shouldn’t do exploratory tests? Exploratory and Confirmatory analyses have different purposes Clearly delineate between the two

Moving from exploration to confirmation Directly replicate an exploratory study to run confirmatory analyses Randomly split dataset in half to run exploratory and confirmatory analyses Run exploratory analyses, but include robustness checks

Current Workflows ● Vast majority of the scientific workflow obscured ● Hard to reproduce others work, hard to reproduce our own work ● Difficult to accumulate unpublished knowledge or use published results for additional analyses

Increasing Workflow Documentation ● Know how things started and how they evolved (version control) ● Save and annotate syntax ● Clearly name variables and/or create a code book ● Keep clearly labeled final materials, syntax, and raw data in open source* formats ● Know where everything is

Increasing Workflow Documentation In two years, will you remember what you did and why?

Open Data and Materials ● Share data that you can share (obvious exceptions for proprietary or sensitive data, but you can still potentially share subsets of datasets ● Share materials (manipulation text, survey questionnaires, stimuli, etc. ) ● Share analysis scripts ● Ideally, all data and materials located in one place with persistent identifiers

Current Norms and Incentives ● Data and materials rarely shared ● Only published studies ‘count’ ● Much of the scientific process is invisible