Open Science School of Psychology Joost Dessing School

Open Science School of Psychology Joost Dessing School of Psychology – Staff Away Day – 13/06/2019

Replication crisis • Some years ago, colleagues started formally questioning replicability of many published findings • It was realized literature is strive with results that likely do not replicate see https: //replicationindex. com/2015/01/24/questionable-research-practices-definition-detect-and-recommendations-for -better-practices/ https: //www. ncbi. nlm. nih. gov/pubmed/? term=psychological+science+2012+loewenstein School of Psychology – Staff Away Day – 13/06/2019

Incentive problem Nosek et al. (2012) • “The incentives” Best for science Transparent and highquality research, regardless of outcome Best for scientists Producing a lot of ‘good results’ • Publish or perish • Journals not publishing negative findings • Journals not publishing replications • Institutions/Grant agencies avoiding risk taking School of Psychology – Staff Away Day – 13/06/2019

This leads to: Questionable Research Practices • Publication bias • Suppression of negative or complex findings • Arbitrary exclusion data (e. g. , not matching hypothesis) • P-hacking • Fishing: running different analyses and only reporting those that yielded significant effects • Stopping data collection based on statistics • HARKing • Hypothesis formulation After Results are Known (to be presented as a priori hypothesis) • Statistical shortcuts • Low power • Type I errors (due to improper corrections for multiple comparisons) • Data manipulation/creation • Cleaning up figures • Faking participant data (e. g. , Diederik Stapel, others) School of Psychology – Staff Away Day – 13/06/2019

Solutions • Being open/honest about what you will do/did • Need for incentivising good behaviour • Government • Do not focus only on journal impact factors; REF: consider methodological rigour • Granting bodies • Requirement for high power, statistical rigour • Link funding to preregistration • Research institutes • Value methodological rigour over (or at least next to) impact • Journals • Start accepting null results/high-impact replications • Require/enforce methodological rigour • Open data • Mistakes spotted quicker by many eyes • Avoid desk-drawer phenomenon Like many, I used to blame the incentives But it ultimately boils down to human decisions: you choose yourself whether to ‘cave in’ to pressures/incentives School of Psychology – Staff Away Day – 13/06/2019

Data/report availability • Best way to confirm validity of conclusions: check data • Data sharing traditionally promised but not fully followed • Trend: share all data and materials • Publications mostly behind paywalls • Open Access is the new norm • Preprints School of Psychology – Staff Away Day – 13/06/2019

Towards a School Open Science Policy • Embed Open Science practices in all our research • • • Preregister protocols (both confirmatory and exploratory; just be honest) Sharing data/materials: e. g. , Open Science Framework Engaging with preprints Optimizing statistical approaches; staff database; training (R) Open Source software Engage with Many Labs Replication Projects • Embed Open Science practices in all our teaching • • Preach what you practice! Preregistration of thesis protocols Modernize methods/statistics teaching Teach replication explicitly (e. g. , through group project) • Adverts for new positions: “Show commitment to Open Science” School of Psychology – Staff Away Day – 13/06/2019

Towards a School Open Science Policy • Open Science Declaration • Aspirations • Get all staff to sign? • Group activity on Open Science Policy • Discuss/list benefits • Discuss/list fears/challenges Research / Teaching • Short anonymous survey School of Psychology – Staff Away Day – 13/06/2019