Culture Change Toward Openness and Reproducibility Brian Nosek

Culture Change Toward Openness and Reproducibility Brian Nosek University of Virginia -- Center for Open Science http: //briannosek. com/ -- http: //cos. io/

Norms Counternorms Communality Secrecy Open sharing Closed

Norms Counternorms Communality Secrecy Open sharing Closed Universalism Particularlism Evaluate research on own merit Evaluate research by reputation

Norms Counternorms Communality Secrecy Open sharing Closed Universalism Particularlism Evaluate research on own merit Evaluate research by reputation Disinterestedness Self-interestedness Motivated by knowledge and discovery Treat science as a competition

Norms Counternorms Communality Secrecy Open sharing Closed Universalism Particularlism Evaluate research on own merit Evaluate research by reputation Disinterestedness Self-interestedness Motivated by knowledge and discovery Treat science as a competition Organized skepticism Organized dogmatism Consider all new evidence, even against one’s prior work Invest career promoting one’s own theories, findings

Norms Counternorms Communality Secrecy Open sharing Closed Universalism Particularlism Evaluate research on own merit Evaluate research by reputation Disinterestedness Self-interestedness Motivated by knowledge and discovery Treat science as a competition Organized skepticism Organized dogmatism Consider all new evidence, even against one’s prior work Invest career promoting one’s own theories, findings Quality Quantity

Anderson, Martinson, & De. Vries, 2007

Munafo et al. , 2017, Nature Human Behavior

Figure credit: fivethirtyeight. com Silberzahn et al. , in press, AMPPS

N = 142 dissertations that were subsequently published 45% dissertation hypotheses were significant 66% publication hypotheses were significant Tests added or removed 333 hypotheses added to publication, 70% significant 1333 hypotheses removed for publication, 38% significant Tests in dissertation and publication 272 unsupported hypotheses, 20. 6% changed to significant 373 supported hypotheses, 4. 6% changed to non-significant O’Boyle, Banks, & Gonzalez-Mule, 2015, Journal of Management

Incentives for individual success are focused on getting it published, not getting it right Nosek, Spies, & Motyl, 2012

The Kindergartener’s Guide to Improving Research

1. Show your work 2. Share

Changing a Research Culture Policy Make it required Incentives Make it rewarding Communities Make it normative User Interface/Experience Infrastructure Make it easy Make it possible

Signals: Making Behaviors Visible Promotes Adoption Kidwell et al. , 2016, PLOS Biology

% Articles reporting that data was available 40% 30% 20% 10% 0%

Context of Justification Confirmation Data independent Hypothesis testing Context of Discovery Exploration Data contingent Hypothesis generating p-values interpretable p-values NOT interpretable Presenting exploratory as confirmatory increases publishability of results at the cost of credibility of results Nosek, Ebersole, De. Haven, & Mellor, 2018, PNAS

p-values interpretable PREREGISTRATION Context of Justification Confirmation Data independent Hypothesis testing Context of Discovery Exploration Data contingent Hypothesis generating p-values NOT interpretable Presenting exploratory as confirmatory increases publishability of results at the cost of credibility of results Nosek, Ebersole, De. Haven, & Mellor, 2018, PNAS

Registered Reports http: //cos. io/rr Design Collect & Analyze Report PEER REVIEW http: //cos. io/rr, Committee Chair: Chris Chambers Publish

https: //www. psychologicalscience. org/observer/preregistration-becoming-the-norm-in-psychological-science

TOP Guidelines http: //cos. io/top/ 1. 2. 3. 4. 5. 6. 7. 8. Data citation Design transparency Research materials transparency Data transparency Analytic methods (code) transparency Preregistration of studies Preregistration of analysis plans Replication

Some TOP Signatory Organizations • • • AAAS/Science American Academy of Neurology American Geophysical Union American Heart Association American Meterological Society American Society for Cell Biology Association for Psychological Science Association for Research in Personality Association of Research Libraries Behavioral Science and Policy Association Bio. Med Central Committee on Publication Ethics Electrochemical Society Elsevier Frontiers MDPI Nature-Springer Peer. J Pensoft Publishers Public Library of Science The Royal Society for Personality and Social Psychology • Society for a Science of Clinical Psychology • Ubiquity Press • Wiley • • •

Journal Policies Reviewed policies of 33 journals that publish social-personality psychology • 73% (24 of 33) >0 steps on TOP • 58% (19) >5 steps on TOP

Journal TOP Comprehensive Results in Social Psychology 20 AMPPS 18 Collabra 17 European Journal of Personality 15 Journal of Research in Personality 14 Psychological Science 12 Social Psychology 11 Experimental Psychology 11 Journal of Personality and Social Psychology 11 Social Cognition 11 Judgment and Decision Making 10 Nature Human Behavior 8 RRs Badges

Journal Policies Reviewed policies of 33 journals that publish social-personality psychology • 73% (24 of 33) >0 steps on TOP • 58% (19) >5 steps on TOP Reviewed policies of 18 management journals • 44% (8 of 18) >0 steps on TOP • 0% had >5 steps on TOP

Journal TOP The Leadership Quarterly 4 Management and Organization Review 3 Journal of Business and Psychology 2 Personnel Psychology 2 Strategic Management Journal 1 Journal of Business Venturing 1 Acad. of Management Learning & Education 1 … 0 … 0 … 0 RRs Badges

UNIVERSITIES PUBLISHING FUNDERS SOCIETIES ecosystem

Changing a Research Culture Policy Make it required Incentives Make it rewarding Communities Make it normative User Interface/Experience Infrastructure Make it easy Make it possible

Changing a Research Culture Diffusion of Innovations; Rogers, 1963

Make it required Make it rewarding Make it normative Make it easy Make it possible

More information Take a picture OSF: http: //osf. io/ TOP Guidelines: http: //cos. io/top/ Registered Reports: http: //cos. io/rr/ Badges for open practices: http: //cos. io/badges/ OSF Preprints: http: //osf. io/preprints/ These slides: Brian Nosek, nosek@virginia. edu

References • • Kidwell, M. C. , Lazarevic, L. B. , Baranski, E. , Hardwicke, T. E. , Piechowski, S. , Falkenberg, L-S. , Kennett, C. , Slowik, A. , Sonnleitner, C. , Hess-Holden, C. , Errington, T. M. , Fiedler, S. , & Nosek, B. A. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLOS Biology, 14, e 1002456. Doi: 10. 1371/journal. pbio. 1002456 Munafò, M. R. , Nosek, B. A. , Bishop, D. V. M. , Button, K. S. , Chambers, C. D. , Percie du Sert, N. , Simonsohn, U. , Wagenmakers, E-J. , Ware, J. J. , & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behavior, 1, 0021. Doi: 10. 1038/s 41562 -016 -0021 Nosek, B. A. , Alter, G. , Banks, G. C. , Borsboom, D. , Bowman, S. D. , Breckler, S. J. , Buck, S. , Chambers, C. D. , Chin, G. , Christensen, G. , Contestabile, M. , Dafoe, A. , Eich, E. , Freese, J. , Glennerster, R. , Goroff, D. , Green, D. P. , Hesse, B. , Humphreys, M. , Ishiyama, J. , Karlan, D. , Kraut, A. , Lupia, A. , Mabry, P. , Madon, T. A. , Malhotra, N. , Mayo-Wilson, E. , Mc. Nutt, M. , Miguel, E. , Levy Paluck, E. , Simonsohn, U. , Soderberg, C. , Spellman, B. A. , Turitto, J. , Vanden. Bos, G. , Vazire, S. , Wagenmakers, E. J. , Wilson, R. , & Yarkoni, T. (2015). Promoting an open research culture. Science, 348, 1422 -1425. Doi: 10. 1126/science. aab 2374 Nosek, B. A. , Ebersole, C. R. , De. Haven, A. , Mellor, D. M. (2018). The preregistration revolution. Proceedings for the National Academy of Sciences. Nosek, B. A. , & Lakens, D. (2014). Registered reports: A method to increase the credibility of published results. Social Psychology, 45, 137 -141. Doi: 10. 1027/1864 -9335/a 000192 Nosek, B. A. , Spies, J. R. , & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615 -631. DOI: 10. 1177/1745691612459058 Open Science Collaboration. (2015). Estimating the Reproducibility of Psychological Science, 349(6251), aac 4716. DOI: 10. 1126/science. aac 4716. Silberzahn, R. , Uhlmann, E. L. , …, & Nosek, B. A. (in press). Many analysts, one dataset: Making transparent how variations in analytical choices affect results. Advances in Methods and Practices in Psychological Science.