Changing a Research Culture toward Openness and Reproducibility

Changing a Research Culture toward Openness and Reproducibility Brian Nosek University of Virginia -- Center for Open Science http: //briannosek. com/ -- http: //cos. io/

Norms Counternorms Communality Secrecy Open sharing Closed

Norms Counternorms Communality Secrecy Open sharing Closed Universalism Particularlism Evaluate research on own merit Evaluate research by reputation

Norms Counternorms Communality Secrecy Open sharing Closed Universalism Particularlism Evaluate research on own merit Evaluate research by reputation Disinterestedness Self-interestedness Motivated by knowledge and discovery Treat science as a competition

Norms Counternorms Communality Secrecy Open sharing Closed Universalism Particularlism Evaluate research on own merit Evaluate research by reputation Disinterestedness Self-interestedness Motivated by knowledge and discovery Treat science as a competition Organized skepticism Organized dogmatism Consider all new evidence, even against one’s prior work Invest career promoting one’s own theories, findings

Norms Counternorms Communality Secrecy Open sharing Closed Universalism Particularlism Evaluate research on own merit Evaluate research by reputation Disinterestedness Self-interestedness Motivated by knowledge and discovery Treat science as a competition Organized skepticism Organized dogmatism Consider all new evidence, even against one’s prior work Invest career promoting one’s own theories, findings Quality Quantity

Anderson, Martinson, & De. Vries, 2007

Incentives for individual success are focused on getting it published, not getting it right Nosek, Spies, & Motyl, 2012

Munafo et al. , 2017, Nature Human Behavior

Changing a Research Culture Diffusion of Innovations; Rogers, 1963

Changing a Research Culture Policy Make it required Incentives Make it rewarding Communities Make it normative User Interface/Experience Infrastructure Make it easy Make it possible

Make it required Make it rewarding Make it normative Make it easy Make it possible

Signals: Making Behaviors Visible Promotes Adoption Kidwell et al. , 2016, PLOS Biology

% Articles reporting that data was available 40% 30% 20% 10% 0%

Percentage of Psych Science Articles Earning Badges

Registered Reports http: //cos. io/rr Design Collect & Analyze Report Publish PEER REVIEW http: //cos. io/rr, Committee Chair: Chris Chambers Nosek & Lakens, 2014

Registered Reports are working as intended More likely to discover that our hypotheses are wrong… Allen & Mehler, 2019 …while the articles are well cited by scientists https: //tinyurl. com/RR-citations Adapted from Chris Chambers

Changing Norms Reproducibility & Transparency Initiatives Features • High visibility • Multiple stakeholders • Incentives aligned • Light commitment • Proof-of-concept • Shows it’s possible • Shows it’s worthwhile • Shows others care Examples • Arnold Ventures & Social Psychology • Children’s Tumor Foundation & PLOS • Arnold Ventures & Pers. on Psych Science • Cancer Research UK & Nicotine and Tobacco Research • Flu Lab & PLOS

http: //cos. io/flulab

Evidence of Culture Change

Paluck et al. , 2018

2019 Files Added: 2. 77 M Public Files: 0. 99 M

2019 Downloads: 16. 3 M

Transparency Policies for Leading Psych Journals TOP 1 TOP 2 TOP 3 TOP 4 TOP 5 TOP 6 TOP 7 TOP 8 Total RRs Badges Comprehensive Results in Social Psychology 2 2 3 3 20 YES NO Advances in Methodologies and Practices in Psychological Science 0 2 2 2 3 3 18 YES Collabra: Psychology 2 2 2 2 3 17 YES NO European Journal of Personality 1 1 2 2 2 3 15 YES Journal of Research in Personality 0 2 2 2 2 14 YES Psychological Science 0 1 1 1 2 2 2 3 12 YES Science 2 2 1 1 12 NO NO Social Psychology 0 2 2 1 11 YES Experimental Psychology 2 2 0 0 0 3 11 YES NO Journal of Personality and Social Psychology 2 1 1 1 3 11 YES NO

and Collaboration: Increase the number and the clarity of journals and funders data policies by classifying the recommendations these policies contain to improve their definition and guidance to researchers Workplan – phase 1: Curate and assess their compliance to the Transparency and Openness Promotion (TOP) guidelines and display the level in FAIRsharing

UNIVERSITIES PUBLISHERS FUNDERS SOCIETIES ecosystem

Institutional Culture Values Perception Reality

Institutional Culture A Simple, Fundamental Change to Align Values, Perceptions, Reality 1 2 3

Norms Incentives Training

Grassroots Campaigns • • Visibility is essential for changing norms Open science as good practice (not as social identity) Embrace incrementalism (not all-or-none) Cultivate intellectual humility – Getting it right over being right – Normalize error – Importance of stupidity in research (Schwartz, 2008: https: //jcs. biologists. org/content/121/11/1771)

More information Take a picture OSF: http: //osf. io/ TOP Guidelines: http: //cos. io/top/ Preregistration: http: //cos. io/prereg/ Registered Reports: http: //cos. io/rr/ Badges for open practices: http: //cos. io/badges/ OSF Registries: http: //cos. io/registries/ These slides: https: //osf. io/4 ew 7 h/ Brian Nosek, nosek@cos. io

References • • • Camerer, C. F. , Dreber, A. , Holzmeister, F. , Ho, T-H. , Huber, J. , Johannesson, M. , Kirchler, M. , Nave, G. , Nosek, B. A. , Pfeiffer, T. , Altmejd, A. , Buttrick, N. , Chan, T. , Chen, Y. , Forsell, E. , Gampa, A. , Heikensten, E. , Hummer, L. , Imai, T. , Isaksson, S. , Manfredi, D. , Rose, J. , Wagenmakers, E-J. , Wu, H. (2018). Evaluating Replicability of Social Science Experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour. Doi: 10. 1038/s 41562 -018 -0399 -z Kidwell, M. C. , Lazarevic, L. B. , Baranski, E. , Hardwicke, T. E. , Piechowski, S. , Falkenberg, L-S. , Kennett, C. , Slowik, A. , Sonnleitner, C. , Hess-Holden, C. , Errington, T. M. , Fiedler, S. , & Nosek, B. A. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLOS Biology, 14, e 1002456. Doi: 10. 1371/journal. pbio. 1002456 Munafò, M. R. , Nosek, B. A. , Bishop, D. V. M. , Button, K. S. , Chambers, C. D. , Percie du Sert, N. , Simonsohn, U. , Wagenmakers, E-J. , Ware, J. J. , & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behavior, 1, 0021. Doi: 10. 1038/s 41562 -016 -0021 Nosek, B. A. , Alter, G. , Banks, G. C. , Borsboom, D. , Bowman, S. D. , Breckler, S. J. , Buck, S. , Chambers, C. D. , Chin, G. , Christensen, G. , Contestabile, M. , Dafoe, A. , Eich, E. , Freese, J. , Glennerster, R. , Goroff, D. , Green, D. P. , Hesse, B. , Humphreys, M. , Ishiyama, J. , Karlan, D. , Kraut, A. , Lupia, A. , Mabry, P. , Madon, T. A. , Malhotra, N. , Mayo-Wilson, E. , Mc. Nutt, M. , Miguel, E. , Levy Paluck, E. , Simonsohn, U. , Soderberg, C. , Spellman, B. A. , Turitto, J. , Vanden. Bos, G. , Vazire, S. , Wagenmakers, E. J. , Wilson, R. , & Yarkoni, T. (2015). Promoting an open research culture. Science, 348, 1422 -1425. Doi: 10. 1126/science. aab 2374 Nosek, B. A. , Ebersole, C. R. , De. Haven, A. , Mellor, D. M. (2018). The preregistration revolution. Proceedings for the National Academy of Sciences. Nosek, B. A. , & Lakens, D. (2014). Registered reports: A method to increase the credibility of published results. Social Psychology, 45, 137 -141. Doi: 10. 1027/1864 -9335/a 000192 Nosek, B. A. , Spies, J. R. , & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615 -631. DOI: 10. 1177/1745691612459058 Open Science Collaboration. (2015). Estimating the Reproducibility of Psychological Science, 349(6251), aac 4716. DOI: 10. 1126/science. aac 4716. Silberzahn, R. , Uhlmann, E. L. , …, & Nosek, B. A. (2018). Many analysts, one dataset: Making transparent how variations in analytical choices affect results. Advances in Methods and Practices in Psychological Science.

Other Slides

and Collaboration: Workplan – phase 1: Curate and assess their compliance to the Transparency and Openness Promotion (TOP) guidelines and display the level in FAIRsharing

Make data and code available to the greatest extent permissible to improve reproducibility, robustness testing, and reuse. Make materials available to assist replication and reuse. Preregister studies to reduce publication bias, and preregister analysis plans to make clear distinction between planned hypothesis tests and unplanned exploratory work.

Evaluating and Shaping the Institutional Culture

Where practices diverge from values AKA Examples of how signals go wrong • • Job candidate introductions Promotion materials What is sent to external reviewers Abstract language gets filled in with perceptions – E. g. , Job ads, ECR instructions, committee instructions – Challenges: values are abstract & disciplines vary – Solution: concrete on what not to do and explicit about process of evaluation • Deploying standards with ad hoc committees

Example • “Successful candidates will demonstrate research excellence” • “Provide examples of how your research meets high standards of rigor and transparency. ”

Institutional Strategy: 3 Steps 1. Assessment: What is the current state? Where are values and practices misaligned? 2. Intervention: Implement pilots and policies focused on areas of misalignment. 3. Evaluation: Are the interventions working as intended? Producing unintended consequences?

1 Institutional Strategy What are the institution’s values for scholarship? How well do the policies and reward systems reflect those values? Where are opportunities to articulate and better align values with policies? 2 3 What do researchers perceive are the incentives and reward systems? Are those perceptions aligned with institutional values? Perception Values What are the actual reward criteria in practice? Are those aligned with institutional values and researcher perceptions? Reality

1 Hypothetical Example A Mission: We contribute to knowledge for advancing humanity by valuing and conducting high quality research characterized by rigor and transparency. 2 3 Survey: 72% of research faculty believe promotion is mostly dependent on quantity of publications, not quality. 81% perceive no reward for transparency in research. Perception Values - - + Review of past 10 years of decisions indicates quantity of publications accounts for only 3% of variation in promotion and tenure decisions. Reality

1 Hypothetical Example B Mission: We contribute to knowledge for advancing humanity by valuing and conducting high quality research characterized by rigor and transparency. 2 3 Survey: 72% of research faculty believe promotion is mostly dependent on quantity of publications, not quality. 81% perceive no reward for transparency in research. Perception Values - + - Review of past 10 years of decisions indicates quantity of publications accounts for 33% of variation in promotion and tenure decisions. Reality

1 Hypothetical Example C Mission: We contribute to knowledge for advancing humanity by valuing and conducting high quality research characterized by rigor and transparency. 2 3 Survey: Just 12% of research faculty believe promotion is mostly dependent on quantity of publications, not quality. 8% perceive no reward for transparency in research. Perception Values + + - Review of past 10 years of decisions indicates quantity of publications accounts for 33% of variation in promotion and tenure decisions. Reality

Low Hanging Fruit: Places to Start • • Assessment of Perceptions Training Technology Pilots: Job Ads, Curriculum, Norm setting

COS Services for Institutions • Consulting: Assessment, culture change strategy, and evaluating interventions • Training: Inculcate knowledge and experience with openness, rigor, reproducibility • OSF Institutions: Lower the barrier to adopting behaviors; institutional signaling of values • Coordination across stakeholders: e. g. , sharing values, perception, reality data across institutions