Transparency and Openness Promotion TOP Guidelines David Mellor

Transparency and Openness Promotion (TOP) Guidelines David Mellor Journal and Funder Initiatives https: //cos. io | https: //osf. io | @OSFramework

What are the TOP Guidelines?

What are the TOP Guidelines? Eight standards that show editors and funders practices for increasing the transparency and reproducibility of the research that they publish or fund.

What are the TOP Guidelines? Eight standards that show editors and funders practices for increasing the transparency and reproducibility of the research that they publish or fund. • Agnostic to discipline • Low barrier to entry • Modular

Three Tiers Disclose Require Eight Standards Data citation Materials transparency Data transparency Code transparency Design transparency Study Preregistration Analysis Preregistration Replication Verify

Three Tiers Disclose Require Eight Standards Data citation Materials transparency Data transparency Code transparency Design transparency Study Preregistration Analysis Preregistration Replication Verify

Three Tiers Disclose Require Eight Standards Data citation Materials transparency Data transparency Code transparency Design transparency Study Preregistration Analysis Preregistration Replication Verify

Organizational signatories are: 1. Expressing their support of the principles of openness, transparency, and reproducibility. 2. Encouraging associated journals to conduct a review of the standards and levels for adoption.

Why were they created? • In response to requests for specific guidance on how to reward work that aligns with scientific values. • Those values include sharing of evidence, motivation by knowledge and discovery, and organized skepticism. • Typical publishing processes reward novel, unexpected, tidy, and significant results. By these criteria, most published research cannot be replicated.

Why were they created? • In response to requests for specific guidance on how to reward work that aligns with scientific values. • Those values include sharing of evidence, motivation by knowledge and discovery, and organized skepticism. • Typical publishing processes reward novel, unexpected, tidy, and significant results. By these criteria, most published research cannot be replicated.

Why were they created? • In response to requests for specific guidance on how to reward work that aligns with scientific values. • Those values include sharing of evidence, motivation by knowledge and discovery, and organized skepticism. • Typical publishing processes reward novel, unexpected, tidy, and significant results. By these criteria, most published research cannot be replicated.

Who created them? • 39 authors who serve in roles on editorial boards, publishers, and funding organizations.

What do they cover?

Data citation “All data, program code and other methods (1) should or (2) must be appropriately cited. Such materials should be recognized as original intellectual contributions and afforded recognition through citation. ” (3) “Articles will not be published until the citations conform to these standards. ”

Data, Analytic Methods (Code), and Research Materials Transparency 1. Authors must disclose sharing

Data, Analytic Methods (Code), and Research Materials Transparency 1. Authors must disclose sharing 2. Authors must share (except when ethical constraints prohibit)

Data, Analytic Methods (Code), and Research Materials Transparency 1. Authors must disclose sharing 2. Authors must share (except when ethical constraints prohibit) 3. Journal or third party will verify that the data can be used to replicate the findings presented in a paper.

Data, Analytic Methods (Code), and Research Materials Transparency 1. Authors must disclose sharing 2. Authors must share (except when ethical constraints prohibit) 3. Journal or third party will verify that the data can be used to replicate the findings presented in a paper.

Design and Analysis Transparency • Disciplines or journals must define the relevant reporting standards that are appropriate for their discipline.

Preregistration of studies is a means of making research more discoverable even if it does not get published. Preregistration of Analysis Plans certifies the distinction between confirmatory and exploratory research.

Exploratory research: • Finds unexpected trends • Pushes knowledge into new areas • Results in a testable hypothesis

Confirmatory research: • Puts a hypothesis to the test • Does not allow data to influence the hypothesis • Results are held to the highest standard of rigor

Replication • Encourage replication of previously published findings

757 Journal and 63 Organizational Signatories

What role and in what ways can publishers support this effort? • Becoming a signatory provides a common framework for a diverse group of disciplines under one publishing roof to consider the actions that TOP covers. • It would not be easy to mandate a blanket policy across different disciplines.

What incentives are needed for researchers to more openly share? How do we enable sharing without placing more burden on researchers?

1. Mandates 2. Tack on extra steps to a process 3. Appeal to everyone’s good nature

1. Mandates 2. Tack on extra steps to a process 3. Appeal to everyone’s good nature

1. Align scientific values (open sharing, value of evidence) with scientific rewards (getting published, funded, and hired). 1. Make tools that make researchers’ lives easier, while making open science as easy as pressing a button.

Collaboration Documentation Archiving

Put data, materials, and code on the OSF

Open. Sesame

Open. Sesame

Signals: Making Behaviors Visible Promotes Adoption

Data Availability in Psychological Science

The Prereg Challenge $1, 000 in prizes for researchers to publish the results of their preregistered work. Makes distinction between exploratory and confirmatory research more clear.

https: //cos. io/prereg

https: //cos. io/prereg Research questions Data collection methods Variables Statistical tests Outliers

Let Experts be Experts

Substantive experts shouldn’t have to deal with… • • • Workflow integration Authentication Permissions File storage File rendering Database Metadata/Annotations/Commenting Persistence External service integrations Search SHARE Data

Substantive experts should focus on… • Defining community standards • Changing community values and incentives • Determining specific community requirements • Doing scholarship

Society-branded Preprint Servers

Thank you! David Mellor, david@cos. io Find me online: https: //osf. io/qthsf/ @Evo. Mellor Find this presentation: https: //osf. io/95 z 9 s