Implementing Data Review Policies Practicalities and Perspectives Mandy

Implementing Data Review Policies: Practicalities and Perspectives Mandy Gooch, Research Data Archivist (Odum Institute) Thu-Mai Christian, Assistant Director for Archives (Odum Institute) Todd Vision, Associate Professor, Department of Biology (UNC-CH) Elizabeth Hull, Operations Manager (Dryad) Support for this research was provided by the Robert Wood Johnson Foundation. The views expressed here do not necessarily reflect the views of the Foundation.

§ Open access to materials is only useful if the data are well -documented and accompanied by the analysis code § TOP Guidelines (Nosek et al. , 2015) § Define actionable standards for data transparency at increasing levels of stringency that are enforceable by journals through open data policies Background § TOP Level I – Article states whether data/code/materials are available and where to access them § TOP Level II – Data/code/materials must be posted to a trusted repository § TOP Level III – Data/code/materials must be posted to a trusted repository and reported analyses will be reproduced independently prior to publication

§ Seeks to identify the most effective and efficient methods for implementing data policies that incorporate review and verification. Our Project § Cannot be a burden § Achieves a common understanding of what constitutes acceptable data § Modifies the current scholarly publication and review workflow

§ Review of journal data policies § Survey of Editors, Authors/Reviewers of social, medical, and biological sciences journals. Questions target: Methodolog y § Editors understanding and practices in regard to the journal’s data policies § Author/Reviewer’s understanding of the policies § Challenges for both Editor and Author/Reviewer roles § Interviews with Editors, Authors/Reviewers, and data journals

§ Received 113 responses from 678 Editors surveyed (Response Rate: 16. 7%) Preliminary Results: Overview § Archivists reviewed and ranked policies based on the language provided by the TOP Guidelines § We had a few instances where we could not find the data policy

Does the data policy issued by the Journal require authors to do the following? Submit data underlying article findings to a trusted repository Preliminary Results: Editor Survey Submit analytic methods to a trusted repository Submit research materials to a trusted repository Explain access restrictions for data that cannot be shared due to legal or ethical reasons Describe the process for accessing data that cannot be shared due to legal or ethical reasons Other

Discrepancy between Editor Understanding of Data Policy and Review of Data Policy 40 34 35 30 Results so far: Policy Review 28 27 25 19 20 15 10 5 0 Analytical Methods Transparency Data Transparency Editors - TOP Level II or III Research Materials Transparency Review - TOP Level II or III

§ Of the 54 respondents it was determined that: Results so far: Policy Review § 2 journals met the requirements of TOP Level III § 25 journals met the requirements of TOP Level II § 27 journals were not Top Level II or III

Preliminary Results: Discrepancy & Ambiguity After conducting a review of the policies: • Some discrepancy between Editor understanding of policy and actual policy language • Rare that the Editor understanding and the policy language matched • Ambiguous phrasing and use of language that may imply a requirement, but was not specific

§ Surveying authors and reviewers for these journals § Conducting one-on-one interviews with selected Editors Future Steps and Authors for insight into their experiences with data policy implementation and workflow § Develop a model for data policy implementation to be shared with journals

§ What factors could lead to a discrepancy between an Discussion Questions Editor’s understanding of a data policy and the language of the data policy? § Why do you think there are so few TOP Level III (Verification required) data policies?

Thank you! Mandy Gooch Research Data Archivist The Odum Institute, UNC-CH agooch@unc. edu Support for this research was provided by the Robert Wood Johnson Foundation. The views expressed here do not necessarily reflect the views of the Foundation.