Tools for Improving Transparency Replicability Introductions Grace Binion

Tools for Improving Transparency & Replicability

Introductions Grace Binion, Ph. D candidate at University of Oregon Jack Arnal, Associate Professor at Mc. Daniel College Pamela Davis-Kean, Professor at University of Michigan Melissa Kline, Research Scientist at the Center for Open Science Benjamin Brown, Associate Professor at Georgia Gwinnett College

Overview ● Purpose - increase accessibility of tools for open & transparent practices ● Topics ○ ○ Preregistration Project Organization Data Sharing Preprints & Dissemination ● Format ○ ○ Why take this step? What tools can I use? How do I use them? I have other questions. . .

Preregistration of Complex Designs

Preregistration - Why do it? ● Decision Independence ○ ○ Result = f 1(data, decisions) Decisions = f 2(data) ● Transparency

Preregistration - What tools exist? ● As Predicted ● Open Science Framework ○ ○ ○ Standard OSF template Open-ended registrations Templates in progress for secondary data analysis ● Alternatives to “traditional” preregistration ○ ○ Generating (and registering) standard operating procedures Contingencies/decision trees

How to Start? ● As. Predicted ○ ○ www. aspredicted. org → “Create” Get started answering questions! ● OSF ○ www. osf. io/prereg/

How to Start? - OSF

Examples ● Detailed Pre-Registration ● Open-Ended ● Template for Secondary Data Analysis ○ ○ Finalized except for programming here Another template described here ● osf. io/registries → “See An Example” below search box

Other Resources ● Papers ○ ○ ○ Srivastava (2018, November 21) Weston, Richie, Rohrer, & Przybylski (2018, July 6) Mertens, G. , & Krypotos, A. (2019, February 20) ● Guides ○ ○ Types of preregistration forms (http: //help. osf. io/m/registrations/l/524206 -understandregistration-forms) Step-by-step OSF preregistration (http: //help. osf. io/m/registrations/l/546603 -create-apreregistration ) ● R markdown preregistrations ○ ○ https: //github. com/crsh/prereg/blob/master/README. md Template built into R markdown

Preregistration - Questions ● How much needs to be specified in advance? ● Can my preregistration be changed? ● What happens if I deviate from my preregistration?

Organizing Projects with the OSF

The Open Science Framework (OSF) What is it?

The OSF - Why use it? ● ● ● ● Increase reproducibility, transparency, and openness Shared workspace Tool integration Centralized storage Clear version control Ease of collaboration Curation Ease of sharing

Using OSF Projects ● Various uses ○ ○ Individual Projects Research Lines Collaborations (various sizes) Pedagogy

Demo https: //osf. io/

Using OSF Projects - Questions ● Do I have to learn something new? ● Why OSF over other services? ● Do all components need to be made public?

Sharing Sensitive Data

Why share data? • Benefits to the researcher – Data in the public domain generate new research which cites the original research – Data are preserved and can be obtained by the original researchers if their copies are lost or destroyed – Archiving data helps researchers meet requirements of NIH and NSF data management plans and NIJ data archiving special conditions requirements – Information on research community’s interest in a dataset can assist with the success of future grant funding proposals

Options for sharing data through ICPSR.

Should the data be restricted? • • Re-identification Can individuals be identified from information in this material? If the data were made public, could someone use a ○ ○ ○ • combination of variables (e. g. age, sex, race, occupation, geography) to find individuals in a publicly available database? Harm Does this material include sensitive information? Would the release of individually identifiable information ○ ○ create a risk of harm (e. g. psychological distress, social embarrassment, financial loss) greater than the risks that people experience in everyday life?

Identifiers Direct identifiers must be removed from data before sharing! Data with indirect identifiers may need to be restricted. • Direct – Names – Addresses, including ZIP codes and other postal codes – Telephone and fax numbers, including area codes – Social security numbers – Other linkable numbers such as driver’s license numbers, certification numbers, medical device numbers, etc. • Indirect – Detailed geographic information – Organizations belonged to, offices or posts held by respondent – Educational institutions attended, year of graduation – Detailed occupational titles – Place where respondent was born or grew up – Exact dates of significant life events (birth, death, marriage, etc) – Detailed income

What sensitive topics could cause harm? • Psychological well being or mental health • Sexual attitudes, preferences, or practices • Use of alcohol or drugs • Illegal behavior • Behavior that puts the respondent at risk of criminal or civil liability • Behavior damaging to an individual’s financial standing, employability, or reputation • Medical information that could lead to discrimination, stigmatization, etc.

Resource • Link to Guidelines for Effective Data Management Plans (PDF), s http: //www. icpsr. umich. edu/files/datamanagement/Data. Management. Plans. All. pdf. • Link to Guidelines and Resources for OSTP Data Access Plans (video), https: //www. youtube. com/watch? v=s. Wn. MFEKmfn. E • Link to ICPSR Deposit Data (web page), http: //www. icpsr. umich. edu/icpsrweb/deposit/index. jsp • Link to open. ICPSR (website and vidoes) https: //www. openicpsr. org/ • Link to Disclosure Risk Training – For Public Use or Not For Public Use (video), https: //www. youtube. com/watch? v=9 vd. Wse. Lay 9 g&feature=youtu. be&list=P Lq. C 9 lrh. W 1 Vva. Kgzk-S 87 Wwrl. SMHli. HQo 6%22

Sharing Data - What tools exist? ● Completely open ○ OSF ● Variable levels of access ○ ○ ICPSR Also OSF! ● Shared with restrictions ○ ○ National Data Archives on Child Abuse and Neglect

Planning for data sharing from the beginning - Even if you think you are never going to share your data publicly - - Your collaborators will thank you You (in 6 months) will thank you Shareable data is usable data What if you get hit by a truck? What if you discover something *really important*? - Methodological advances An extraordinary claim, that requires extraordinary evidence Cure for cancer, etc. . .

Consent Forms - Ask your participants what they want! ‘Step down’ models allow participants to really understand their choices (what in the world is ‘research purposes? ’) Examples - - Databrary Lookit Many. Babies Work with your IRB - Advocate for ethical data sharing!

Planning for data sharing ● If there’s just one piece of data that renders your data identifiable, do you *really* need to record it for research purposes? ○ Salty. Dates ● “Born Open” scientific workflows ○ Share early, share often ● …. Are you already ‘sharing’ by mistake? ● “The Internet” ○ ○ We’re past storing the computer ‘in a locked room’ (Don’t email your data to each other) Harvard Data Security levels Repositories have security pages

How much data should I share? - As much as you ethically can! Consider a ‘hold out’ model Just one file is much better than zero Less data, better documented > a lot of data I can’t use - BIDS neuroimaging standard Psych-DS

What to do when you really can’t share the data ● Open Science is *MUCH* more than open data! ● Codebooks and metadata are *especially* useful when the data itself is absent ○ Psych-DS Plug! ● Reproducible analyses ○ Code that *would* run if you had the actual datasheet, so you can see exactly how it works ● Make and share dummy data ○ ○ …. just fill in a spreadsheet with a few rows, you’d be surprised how many questions this answers! Simulating dummy data ● Format & document your data just the same as if you *were* planning on sharing it - your future self will thank you

Sharing Work on Psy. Ar. Xiv

Psy. Ar. Xiv - Why use it? ● ● ● Dissemination of null results Solicit feedback on works in progress Increase speed of dissemination Increase access and discoverability Increase transparency

Psy. Ar. Xiv - What can be shared? ● ● Working papers/manuscripts in progress Conference proceedings Papers unable to be published elsewhere Papers accepted to traditional journals ○ SHe. RPa/Ro. MEO - http: //www. sherpa. ac. uk/romeo/index. php

Demo https: //psyarxiv. com/

Licenses on Psy. Ar. Xiv: Three Options ● No license ○ Provides no formal instructions about how to use your work ● CC-0 1. 0 Universal ○ ○ Open Work is dedicated to the public; others can copy, modify, distribute, use commercially, work without asking permission. ● CC-By Attribution 4. 0 International ○ ○ Open Others can modify, distribute, use commercially and rename always with attribution (which can be revoked by owner), plus a URL, copyright notice, statement of changes.

Psy. Ar. Xiv - Questions ● ● What can’t be shared on Psy. Ar. Xiv? What are downsides of sharing work? Can products on Psy. Ar. Xiv be cited? FAQ on our blog

General Q&A