Transparency in Qualitative Research Beyond Sharing Data Mannheim

Transparency in Qualitative Research Beyond Sharing Data Mannheim Open Science Meetup, August 26, 2020 Sebastian Karcher, Qualitative Data Repository

What Is QDR? • Online since 2014: qdr. syr. edu, NSF funded QDR curates, stores, preserves, publishes, and enables the download of digital data generated through qualitative and multi-method research in the social sciences. • HQ at Syracuse; other team members at Georgetown and UW Seattle • Originated in political science: International and interdisiplinary • Currently 90 data projects published

Why Transparency? • Adhering to methodological prescription is what distinguishes science o science is process-dependent • Rule-following must be public o B/c “the content is the method” o B/c demonstrating the processes that produced a conclusion empowers/allows evaluation o social science is process-visibility dependent

Transparency vs. Reproducibility Transparency: Show data and methods used to arrive at conclusions Reproducibility: Same data, same methods same results In quantitative research, reproducibility is a byproduct of transparency “The replication standard holds that sufficient information exists with which to understand, evaluate, and build upon a prior work if a third party could

Transparency vs. Reproducibility: Epistemology Reproducibility Solipsism: Only author can understand their own data Interpretation, “understanding” Positivism: Search for universal laws Transparency

Transparency vs. Reproducibility: Research cycle Reproducibility Hypothesis generation Theory building Theory testing Exploration Transparency

Transparency in Practice

Types of Transparency • Data access o What data were used, where are they, are they available? o If you generate your own data, share them or say why you cannot. • Production transparency o If authors’ own data, how were they produced? o Requires providing documentation describing how the data were generated/collected • Analytic transparency o How were data analyzed to arrive at the conclusion? o How are the evidence and claims are connected? Based on APSA, 2012

Data Access for Qualitative Data • Controversial • Often possible, even with sensitive data • May be impossible due to • • Nature of data Informed consent Copyright/licensing Partnership agreements

Beyond Sharing Data: A Menu • Qualitative Preregistration • Methodological Appendices • Annotation / ATI • QDA Software Output • Checklists / Reporting Guidelines Based on Kapiszewski & Karcher, 2020

Preregistration • Move to preregistration in qualitative research • Several theoretical articles, few existing applications • Works best where qual is used to test theory, e. g. mechanism: “Our case study of Golden Veroleum Liberia (GVL) makes two contributions. First, it validates our claims about how property rights systems shape external investment. ” “Second, the case study suggests an extension of our formal model” Christensen et al. 2020

Methodological Appendices: Interviews • Ethnographic (e. g. Goffman 2014) • Interpretative (e. g. Shesterinina 2016) • Interview Methods Appendix (e. g. Bleich 2018)

Methodological Appendices: Historical • Archival log • Use of secondary historical sources (e. g. Verghese 2016) • Coding of historical materials (e. g. Treisman 2020) Although please don’t do this: https: //static 1. squarespace. com/static/5 a 4 d 2512 a 803 bb 1 a 5 d 9 aca 35/t/5 e 82 c 53510428029 ac 9 b 4888/15856 28479641/Online+Appendix+Treisman+Democratizati on+by+Mistake+mar+2020. pdf

Annotation for Transparent Inquiry Annotation Source document Publication Link: bit. ly/ati-omahoney

Sharing QDA Software Output • Software like Nvivo, Atlas. ti, etc. used widely • Outputs rarely shared, lack of common standard…. • Until recently: REFI-QDA project exchange now supported by most major tools • Recommended by DANS, Dataverse. NO, and QDR • First project in QDR using the standard forthcoming

Checklist and Reporting Guidelines • Reporting guidelines very common in medicine, especially for clinical studies • Rarely used in social science beyond psychology, virtually nonexistent in qualitative social science • Existing reporting guidelines • COREQ: COnsolidated Criteria for REporting Qualitative Research • ENTREQ: Enhancing Transparency in Reporting the Synthesis of Qualitative Research • SPQR: Standards for Reporting Qualitative Research

Example Reporting Guidelines: SPQR

Qualitative Transparency Checklist: Transparency as a Workflow • Frohwirth & Karcher (forthcoming) • Used both as internal tracking document and as supplement • Allows for but does not require data sharing • Non-public draft at: bit. ly/qualchecklist

Questions? Comments? Please stay in touch: https: //qdr. syr. edu @adam 42 smith (Sebastian) @qdrepository (QDR) Email: skarcher@syr. edu qdr@syr. edu