Open assessment in research Jouni Tuomisto THL 29092020

Open assessment in research Jouni Tuomisto, THL 29/09/2020 Presentation name / Author 1

Acknowledgements • The research about open assessment has been performed in several projects: – EU-funded: Intarese, Heimtsa, Beneris – Tekes: Cehra, Finmerac, Erac – Academy of Finland: Cehra, Scud, Bioher, Claih • The ideas have been developed by several people as a collaborative effort: – – – 29/09/2020 Risk research group, THL The Department of Environmental Health, THL Partners in other research projects Open (risk) assessment workshops 2007, 2008 All sceptical opponents Presentation name / Author 2

Objectives and outline • To briefly describe open assessment and Opasnet • To describe the concept of trialogue. • To clarify issues related to trialogue, science, and research. • To describe how open assessment method can be utilised in basic research in practice. • To discuss the critical reasons for scepticism against open assessment. • To convince the audience to try open assessment out in their own work. 29/09/2020 Presentation name / Author 3

What is open assessment? • How can scientific information and value judgements be organised for improving societal decision-making in a situation where open participation is allowed? – – – 29/09/2020 Serves practical information needs for policy. Seeks for the truth. Uses the scientific method as the ultimate criterion. Allows for open participation by anyone. Utilises Opasnet web toolbox. Presentation name / Author 4

Opasnet – where anyone can solve common problems • Opasnet (Opasnet Wiki): a workspace for ANY scientific or policy assessment work – Public (English and Finnish) and project sites • Opasnet Base: database for ANY research study and model results • Opasnet File: for file-formatted information • Open Assessors’ Network: A society for promoting the use of open assessment and Opasnet (informal, officially not yet founded. Found this week? ) • Journal of Open Assessment: An open-access scientific journal that peer reviews material that has been published in Opasnet. (hypothetical) 29/09/2020 Presentation name / Author 5

Opasnet challenge • I’ll pay 100 € for any researcher who is able to show that Opasnet cannot be utilised in a substantive way for planning, executing, analysing, or publishing in his/her study. – If the attempt fails, the researcher has to utilise Opasnet in the study. – If the critical functionality exists in theory but is missing in practice, I have two months time to install it. 29/09/2020 Presentation name / Author 6

Comparison of Wikipedia and Opasnet • Three major differences in Opasnet: – Opasnet is a workspace for research and policy assessments, not (primarily) an encyclopedia. – Original research is allowed. – Open assessment method is the ultimate criteria. • The other of five pillars of Wikipedia apply: – Neutral point of view (relaxed within discussions) – Assume good faith and other code of conduct – Free content that anyone may edit • See details: http: //en. opasnet. org/w/Open_Assessors%27_Network 29/09/2020 Presentation name / Author 7

Harvesting in the Soviet Union © AB Volvo 29/09/2020 Presentation name / Author 8

The church of Porvoo in fire, 2006 © Itä-Uudenmaan pelastuslaitos 29/09/2020 Presentation name / Author 9

Theory and practice • The fact that something works in practice, does not tell much about how well it could optimally work. • Practical “constraints” may constrain your thinking: – Helsinki and nuclear district heating: ”If there is political survival for the idea, we’ll start an assessment on its impacts. ” (Mayor Jussi Pajunen, Helsinki, according to Helsingin Sanomat 11 Feb, 2009) 29/09/2020 Presentation name / Author 10

Individual survival strategy I Observe Act Reality and its phenomena 29/09/2020 Presentation name / Author 11

Societal survival strategy: shared information and roles You Discuss Study I Policy Observe Act Reality and its phenomena 29/09/2020 Presentation name / Author 12

Trialogue: organising societal understanding You Discuss Read, write Study I Policy Variable Observe The topic Act Reality and its phenomena 29/09/2020 Presentation name / Author 13

What is the essence of science? • Science is a collaborative effort to understand reality. • Science is based on mutual information sharing, analysis, and rejection or acceptance. • The scientific method is the ultimate criterion. – – Ask questions about reality. Suggest answers. Test the plausibility of answers against observations. Make the answers and observations openly available for others to be evaluated. – Reject implausible answers. • Based on currently non-rejected answers, a coherent belief system is accepted within the (scientific) community. 29/09/2020 Presentation name / Author 14

Traditional science-policy interface 29/09/2020 Presentation name / Author 15

Open assessment interface 29/09/2020 Presentation name / Author 16

Two levels of Opasnet use • Use in addition to traditional research methods – Use Opasnet as an additional source of information. – Major scientific contributions in traditional journals. – Upload some original data if collaboration is wanted. • Full use of Opasnet – The main workspace for all research work. – When other software is used, upload files to Opasnet. – Major scientific contributions directly in Opasnet. Peer review in Journal of Open Assessment. 29/09/2020 Presentation name / Author 17

Examples of Opasnet in research • Describe methods and data (KTL sarcoma study) http: //en. opasnet. org/w/Ktl_Sarcoma_Study • Organise meetings (Bioher) http: //en. opasnet. org/w/Bioher_stakeholder_meeting_2009 • Manage projects (YMAL) http: //heande. opasnet. org/wiki/YMAL_weekly_meetings • Develop study designs (Occam) http: //heande. opasnet. org/wiki/Open_assessment_startup: _adolesc ents%2 C_biodiversity%2 C_and_climate_change • Analyse data (Epitox) http: //en. opasnet. org/w/Variable: Exposure―response_functions_for _tooth_defects_caused_by_TCDD_(study-specific) 29/09/2020 Presentation name / Author 18

Statements • The traditional scientific article is an outdated concept. • Publishing should be done first, peer review only after that. • Studies should focus on observations (i. e. , materials and methods and data itself), not interpretation (i. e. discussion and conclusions). • Interpretations of phenomena should contain data from all relevant studies and be placed separately as variables. • There is no fundamental difference between a research study and an assessment model. 29/09/2020 Presentation name / Author 19

Reasons why people are sceptic about Opasnet. Problems Solutions You can't make all people participate. Strong stakeholders will hijack the system. It will be a chaos and a huge pile of junk. Information is not found. I have to give away my data without getting merit. 29/09/2020 • Reading is more important than writing. • Opasnet produces marginal net benefit even for a small group. • It is easier than before to shoot down untrue statements. • Quality control systems must be developed further. • Clear information structure with variables. • Argumentation organises discussion. • Redundant contributions are deleted. • Linking is easy. • Google finds the content. • Moderating and cleaning up is necessary. • Protected project area can be used. • Showing data creates cooperation. • Journal of Open Assessment can be founded. Presentation name / Author 20

Phases of a research study: traditional and open assessment approaches 29/09/2020 Presentation name / Author 21