Quality trust and peer review researchers perspectives 10

Table of Contents Section Slide Number 1. Overview of approach 3 2. Executive Summary

Back to contents Background approach Research objectives are to: • Examine the drivers and

Back to contents EXECUTIVE SUMMARY Trust in peer review and communication Satisfaction with peer

SATISFACTION WITH PEER REVIEW: Has increased significantly since 2009 from 69% to 75%. Shift

8 ATTITUDES TO PEER REVIEW: widely seen to be improving the quality of research

PEER REVIEW SYSTEM: Who should undertake peer review? Most (87%) believe an evaluation of

10 PEER REVIEW SYSTEM: Clear guidance on reviewing is considered most important for maintaining

ASSESSMENT OF ARTICLES: Researchers would like to see a metric of ‘reproducibility’ to help

TRUSTWORTHINESS OF RESEARCH OUTPUTS: Most researchers 62% trust the majority of research outputs they

REASONS RESEARCH OUTPUTS ARE REGARDED AS UNTRUSTWORTHY: for those 13 researchers who rate at

TRUSTWORTHINESS OF RESEARCH PROFILES: Most researchers (75%) trust at least the majority of research

Reasons research-related profiles are regarded as untrustworthy: among those that question the quality of

SOURCING TRUSTED CONTENT: being able to adjust algorithms/parameters, transparency and interoperability seen as most

COMMUNICATION TO THE PUBLIC: Explaining research in lay terms is seen as the best

PUBLIC CONFIDENCE IN RESEARCH EVIDENCE: biggest problem seen to be misinterpretation or deliberate misrepresentation

Back to contents Attitudes to peer review Results by geographic region, country, broad subject

SATISFACTION WITH PEER REVIEW: lowest in Western Europe and North America. Also lower than

ATTITUDES TO PEER REVIEW: SSE/AH least accepting of data and supporting materials being reviewed

ATTITUDES TO PEER REVIEW: SSE/AH least likely to think peer review is well understood

Higher Significant difference between subset and Lower total (p=90%) ATTITUDES TO PEER REVIEW:

ATTITUDES TO PEER REVIEW: only 14% of researchers in the UK and USA think

ATTITUDES TO PEER REVIEW: Western European researchers are least likely to believe peer review

ATTITUDES TO PEER REVIEW: researchers in North America, Western Europe and Latin America are

Back to contents Who researchers think should undertake peer review Results by geographic region,

WHO RESEARCHERS THINK SHOULD UNDERTAKE PEER REVIEW: Earth/environmental science researchers more likely to consider

WHO RESEARCHERS THINK SHOULD UNDERTAKE PEER REVIEW: SSE/AH more likely to 31 expect peer

WHO RESEARCHERS THINK SHOULD UNDERTAKE PEER REVIEW: Researchers in China 32 less likely to

WHO RESEARCHERS THINK SHOULD UNDERTAKE PEER REVIEW: Researchers in India most likely to accept

WHO RESEARCHERS THINK SHOULD UNDERTAKE PEER REVIEW: 37% of North American researchers consider review

WHO RESEARCHERS THINK SHOULD UNDERTAKE PEER REVIEW: Researchers from Africa are less likely to

WHO RESEARCHERS THINK SHOULD UNDERTAKE PEER REVIEW: Early career researchers slightly more likely to

Back to contents How to maintain a healthy peer review system Results by geographic

HOW TO MAINTAIN PEER REVIEW SYSTEM: Clear guidance for reviewers is considered the most

HOW TO MAINTAIN PEER REVIEW SYSTEM: Researchers in North America and Western Europe more

HOW TO MAINTAIN PEER REVIEW SYSTEM: 41% of researchers from the Middle East want

HOW TO MAINTAIN PEER REVIEW SYSTEM: Early career researchers are slightly more likely to

Back to contents ADDITIONAL INFORMATION FOR ASSESSING RESEARCH OUTPUTS Results by geographic region, country,

ASSESSING RESEARCH OUTPUTS: 90% of chemists and life scientists want to see a reproducibility

ASSESSING RESEARCH OUTPUTS: 57% Materials Scientists think news articles in popular press would be

ASSESSING RESEARCH OUTPUTS: German researchers more likely to want signifier of reproducibility; researchers in

ASSESSING RESEARCH OUTPUTS: Researchers in India more likely to find all indicators helpful; researchers

ASSESSING RESEARCH OUTPUTS: North American researchers are more likely to find signifiers to show

ASSESSING RESEARCH OUTPUTS: Researchers in Africa are more likely to want a signifier of

ASSESSING RESEARCH OUTPUTS: Early career researchers are slightly more interested in seeing a signifier

Back to contents TRUSTWORTHINESS OF RESEARCH OUTPUTS Results by geographic region, country, broad subject

TRUSTWORTHINESS OF RESEARCH OUTPUTS: Computer scientists have least confidence in the research outputs they

TRUSTWORTHINESS OF RESEARCH OUTPUTS: Researchers from China, Japan and India are most polarised on

TRUSTWORTHINESS OF RESEARCH OUTPUTS: Latin American researchers most likely to trust research outputs, while

TRUSTWORTHINESS OF RESEARCH OUTPUTS: Little difference by age and gender. Early career researchers slightly

Back to contents TRUSTWORTHINESS OF RESEARCH PROFILES Results by geographic region, country, broad subject

TRUSTWORTHINESS OF RESEARCH PROFILES: XX Engineering and Materials Science researchers most likely to trust

TRUSTWORTHINESS OF RESEARCH PROFILES: Chinese researchers most likely to trust all profiles (41%), while

TRUSTWORTHINESS OF RESEARCH PROFILES: 46% of Latin America researchers trust all research profiles they

TRUSTWORTHINESS OF RESEARCH PROFILES: More than a third (35%) of early career researchers more

Back to contents SOURCING TRUSTED CONTENT Results by geographic region, country, broad subject area,

SOURCING TRUSTED CONTENT: Being able to adjust/choose search algorithm/parameters is considered most effective way

SOURCING TRUSTED CONTENT: Chinese researchers rate all solutions as more effective than average. Researchers

66 SOURCING TRUSTED CONTENT: Researchers in North America and Eastern Europe less likely to

SOURCING TRUSTED CONTENT: Source neutrality is less important among female researchers and those aged

Back to contents WHAT WILL HELP MOST TO IMPROVE COMMUNICATING TO THE PUBLIC Results

COMMUNICATING TO THE PUBLIC: Explanations in lay terms considered most helpful way to communicate

Back to contents PUBLIC CONFIDENCE IN RESEARCH EVIDENCE Results by geographic region, country, broad

PUBLIC CONFIDENCE IN RESEARCH EVIDENCE: Life Scientists and Earth/Environmental scientists slightly more likely to

PUBLIC CONFIDENCE IN RESEARCH EVIDENCE: SSE/AH and medical/health researchers more likely to consider volume

PUBLIC CONFIDENCE IN RESEARCH EVIDENCE: Nearly all researchers in Germany and USA consider misinterpretation

PUBLIC CONFIDENCE IN RESEARCH EVIDENCE: Misinterpretation of research outcomes in media/policy/ public discussion and

PUBLIC CONFIDENCE IN RESEARCH EVIDENCE: Researchers in North America and Western Europe more likely

PUBLIC CONFIDENCE IN RESEARCH EVIDENCE: African researchers more concerned about the number of non-peer

PUBLIC CONFIDENCE IN RESEARCH EVIDENCE: Concerns about volume of information and non-peer reviewed research

N=3133 Demographics Age Subject Chemistry 5% Computer Science 6% Earth & Env. Science 8%

Back to contents Why journal articles viewed are not trustworthy 86

87 Verbatim analysis – Trust in content “. . . Why do you think

88 Verbatim analysis – Trust in content “. . . Why do you think

89 Verbatim analysis – Trust in content “. . . Why do you think

90 Verbatim analysis – Trust in content “. . . Why do you think

91 Verbatim analysis – Trust in content “. . . Why do you think

92 Verbatim analysis – Trust in content “. . . Why do you think

93 Verbatim analysis – Trust in content “. . . Why do you think

94 Verbatim analysis – Trust in content “. . . Why do you think

Back to contents Why research-related profiles viewed are not trustworthy 95

96 Verbatim analysis – Trust in researcher profiles “. . . Why do you

97 Verbatim analysis – Trust in researcher profiles “. . . Why do you

98 Verbatim analysis – Trust in researcher profiles “. . . Why do you

99 Verbatim analysis – Trust in researcher profiles “. . . Why do you

Verbatim analysis – Trust in researcher profiles 100 “. . . Why do you

101 Verbatim analysis – Trust in researcher profiles “. . . Why do you

Slides: 101

Download presentation

Quality, trust and peer review: researchers’ perspectives 10 years on Survey results September, 2019

Table of Contents Section Slide Number 1. Overview of approach 3 2. Executive Summary 4 3. Overview of results 6 4. Attitudes to peer review 19 5. Who researchers think should undertake peer review 29 6. How to maintain a healthy peer review system 37 7. Additional information for assessing research outputs 45 8. Trustworthiness of research outputs 53 9. Trustworthiness of research-related profiles 58 10. Sourcing trusted content 63 11. Communicating to the public 68 12. Public confidence in research evidence 76 13. Demographics 84 14. Appendix: Verbatim responses 86

Back to contents Background approach Research objectives are to: • Examine the drivers and influences on the communication of scholarly research. • This report is about peer review, assessment of research content and helping people outside the research community judge the quality of research evidence About the survey About Sense About Science • 3133 researchers responded to a survey of 98160 individuals randomly selected from database • Sense about Science is an independent of 3. 6 million researchers (3. 2 % response rate). • Survey tool: Co-branded (Elsevier and Sense About Science) online survey available in English charity that challenges misrepresentation of science and evidence in public life. only. Survey took 15 minutes to complete (median average). Fieldwork took place in May 2019. • Results: Responses have been weighted to be representative of the global researcher population by country (UNESCO 2014 data). Base sizes shown in this report are unweighted unless otherwise stated • Statistical testing: Maximum error margin for 3133 responses is ± 1. 5% at 90% confidence levels. When comparing main group and sub-group we have used a Z-test of proportion to identify differences between the overall average and the sub-group (90% confidence levels). Differences are indicated by a tick . A green tick indicates the subgroup result is higher than the overall result while a red tick indicates it lower. 18. 09. 2020 About Elsevier • A global information analytics business specializing in science and health helping institutions and professionals progress science, advance healthcare and improve performance for the benefit of humanity. 3

Back to contents EXECUTIVE SUMMARY 4

Back to contents EXECUTIVE SUMMARY Trust in peer review and communication Satisfaction with peer review has increased significantly since 2015 from 65% to 75%. The shift has been mainly from ‘neutral’ to ‘satisfied’. The main reasons outputs are considered untrustworthy are poor interpretation, lack of clarity of the peer review process and flaws in the methodology Peer review is widely seen to be improving the quality of research and keeping scientific communication under control. , though only 38% believe it is well understood by the public 70% of researchers doubt the quality of at least some of the research profiles they encounter. To compensate for this they use multiple platforms or visit websites of institutions. Reasons for untrustworthiness include exaggeration of achievements, lack of verification and poor quality outputs. The preferred format for peer review is two relevant reviewers either with or without input from a member of the editorial team; only 8% think AI/ML would qualify as peer review. Clear guidance on reviewing is considered most important for maintaining a healthy peer review system; only 28% think financial rewards for reviewers are needed. Researchers would like to see a metric of ‘reproducibility’ to help assess the quality of research outputs 86% of researchers doubt the quality of at least some of the research outputs they encounter. To compensate for this they check supplementary material/data carefully, read only information associated with peer reviewed journals or seek corroboration from other trusted sources. 18. 09. 2020 When using tools to find research outputs being able to adjust algorithms/parameters, transparency and interoperability are seen as effective ways to be directed to trusted content When communicating research outcomes to the public explaining research in lay terms is seen as the best way to help people outside the research community judge the quality of research The main causes of lack of public confidence in research evidence are seen to be misinterpretation or deliberate misrepresentation by the media as well as difficulty identifying high quality research. 5

Back to contents OVERVIEW OF RESULTS 6

SATISFACTION WITH PEER REVIEW: Has increased significantly since 2009 from 69% to 75%. Shift mainly from ‘neutral’ to ‘satisfied’. Overall, how satisfied are you with the peer review system used by scholarly journals? Satisfaction with peer review, trend Very Satisfied 100% % satisfied Satisfied 7%1% 13% Neither satisfied nor dissatisfied Very Dissatisfied 16% 90% Dissatisfied 80% 75% 70% 2019 63% 75% satisfied 8% dissatisfied Base: All respondents, 2019 (n=3133), 2015 (n=1988), 2009 (n=4037), 2007 (n=3040) 69% 60% 50% 40% 2009 2019 7

8 ATTITUDES TO PEER REVIEW: widely seen to be improving the quality of research and keeping scientific communication under control. Only 38% believe it is well understood by the public Please indicate the extent to which you agree or disagree with the following statements Ordered by impact on overall satisfaction with peer review Strongly Agree Neither agree nor disagree 37% Peer review improves the quality of research articles overall The time it took for peer review on my last article was reasonable 11% Peer review is a concept well understood by the public 9% Peer review is a concept well understood by researchers Without peer review there is no control in scientific communication Data and supporting material that accompany research articles should be reviewed Base: All respondents (n=3133) 18. 09. 2020 Disagree 90% 2% 66% 18% 34% 8% 4% 1% 88% 4% 10% 4% 1% 85% 5% 5% 1% 76% 6% Strongly Disagree 8% 2% 15% 27% 36% 13% 26% 39% 46% 5% 8% 52% 46% 30% % disagree 53% 55% 29% % agree 18%

PEER REVIEW SYSTEM: Who should undertake peer review? Most (87%) believe an evaluation of research by two relevant researchers either with or without input from a member of the editorial team qualifies as peer review, only 8% think AI/ML would qualify as peer review. Which of the below qualifies as peer review; does it qualify when the evaluation of a research article is undertaken by: 70% Member of editorial team and two relevant researchers 58% Two relevant researchers (not part of editorial team) 38% Member of editorial team and one relevant researcher earch published online and evaluated by other researchers by posting comments and rating the manuscript 30% r more members of the editorial team (e. g. editor and/or board members) ne member of the editorial team (e. g. editor or board member) Machines using artificial intelligence/machine learning Base: All respondents (n=3133) 27% 12% 8% A Type of peer review/ evaluation % respondents selecting B A or B 87% A+B 41% A ONLY 29% B ONLY 17% NEITHER A NOR B 13% C D E F G 9

10 PEER REVIEW SYSTEM: Clear guidance on reviewing is considered most important for maintaining a healthy peer review system; only 28% think financial rewards for reviewers are needed. Which of the below do you consider most important for maintaining a healthy system of peer review? (select top 5) 66% Clear guidance given to reviewers on criteria for reviewing 45% Employer recognition of time spent reviewing Reviewer(s) reports made available alongside research article (reviewer can choose to remain anonymous) 38% Greater diversity within the peer review community 38% 37% Formal training of reviewers to create a quality benchmark 34% Accreditation given to reviewers (CME/CPD points) Acknowledgement for the reviewer (e. g. reviewer name(s) published on the article) 28% Financial reward for reviewers 28% 25% All researchers who publish an article undertake a review within a year Time taken for peer review on a research article is 25% less than it is today (submission to final decision) Software is used to triage manuscripts prior to formal peer review Base: All respondents (n=3133) 21% 17%

ASSESSMENT OF ARTICLES: Researchers would like to see a metric of ‘reproducibility’ to help assess the quality of research outputs When a reader is interacting with research outputs (e. g. research articles, preprints, data), what additional information is helpful for assessing those research outputs? Very helpful Helpful Neither helpful nor unhelpful 33% Citations Indicator to show whether someone else tried to reproduce the research and their success Post publication commentary (from research community) Clear signifier to show content has been peer reviewed Downloads 88% 55% 15% 3% 82% 61% 17% 3% 79% 21% 56% 20% 2% 77% 23% 6% 2% 69% 27% 5%1% 68% 3% 56% 13% 4% 50% 13% 4% 42% 8% 31% 19% Peer reviewers' rating(s) (e. g. score out of ten) 10% 53% 46% 9% Number of collaborators on research 7% Number of retweets and/or mentions on blogs 5% 18. 09. 2020 2% 10% 19% 14% Base: All respondents (n=3133) % helpful Very unhelpful 55% 26% Indicator to show number of times the research was included in systematic reviews News articles in popular press Unhelpful 31% 41% 33% 35% 26% 42% 43% 11% 19% 11

TRUSTWORTHINESS OF RESEARCH OUTPUTS: Most researchers 62% trust the majority of research outputs they encounter but a third doubt the quality of at least half. To compensate for this they check supplementary material/data carefully, read only information associated with peer reviewed journals or seek corroboration from other trusted sources. Thinking about the various research outputs that you interacted with (or encountered) last week what proportion of the outputs would you consider trustworthy? 0% 1% 10% 21% 20% 30% 15% 40% Some of them 37% About half of them 48% 70% 80% 90% 57% The majority of them All of them Only read/access content that is in or linked to a peer reviewed journal 52% Seek corroboration from other trusted sources (e. g. see if research is cited in a known journal) 52% 37% Read/access research from researchers I know 50% 60% Which of the following mechanisms do you employ to compensate for any lack of confidence you have in the content you are considering reading/accessing? Check supplementary material or data carefully None of them 62% 29% Read/access research from specific institutes Only read/access research that has been personally recommended 6% Other (please specify) 7% 14% 100% Research outputs Base: All respondents (n=3133) Base: All respondents that do not think all research outputs are trustworthy (n=2715) 18. 09. 2020 12

REASONS RESEARCH OUTPUTS ARE REGARDED AS UNTRUSTWORTHY: for those 13 researchers who rate at least some outputs as untrustworthy; the main reasons are lack of or low quality peer review, uncertainty if content is peer reviewed, poor interpretation and flaws in the methodology You indicated that some/all of the research outputs are trustworthy. Why do you think all/some of the content you encounter is not trustworthy? Inhibitors of trust Unclear if peer reviewed Not peer reviewed Low quality peer review Peer review scope "Not familiar with the journals or media/not sure what the peer review process might be, or how reviewers are selected" (Psychology, United States of America, 26 to 35) Forces driving volume of research Components of mistrust Pressure to publish New research outputs (preprints, data) Unsupported claims: Poor conclusions drawn Reduced focus on novel/ high-quality research Predatory journals Growth in researchers Errors: inflated statistical power/ grammatical/calculations New channels: (social media, media outlets) Open Science Methodological flaws Lack of supplementary material 18. 09. 2020 “Content that comes from predatory journals is not trustworthy. It is not rigorously peer reviewed. This diminishes the trust in scientific research. As scientists we need to be held to a high standard. The traditional peer review system does that. " (Physician, USA, prefer not to say age) Biases (in peer review, funding, negative findings not published) "Experiments poorly designed, some analyses seemed suspect, areas that I know well improperly characterized" (Environmental Sciences, Canada, Over 65) "There is published Research biased by financial or other Support to the authors and not properly declared. " (Medicine and Allied Health, Switzerland, 56 to 65) "Authors often do not provide data/code/tools/proper description of the scenarios used for the evaluation contained in their papers. In particular, the correctness of code used for simulations reported in papers is often unverifiable. " (Computer Sciences / IT, Brazil, 26 to 35)

TRUSTWORTHINESS OF RESEARCH PROFILES: Most researchers (75%) trust at least the majority of research outputs they encounter, however 25% of researchers doubt the quality of at least some of the research profiles they encounter. To compensate for this they use multiple platforms or visit websites of institutions. Now thinking about the various research-related profiles that you interacted with (or encountered) last week (including researcher and institutional profiles), what proportion of the profiles would you consider trustworthy 0% 1% 10% 14% 20% 10% Some of them 30% 40% 50% 45% 60% 30% Which of the following mechanisms do you employ to compensate for any lack of confidence you have in the content you are considering reading/accessing? Use multiple platforms (e. g. Scopus, Research. Gate, Google Scholar) 79% 25% About half of them 54% Go to the website(s) of the institution(s) The majority of them 70% 80% None of them 14 75% All of them Other (please specify) 90% 9% 100% Research outputs Base: All respondents that encountered research-related profiles in the last week (n=2588) 18. 09. 2020 Base: All respondents that do not think all research –related profiled are trustworthy (n=1825)

Reasons research-related profiles are regarded as untrustworthy: among those that question the quality of at least some research output; the biggest problems are exaggeration of achievements, lack of verification and poor quality outputs 15 You indicated that some/all of the research-related profiles (researcher and institutional profiles) are trustworthy. Why do you think some/all of the profiles you encounter are not trustworthy? Exaggeration (self promotion, falsification) Quality of research (reputation of associated journals, lack of metrics) Unverified (references, endorsements) Out of date or poor profiles (errors, duplicate profiles, incomplete) Reputation Source of profile (Not reputable / open to edit) (associated with the institution, author, country) "Both researchers and institutions tend to exaggerate their achievements. " (Chemistry, Poland, 46 to 55) “The low quality of papers published" (Medicine and Allied Health, Indonesia, Over 65) "If I am contacted by someone with a profile disconnected from everybody, I do not trust it" (Engineering and Technology, Switzerland, 26 to 35) "Many are updated infrequently. Others appear to be updated semi-automatically and contain errors. " (Biological Sciences, United States of America, 36 to 45) "I only trust top ranked outlets" (Earth and Planetary Sciences, Canada, Over 65) More comments from respondents can be found here 18. 09. 2020

SOURCING TRUSTED CONTENT: being able to adjust algorithms/parameters, transparency and interoperability seen as most effective ways to be directed to trusted content Please indicate which of the below solutions you believe would be effective in ensuring you are directed to appropriate trusted content? Very Effective You are able to adjust/choose search algorithms and parameters Neither effective or ineffective 22% Ineffective % ineffective 78% 4% Very ineffective 56% 19% 3% When platforms serve you an article to read, they explain why they have selected it 15% 56% 23% 4% 1% 71% 6% The content is inter-operable across databases, i. e. move seamlessly between outputs 17% 54% 25% 3% 1% 71% 3% The information is source neutral (it is not from specific journals or publishers) 16% 62% 11% Base: All respondents (n=3133) 18. 09. 2020 46% 27% 9% 3% 16

COMMUNICATION TO THE PUBLIC: Explaining research in lay terms is seen as the best way to help people outside the research community judge the quality of research To help people outside the research community judge the quality of research articles they view, which of the following would be most helpful Explain research context, findings and implications in lay terms 70% Enable them to ask questions of the authors (e. g. discussion to be posted alongside the article) 40% Clear explanation of peer review and make it easily accessible from content (e. g. how many people involved, their role) 38% Peer reviewed content is clearly marked as such 35% Enable updates to the article after publication (e. g. as more work is done on the topic by author) 35% Provide guidance on statistics (e. g. probability) Base: All respondents (n=3133) 30% 17

PUBLIC CONFIDENCE IN RESEARCH EVIDENCE: biggest problem seen to be misinterpretation or deliberate misrepresentation by the media as well as difficulty identifying high quality research. Thinking about public confidence in research evidence, how much, if at all, do you believe any of the following are a problem? Large Medium Small Not a problem at all 49% Misinterpretation of research outcomes in media, policy or public discussion 43% 30% 41% 37% Deliberate misrepresentation by media Increased low quality research available (i. e. research meets minimum technical standard only) 33% Difficulty in distinguishing high quality research 38% Too many non-peer reviewed research publications 37% 35% Misinterpretation of research outcomes by researchers/their institutions 26% 37% Contradictory research findings published (e. g. research from different groups have opposing conclusions) Increased alternative sources (e. g. different versions of articles: preprint, accepted, published) Large number of competing platforms providing access to the same research outputs Base: All respondents (n=3133) 18. 09. 2020 21% 13% 11% 24% 8% 28% 8% 15% 30% 26% 6% 9% 23% 32% 25% 17% 20% 30% 28% 6% 17% 33% Volume of information 4% 21% 37% 33% Deliberate misrepresentation by researchers/their institutions 14% 7% 23% 31% 39% 18

Back to contents Attitudes to peer review Results by geographic region, country, broad subject area, age group and gender 19

SATISFACTION WITH PEER REVIEW: lowest in Western Europe and North America. Also lower than average in SSE/AH Overall, how satisfied are you with the peer review system used by scholarly journals? Specialty Age & gender Country (top 10 by researcher count) Total n=3133 75% Chemistry n=146 Total n=3133 75% China n=363 80% Computer Science n=165 75% Earth & Env. Science n=257 73% Engineering n=443 80% Life Sciences n=602 70% Material Science n=129 87% Maths n=116 Medicine and Allied Healthn=449 83% Under 36 n=492 74% 36 -54 n=1581 76% 55+ n=1001 76% USA n=558 0% 100% 72% Russia n=137 78% Germany n=83 61% UK n=153 61% France n=76 59% 77% 71% 68% Japan n=135 75% Female n=932 76% 0% 100% Significant difference to 2012 result ( p=90%) Region % satisfied 75% 90% Republic of Korea n=67 Male n=2107 Total n=3133 75% Physics & Astronomy n=220 Soc. Sci + Arts Hum + Economics n=587 75% Higher Significant difference 20 between subset and Lower total (p=90%) 72% India n=156 84% Canada n=98 69% 0% 100% Africa n=97 87% APAC n=926 82% Eastern Europe n=303 75% Latin America n=236 75% 83% Middle East n=117 North America n=656 68% Western Europe n=778 65% 0% 100%

ATTITUDES TO PEER REVIEW: SSE/AH least accepting of data and supporting materials being reviewed and least likely to think the time taken to review was reasonable Higher Significant difference between subset and Lower total (p=90%) Significant difference to 2012 result ( p=90%) Please indicate the extent to which you agree or disagree with the following statements: % AGREE Peer review improves the quality of research articles overall Data and supporting material that accompany research articles should be reviewed Total n=3133 90% Total n=3133 Chemistry n=146 91% Chemistry n=146 Computer Science n=165 84% Earth & Env. Science n=257 94% Computer Science n=165 Earth & Env. Science n=257 76% 90% The time it took for peer review on my last article was reasonable Total n=3133 72% 80% Chemistry n=146 63% Earth & Env. Science n=257 66% 88% Engineering n=443 Life Sciences n=602 90% Life Sciences n=602 81% Material Science n=129 Maths n=116 Medicine and Allied Healthn=449 78% Material Science n=129 79% Maths n=116 Medicine and Allied Healthn=449 Physics & Astronomy n=220 Soc. Sci + Arts Hum + Economics n=587 85% Maths n=116 Medicine and Allied Healthn=449 95% 93% Physics & Astronomy n=220 Soc. Sci + Arts Hum + Economics n=587 91% 87% 0% 100% Engineering n=443 73% 80% Physics & Astronomy n=220 Soc. Sci + Arts Hum + Economics n=587 74% 65% 0% 100% 77% Computer Science n=165 Engineering n=443 Material Science n=129 66% 71% Life Sciences n=602 66% 77% 60% 64% 67% 59% 0% 100% 21

ATTITUDES TO PEER REVIEW: SSE/AH least likely to think peer review is well understood by researchers and the public Higher Significant difference between subset and Lower total (p=90%) Please indicate the extent to which you agree or disagree with the following statements: % AGREE Without peer review there is no control in scientific communication Peer review is a concept well understood by researchers Peer review is a concept well understood by the public Total n=3133 85% Total n=3133 88% Total n=3133 Chemistry n=146 88% Chemistry n=146 90% Chemistry n=146 84% Computer Science n=165 Earth & Env. Science n=257 83% Earth & Env. Science n=257 90% Earth & Env. Science n=257 Engineering n=443 86% Engineering n=443 89% Engineering n=443 Life Sciences n=602 86% Life Sciences n=602 89% Life Sciences n=602 Material Science n=129 92% Material Science n=129 89% Maths n=116 Medicine and Allied Healthn=449 92% 88% Physics & Astronomy n=220 Soc. Sci + Arts Hum + Economics n=587 0% 82% 100% Maths n=116 Medicine and Allied Healthn=449 86% 90% 84% 0% 100% 50% 45% 40% 52% 31% 54% 32% Physics & Astronomy n=220 Soc. Sci + Arts Hum + Economics n=587 47% 20% 0% 58% Maths n=116 Medicine and Allied Healthn=449 89% Physics & Astronomy n=220 Soc. Sci + Arts Hum + Economics n=587 38% Computer Science n=165 86% Significant difference to 2012 result ( p=90%) 100% 22

Higher Significant difference between subset and Lower total (p=90%) ATTITUDES TO PEER REVIEW: Only half of French researchers think the time taken for peer review is reasonable Please indicate the extent to which you agree or disagree with the following statements: % AGREE Peer review improves the quality of research articles overall Total n=3133 90% China n=363 94% USA n=558 Total n=3133 91% Japan n=135 87% Russia n=137 91% Germany n=83 80% Data and supporting material that accompany research articles should be reviewed China n=363 73% Japan n=135 72% Russia n=137 76% Russia n=137 70% Germany n=83 86% UK n=153 96% Canada n=98 92% 0% 100% 78% China n=363 Japan n=135 UK n=153 India n=156 66% USA n=558 Republic of Korea n=67 Total n=3133 75% 82% 81% The time it took for peer review on my last article was reasonable USA n=558 Republic of Korea n=67 France n=76 76% Significant difference to 2012 result ( p=90%) 72% Germany n=83 67% 72% France n=76 India n=156 93% Canada n=98 68% 0% Republic of Korea n=67 63% 100% 66% France n=76 India n=156 Canada n=98 56% UK n=153 82% 63% 50% 71% 62% 0% 100% 23

ATTITUDES TO PEER REVIEW: only 14% of researchers in the UK and USA think peer review is well understood by the public Higher Significant difference between subset and Lower total (p=90%) Please indicate the extent to which you agree or disagree with the following statements: % AGREE Without peer review there is no control in scientific communication Peer review is a concept well understood by researchers Total n=3133 85% Total n=3133 China n=363 86% China n=363 USA n=558 84% USA n=558 Japan n=135 81% 88% 93% 83% Japan n=135 95% Peer review is a concept well understood by the public Total n=3133 China n=363 USA n=558 Japan n=135 Russia n=137 87% Russia n=137 Germany n=83 84% Germany n=83 Republic of Korea n=67 84% Republic of Korea n=67 90% Republic of Korea n=67 UK n=153 81% UK n=153 89% UK n=153 France n=76 83% France n=76 91% France n=76 India n=156 91% Canada n=98 87% 0% 100% 88% 80% 68% India n=156 Canada n=98 89% Canada n=98 14% 44% 63% Germany n=83 87% 100% 38% Russia n=137 India n=156 0% Significant difference to 2012 result ( p=90%) 32% 38% 14% 22% 41% 16% 0% 100% 24

ATTITUDES TO PEER REVIEW: Western European researchers are least likely to believe peer review improves the quality of articles and think the time taken to review with last article was reasonable Higher Significant difference between subset and Lower total (p=90%) Significant difference to 2012 result ( p=90%) Please indicate the extent to which you agree or disagree with the following statements: % AGREE Peer review improves the quality of research articles overall Total n=3133 90% Africa n=97 95% Data and supporting material that accompany research articles should be reviewed Total n=3133 Africa n=97 APAC n=926 76% 73% Total n=3133 66% Africa n=97 91% Eastern Europe n=303 90% Eastern Europe n=303 75% Eastern Europe n=303 Latin America n=236 92% Latin America n=236 76% Latin America n=236 61% Middle East n=117 89% Middle East n=117 76% Middle East n=117 63% North America n=656 91% North America n=656 74% North America n=656 62% Western Europe n=778 73% Western Europe n=778 84% 0% 100% APAC n=926 70% APAC n=926 Western Europe n=778 78% The time it took for peer review on my last article was reasonable 0% 73% 65% 58% 100% 25

ATTITUDES TO PEER REVIEW: researchers in North America, Western Europe and Latin America are least likely to think the public understand peer review well Higher Significant difference between subset and Lower total (p=90%) Significant difference to 2012 result ( p=90%) Please indicate the extent to which you agree or disagree with the following statements: % AGREE Without peer review there is no control in scientific communication Total n=3133 85% Africa n=97 Peer review is a concept well understood by researchers Total n=3133 90% 88% Africa n=97 79% Total n=3133 38% Africa n=97 APAC n=926 54% APAC n=926 85% APAC n=926 Eastern Europe n=303 87% Eastern Europe n=303 86% Eastern Europe n=303 Latin America n=236 85% Latin America n=236 Middle East n=117 Latin America n=236 91% 92% Peer review is a concept well understood by the public Middle East n=117 86% Middle East n=117 83% North America n=656 85% North America n=656 84% Western Europe n=778 83% Western Europe n=778 0% 100% 87% 0% 100% North America n=656 Western Europe n=778 0% 33% 27% 41% 14% 24% 100% 26

Higher Significant difference between subset and Lower total (p=90%) ATTITUDES TO PEER REVIEW: Female researchers are least likely to want data and supporting materials to be reviewed Significant difference to 2012 result ( p=90%) Please indicate the extent to which you agree or disagree with the following statements: % AGREE Peer review improves the quality of research articles overall Data and supporting material that accompany research articles should be reviewed The time it took for peer review on my last article was reasonable Total n=3133 90% Total n=3133 76% Total n=3133 66% Under 36 n=492 88% Under 36 n=492 76% Under 36 n=492 65% 36 -54 n=1581 90% 36 -54 n=1581 76% 36 -54 n=1581 66% 55+ n=1001 90% 55+ n=1001 75% 55+ n=1001 67% Male n=2107 89% Male n=2107 78% Male n=2107 67% Female n=932 91% Female n=932 65% 0% 100% Female n=932 71% 0% 100% 27

Higher Significant difference between subset and Lower total (p=90%) ATTITUDES TO PEER REVIEW: Female researchers and those aged 55+ least likely to think the public understand peer review Significant difference to 2012 result ( p=90%) Please indicate the extent to which you agree or disagree with the following statements: % AGREE Without peer review there is no control in scientific communication Total n=3133 86% Peer review is a concept well understood by researchers Total n=3133 88% Peer review is a concept well understood by the public Total n=3133 38% Under 36 n=492 39% 40% Under 36 n=492 83% Under 36 n=492 36 -54 n=1581 85% 36 -54 n=1581 89% 36 -54 n=1581 55+ n=1001 88% 55+ n=1001 89% 55+ n=1001 Male n=2107 86% Male n=2107 89% Male n=2107 Female n=932 85% Female n=932 87% Female n=932 0% 100% 84% 0% 100% 35% 41% 32% 0% 100% 28

Back to contents Who researchers think should undertake peer review Results by geographic region, country, broad subject area, age group and gender 29

WHO RESEARCHERS THINK SHOULD UNDERTAKE PEER REVIEW: Earth/environmental science researchers more likely to consider postpublication commentary/ratings as peer review 30 Which of the below qualifies as peer review; does it qualify when the evaluation of a research article is undertaken by: Chemistry Member of editorial team and two relevant researchers Computer Science 73% 68% Two relevant researchers (not part of editorial team) 64% 62% Member of editorial team and one relevant researcher 32% 30% Research published online and evaluated by other researchers by posting comments and rating the manuscript Two or more members of the editorial team (e. g. editor and/or board members) 67% 61% 37% 34% 24% Earth & Env. Science Engineering 28% Global 75% 70% 61% 49% 56% 33% 35% 34% 45% 28% Life Sciences 21% 58% 38% 25% 30% 26% 27% One member of the editorial team (e. g. editor or board member) 12% 13% 10% 13% 8% 12% Machines using artificial intelligence/machine learning 10% 14% 8% 13% 7% 8% n=146 n=165 n=257 n=443 n=602 n=3133

WHO RESEARCHERS THINK SHOULD UNDERTAKE PEER REVIEW: SSE/AH more likely to 31 expect peer review to involve review by a researcher or more than one editor before publication, but less likely to accept post-publication commentary as peer review. 20% of materials scientists think review by one editor is acceptable Which of the below qualifies as peer review; does it qualify when the evaluation of a research article is undertaken by: : Materials Science Member of editorial team and two relevant researchers 58% Two relevant researchers (not part of editorial team) 55% Member of editorial team and one relevant researcher 36% Research published online and evaluated by other researchers by posting comments and rating the manuscript 38% Two or more members of the editorial team (e. g. editor and/or board members) One member of the editorial team (e. g. editor or board member) Machines using artificial intelligence/machine learning Maths 62% 48% 37% 28% 9% n=129 18% 4% 36% Soc. Sci + Arts Hum + Economics 71% 76% 58% 56% 47% 28% 70% 49% 25% 8% n=449 21% 14% 7% n=220 Global 70% 58% 38% 22% 27% 7% n=116 Physics & Astronomy 73% 50% 27% 20% Medicine and Allied Health 39% 13% 6% n=587 30% 27% 12% 8% n=3133

WHO RESEARCHERS THINK SHOULD UNDERTAKE PEER REVIEW: Researchers in China 32 less likely to expect peer review to be undertaken by at least one relevant researcher, but more likely to accept post-publication commentary/ratings as a valid form of peer review; the reverse view is found in the USA Which of the below qualifies as peer review; does it qualify when the evaluation of a research article is undertaken by: China Member of editorial team and two relevant researchers 49% Research published online and evaluated by other researchers by posting comments and rating the manuscript Two or more members of the editorial team (e. g. editor and/or board members) 76% 48% 30% 39% 22% 15% Machines using artificial intelligence/machine learning 16% n=363 Russia 61% 69% 39% 8% 2% n=558 n=135 63% 49% 23% 21% 12% 75% 36% 23% 10% Germany 50% 44% 33% 22% 40% One member of the editorial team (e. g. editor or board member) Japan 90% 56% Two relevant researchers (not part of editorial team) Member of editorial team and one relevant researcher USA 11% 7% n=137 Global 70% 58% 38% 29% 30% 28% 27% 11% 12% 8% 4% n=83 n=3133

WHO RESEARCHERS THINK SHOULD UNDERTAKE PEER REVIEW: Researchers in India most likely to accept post-publication commentary, review by a single editor and AI as forms of peer review 33 Which of the below qualifies as peer review; does it qualify when the evaluation of a research article is undertaken by: Republic of Korea UK Member of editorial team and two relevant researchers 70% 86% Two relevant researchers (not part of editorial team) 43% 85% Member of editorial team and one relevant researcher Two or more members of the editorial team (e. g. editor and/or board members) One member of the editorial team (e. g. editor or board member) Machines using artificial intelligence/machine learning 35% 21% 12% 13% 0% 9% n=67 68% 9% 35% 41% 13% n=76 n=156 Global 78% 70% 78% 58% 35% 19% 27% 18% 7% n=153 48% 30% 20% Canada 64% 38% 29% 31% India 72% 50% 28% Research published online and evaluated by other researchers by posting comments and rating the manuscript France 28% 30% 27% 8% 12% 5% 8% n=98 n=3133

WHO RESEARCHERS THINK SHOULD UNDERTAKE PEER REVIEW: 37% of North American researchers consider review by 2+ editorial board members an acceptable form of peer review 34 Which of the below qualifies as peer review; does it qualify when the evaluation of a research article is undertaken by: North America Member of editorial team and two relevant researchers 88% Two relevant researchers (not part of editorial team) Member of editorial team and one relevant researcher Research published online and evaluated by other researchers by posting comments and rating the manuscript Machines using artificial intelligence/machine learning 76% 48% 70% 43% 21% Two or more members of the editorial team (e. g. editor and/or board members) One member of the editorial team (e. g. editor or board member) Western Europe 28% 10% 3% 4% n=656 Eastern Europe 61% 66% 47% 32% 14% n=926 Global 70% 58% 38% 40% 35% 23% 13% n=778 52% 35% 28% 37% 8% APAC 22% 13% 7% 30% 27% 12% 8% n=303 n=3133

WHO RESEARCHERS THINK SHOULD UNDERTAKE PEER REVIEW: Researchers from Africa are less likely to consider post-publication commentary as a valid form of peer review Which of the below qualifies as peer review; does it qualify when the evaluation of a research article is undertaken by: Latin America Member of editorial team and two relevant researchers 70% Two relevant researchers (not part of editorial team) Member of editorial team and one relevant researcher Middle East 56% 36% 61% 33% 31% Two or more members of the editorial team (e. g. editor and/or board members) 30% 31% Machines using artificial intelligence/machine learning 5% 10% 11% n=236 56% n=117 Global 70% 58% 36% 30% 15% 71% 48% Research published online and evaluated by other researchers by posting comments and rating the manuscript One member of the editorial team (e. g. editor or board member) Africa 20% 26% 13% 4% n=97 30% 27% 12% 8% n=3133 35

WHO RESEARCHERS THINK SHOULD UNDERTAKE PEER REVIEW: Early career researchers slightly more likely to accept post-publication commentary and AI/ML as valid forms of peer review than older researchers 36 Which of the below qualifies as peer review; does it qualify when the evaluation of a research article is undertaken by: Under 36 Member of editorial team and two relevant researchers 36% Research published online and evaluated by other researchers by posting comments and rating the manuscript Two or more members of the editorial team (e. g. editor and/or board members) 15% Machines using artificial intelligence/machine learning 12% n=492 12% 9% n=1581 38% 5% n=1001 61% 27% 13% 9% n=2107 Global 70% 58% 39% 32% 27% 10% 74% 58% 27% 29% One member of the editorial team (e. g. editor or board member) 43% Female 68% 61% 34% Male 73% 58% 55% 37% 56+ 69% Two relevant researchers (not part of editorial team) Member of editorial team and one relevant researcher 36 -55 25% 30% 28% 27% 9% 6% n=932 12% 8% n=3133

Back to contents How to maintain a healthy peer review system Results by geographic region, country, broad subject area, age group and gender 37

HOW TO MAINTAIN PEER REVIEW SYSTEM: Clear guidance for reviewers is considered the most important way to maintain peer review system across all subject areas. 38% of computer scientists want financial rewards for reviewers 38 Which of the below do you consider most important for maintaining a healthy system of peer review? Chemistry Clear guidance given to reviewers on criteria for reviewing Employer recognition of time spent reviewing Reviewer(s) reports made available alongside research article (reviewer can choose to remain anonymous) 70% 62% Formal training of reviewers to create a quality benchmark 29% 62% 60% 49% 30% 34% 38% Acknowledgement for the reviewer (e. g. reviewer name(s) published on the article) 28% 38% Financial reward for reviewers 31% 38% All researchers who publish an article undertake a review within a year 30% Software is used to triage manuscripts prior to formal peer review Engineering 45% 41% 26% Time taken for peer review on a research article is 25% less than it is today (submission to final decision) Earth & Env. Science 42% Greater diversity within the peer review community Accreditation given to reviewers (CME/CPD points) Computer Science 19% 25% n=146 19% 18% 13% n=165 Life Sciences 67% 35% 46% Global 66% 45% 43% 38% 38% 41% 30% 41% 38% 34% 37% 37% 33% 36% 35% 34% 31% 26% 28% 27% 31% 26% 28% 29% 31% 25% 31% 19% 21% 37% 18% n=257 20% n=443 15% n=602 17% n=3133

HOW TO MAINTAIN PEER REVIEW SYSTEM: Clear guidance for reviewers is considered the most important way to maintain peer review system across all subject areas. 39 Which of the below do you consider most important for maintaining a healthy system of peer review? Materials Science Clear guidance given to reviewers on criteria for reviewing Employer recognition of time spent reviewing Reviewer(s) reports made available alongside research article (reviewer can choose to remain anonymous) 60% 40% 26% Greater diversity within the peer review community 34% Formal training of reviewers to create a quality benchmark 35% Accreditation given to reviewers (CME/CPD points) 34% Acknowledgement for the reviewer (e. g. reviewer name(s) published on the article) 29% Financial reward for reviewers 30% All researchers who publish an article undertake a review within a year 25% Time taken for peer review on a research article is 25% less than it is today (submission to final decision) 24% Software is used to triage manuscripts prior to formal peer review 17% n=129 Maths Medicine and Allied Health 51% 76% 40% Physics & Astronomy 67% 44% 46% 45% 40% 39% 31% 17% 50% 30% 25% 37% 29% 30% 23% 30% 21% 26% 17% n=116 28% 66% 31% 38% 45% 38% 42% 37% 34% 22% 25% 28% 28% 25% 21% 22% n=449 Global 45% 56% 28% 16% 13% 70% 43% 29% 19% Soc. Sci + Arts Hum + Economics 14% n=220 17% n=587 n=3133

HOW TO MAINTAIN PEER REVIEW SYSTEM: Clear guidance for reviewers is considered the most important way to maintain peer review system across all countries. 38% of researchers in Russia want financial rewards for reviewers 40 Which of the below do you consider most important for maintaining a healthy system of peer review? China Clear guidance given to reviewers on criteria for reviewing Employer recognition of time spent reviewing Reviewer(s) reports made available alongside research article (reviewer can choose to remain anonymous) 23% 32% Formal training of reviewers to create a quality benchmark 34% Accreditation given to reviewers (CME/CPD points) 31% Acknowledgement for the reviewer (e. g. reviewer name(s) published on the article) 33% All researchers who publish an article undertake a review within a year Time taken for peer review on a research article is 25% less than it is today (submission to final decision) Software is used to triage manuscripts prior to formal peer review 62% 29% 44% 26% 30% 24% n=363 Japan 75% 58% Greater diversity within the peer review community Financial reward for reviewers USA 43% 27% 20% 29% 18% 12% n=558 Germany 72% 70% 47% 33% 61% 43% 36% 48% Russia 42% 30% 34% 22% 28% 26% 38% 34% 37% 23% 29% 27% 23% 19% 20% 16% 24% 23% 13% n=135 n=137 26% 16% 45% 33% 22% 33% 66% 38% 40% 36% Global n=83 34% 28% 25% 21% 17% n=3133

HOW TO MAINTAIN PEER REVIEW SYSTEM: Clear guidance for reviewers is considered the most important way to maintain peer review system across all countries. 72% of researchers in the UK employer recognition. 41 Which of the below do you consider most important for maintaining a healthy system of peer review? Republic of Korea Clear guidance given to reviewers on criteria for reviewing Employer recognition of time spent reviewing Reviewer(s) reports made available alongside research article (reviewer can choose to remain anonymous) 51% 21% 46% France 75% 61% 72% 58% 39% Greater diversity within the peer review community 31% Formal training of reviewers to create a quality benchmark 30% Accreditation given to reviewers (CME/CPD points) 36% Acknowledgement for the reviewer (e. g. reviewer name(s) published on the article) 36% Financial reward for reviewers 28% All researchers who publish an article undertake a review within a year 27% Time taken for peer review on a research article is 25% less than it is today (submission to final decision) 25% Software is used to triage manuscripts prior to formal peer review UK 18% n=67 42% 35% 32% 26% 42% 29% 18% 8% n=153 62% 48% 36% 30% 12% 4% n=76 22% 29% 37% 36% 34% 32% 28% 51% 39% 45% 38% 50% 42% 66% 38% 37% 30% 23% 69% 40% 35% 40% Global Canada 55% 39% 38% 36% 46% India 18% 26% 20% 28% 25% 21% 25% 12% 17% n=156 n=98 n=3133

HOW TO MAINTAIN PEER REVIEW SYSTEM: Researchers in North America and Western Europe more likely than average to want employer recognition; researchers in APAC least likely to want this Which of the below do you consider most important for maintaining a healthy system of peer review? North America Clear guidance given to reviewers on criteria for reviewing 74% Employer recognition of time spent reviewing Reviewer(s) reports made available alongside research article (reviewer can choose to remain anonymous) Greater diversity within the peer review community 30% 61% 39% 44% 40% 28% 36% 22% 28% Financial reward for reviewers 20% 29% 28% 12% n=656 30% 72% 25% 16% 11% n=778 35% 32% 28% 66% 38% 34% Global 45% 51% 34% Acknowledgement for the reviewer (e. g. reviewer name(s) published on the article) All researchers who publish an article undertake a review within a year Time taken for peer review on a research article is 25% less than it is today (submission to final decision) Software is used to triage manuscripts prior to formal peer review Eastern Europe 60% 41% 36% 48% APAC 68% 62% Formal training of reviewers to create a quality benchmark Accreditation given to reviewers (CME/CPD points) Western Europe 41% 25% 33% 29% 36% 38% 37% 34% 28% 25% 26% 21% 25% 19% 21% 22% 20% 17% n=926 n=303 n=3133 42

HOW TO MAINTAIN PEER REVIEW SYSTEM: 41% of researchers from the Middle East want financial rewards. Over half of researchers from Africa want formal training/quality benchmark and/or accreditation (e. g. CPD points) Which of the below do you consider most important for maintaining a healthy system of peer review? Latin America Clear guidance given to reviewers on criteria for reviewing 69% Employer recognition of time spent reviewing Reviewer(s) reports made available alongside research article (reviewer can choose to remain anonymous) 43% 38% Greater diversity within the peer review community 47% Middle East 59% Africa Global 61% 66% 41% 30% 29% 38% 31% 20% 45% 38% Formal training of reviewers to create a quality benchmark 43% 37% 53% 37% Accreditation given to reviewers (CME/CPD points) 45% 37% 53% 34% Acknowledgement for the reviewer (e. g. reviewer name(s) published on the article) 27% 35% Financial reward for reviewers All researchers who publish an article undertake a review within a year Time taken for peer review on a research article is 25% less than it is today (submission to final decision) Software is used to triage manuscripts prior to formal peer review 27% 28% 20% 14% 41% 24% 14% n=236 34% 28% 32% 28% 18% 22% 18% n=117 n=97 25% 21% 17% n=3133 43

HOW TO MAINTAIN PEER REVIEW SYSTEM: Early career researchers are slightly more likely to want financial rewards for reviewers (32% vs. 28% overall). 43% of females want greater diversity within peer review community (36% of males want greater diversity) 44 Which of the below do you consider most important for maintaining a healthy system of peer review? Under 36 Clear guidance given to reviewers on criteria for reviewing Employer recognition of time spent reviewing Reviewer(s) reports made available alongside research article (reviewer can choose to remain anonymous) Greater diversity within the peer review community 45% 43% 35% 45% Accreditation given to reviewers (CME/CPD points) 36% Acknowledgement for the reviewer (e. g. reviewer name(s) published on the article) 33% 32% All researchers who publish an article undertake a review within a year 25% Time taken for peer review on a research article is 25% less than it is today (submission to final decision) 26% Software is used to triage manuscripts prior to formal peer review 16% n=492 62% 76% 38% 40% 36% Global 66% 45% 49% 40% 35% Female 44% 37% 36% Male 72% 47% 37% 41% 56+ 64% 62% Formal training of reviewers to create a quality benchmark Financial reward for reviewers 36 -55 38% 31% 36% 43% 38% 34% 44% 37% 34% 26% 30% 31% 22% 28% 27% 29% 25% 28% 25% 26% 24% 21% 19% 22% 36% 34% 32% 25% 19% 21% 17% 18% 15% 17% 16% n=1581 n=1001 n=2107 n=932 n=3133

Back to contents ADDITIONAL INFORMATION FOR ASSESSING RESEARCH OUTPUTS Results by geographic region, country, broad subject area, age group and gender 45

ASSESSING RESEARCH OUTPUTS: 90% of chemists and life scientists want to see a reproducibility indicator 46 When a reader is interacting with research outputs (e. g. research articles, preprints, data), what additional information is helpful for assessing those research outputs? % helpful Chemistry Citations Indicator to show whether someone else tried to reproduce the research and their success Post publication commentary (from research community) Clear signifier to show content has been peer reviewed Computer Science 92% 85% 90% 86% Downloads 70% Indicator to show number of times the research was included in systematic reviews 70% News articles in popular press 52% Number of collaborators on research 38% Number of retweets and/or mentions on blogs 36% n=146 92% 88% 90% 82% 68% 62% 52% 44% 31% n=165 n=257 76% 77% 72% 70% 69% 72% 70% 68% 66% 57% 42% 79% 83% 77% 75% 65% 41% 92% 76% 58% 61% Life Sciences 81% 82% Global Engineering 84% 76% Peer reviewers' rating(s) (e. g. score out of ten) 89% 81% 83% Earth & Env. Science 57% 61% 52% 56% 50% 38% 42% 40% 36% 31% n=443 n=602 n=3133 50%

ASSESSING RESEARCH OUTPUTS: 57% Materials Scientists think news articles in popular press would be helpful 47 When a reader is interacting with research outputs (e. g. research articles, preprints, data), what additional information is helpful for assessing those research outputs? % helpful Materials Science Citations Indicator to show whether someone else tried to reproduce the research and their success Post publication commentary (from research community) Clear signifier to show content has been peer reviewed 92% 89% 69% 84% Indicator to show number of times the research was included in systematic reviews 69% 67% Peer reviewers' rating(s) (e. g. score out of ten) 57% News articles in popular press 57% Number of collaborators on research 48% 27% n=129 83% 77% 66% 48% 37% n=116 85% 74% 83% n=449 Global 88% 77% 82% 80% 79% 77% 74% 65% 66% 69% 72% 65% 68% 50% 30% 86% 68% 48% 58% 46% Soc. Sci + Arts Hum + Economics 84% 83% 79% 76% Physics & Astronomy 89% 83% 75% Downloads Number of retweets and/or mentions on blogs Medicine and Allied Health Maths 44% 37% 26% n=220 49% 56% 46% 50% 37% 25% n=587 42% 31% n=3133

ASSESSING RESEARCH OUTPUTS: German researchers more likely to want signifier of reproducibility; researchers in USA more likely to want a clear signifier of content 48 When a reader is interacting with research outputs (e. g. research articles, preprints, data), what additional information is helpful for assessing those research outputs? % helpful China Citations Indicator to show whether someone else tried to reproduce the research and their success Post publication commentary (from research community) Clear signifier to show content has been peer reviewed USA Japan 88% 93% 79% 83% 86% 81% Downloads Indicator to show number of times the research was included in systematic reviews 63% 79% Peer reviewers' rating(s) (e. g. score out of ten) 75% News articles in popular press 75% Number of collaborators on research 59% Number of retweets and/or mentions on blogs 56% n=363 42% 29% 15% n=558 52% 36% 18% n=135 63% 50% 48% 31% 53% 62% 59% 78% 64% 73% 82% 79% 54% 61% 57% 84% 89% 74% 67% 88% 82% 80% 72% Global Germany 87% 84% 81% 85% Russia 42% 77% 69% 68% 56% 39% 28% 50% 38% 28% 42% 19% n=137 19% n=83 31% n=3133

ASSESSING RESEARCH OUTPUTS: Researchers in India more likely to find all indicators helpful; researchers in the UK least likely to find indicators useful (except for post-publication commentary and content signifier) 49 When a reader is interacting with research outputs (e. g. research articles, preprints, data), what additional information is helpful for assessing those research outputs? % helpful Republic of Korea Citations Indicator to show whether someone else tried to reproduce the research and their success Post publication commentary (from research community) Clear signifier to show content has been peer reviewed France 73% 86% 73% 76% 84% 76% 81% 85% 58% Peer reviewers' rating(s) (e. g. score out of ten) 58% 54% Number of collaborators on research 39% Number of retweets and/or mentions on blogs 41% n=67 48% 41% 30% 27% 21% n=153 India Canada Global 83% 88% 89% 86% 82% 93% 86% 79% 96% 87% 91% 77% 63% 82% 59% 69% 57% 90% 58% 68% 77% 60% 70% Indicator to show number of times the research was included in systematic reviews 76% 80% 66% Downloads News articles in popular press UK 73% 40% 77% 33% 17% n=76 74% 50% n=156 43% 33% 27% 16% n=98 56% 50% 42% 31% n=3133

ASSESSING RESEARCH OUTPUTS: North American researchers are more likely to find signifiers to show content useful, while APAC researchers are more likely to find downloads, peer review ratings, number of collaborators and retweets/blog mentions useful When a reader is interacting with research outputs (e. g. research articles, preprints, data), what additional information is helpful for assessing those research outputs? % helpful North America 87% Citations Indicator to show whether someone else tried to reproduce the research and their success Post publication commentary (from research community) Clear signifier to show content has been peer reviewed 79% 87% News articles in popular press 31% Number of collaborators on research 28% 15% n=656 76% 62% 66% 35% 66% 31% n=778 66% 73% 46% 21% 61% 79% 52% 44% n=926 Global 88% 82% 79% 76% 62% 42% 83% 79% 60% 62% 88% 81% 78% 57% Peer reviewers' rating(s) (e. g. score out of ten) Eastern Europe 90% 84% 80% Indicator to show number of times the research was included in systematic reviews APAC 83% 81% Downloads Number of retweets and/or mentions on blogs Western Europe 52% 77% 69% 68% 56% 41% 50% 41% 42% 22% n=303 31% n=3133 50

ASSESSING RESEARCH OUTPUTS: Researchers in Africa are more likely to want a signifier of content and to see the number of collaborators than average When a reader is interacting with research outputs (e. g. research articles, preprints, data), what additional information is helpful for assessing those research outputs? % helpful Latin America Citations Indicator to show whether someone else tried to reproduce the research and their success Post publication commentary (from research community) Clear signifier to show content has been peer reviewed 91% 78% 79% Indicator to show number of times the research was included in systematic reviews Number of collaborators on research Number of retweets and/or mentions on blogs 93% 86% n=236 79% 86% 89% 77% 76% 75% 78% 69% 75% 76% 68% 50% 31% 82% 74% 56% 43% 88% 90% 78% 69% Peer reviewers' rating(s) (e. g. score out of ten) Global Africa 74% Downloads News articles in popular press Middle East 70% 64% 53% 41% n=117 38% n=97 56% 61% 50% 63% 42% 31% n=3133 51

ASSESSING RESEARCH OUTPUTS: Early career researchers are slightly more interested in seeing a signifier of reproducibility, news articles in popular press, number of collaborators and number of retweets/blog mentions than average When a reader is interacting with research outputs (e. g. research articles, preprints, data), what additional information is helpful for assessing those research outputs? % helpful Under 36 Citations Indicator to show whether someone else tried to reproduce the research and their success Post publication commentary (from research community) Clear signifier to show content has been peer reviewed 36 -55 66% 71% News articles in popular press 56% 48% 36% n=492 57% 53% 44% 33% n=1581 69% 70% 69% 41% 37% 24% n=1001 78% 82% 79% 80% 68% 65% 57% 53% 51% 43% 88% 76% 80% 71% 67% 57% 80% 75% 77% Peer reviewers' rating(s) (e. g. score out of ten) 83% 82% Global Female 88% 80% 77% 83% Male 87% 81% 87% Indicator to show number of times the research was included in systematic reviews Number of retweets and/or mentions on blogs 89% 91% Downloads Number of collaborators on research 56+ 52% 48% 42% 30% n=2107 n=932 77% 69% 68% 56% 50% 42% 31% n=3133 52

Back to contents TRUSTWORTHINESS OF RESEARCH OUTPUTS Results by geographic region, country, broad subject area, age group and gender 53

TRUSTWORTHINESS OF RESEARCH OUTPUTS: Computer scientists have least confidence in the research outputs they interact with. Over half of mathematicians that distrust at least some content only read research from researchers they know Thinking about the various research outputs that you interacted with (or encountered) last week (including. data, code, graphics, video, theses, preprints and articles), what proportion of the outputs would you consider trustworthy (i. e. outputs could be relied upon)? All of them Majority of them Total n=3133 14% Chemistry 18% n=146 Computer Science 11% n=165 Earth & Env. Science 12% n=257 Engineering 19% n=443 Life Sciences 15% n=602 Materials Science 19% n=129 Maths 20% n=116 Medicine and Allied Health 11% n=449 Physics & Astronomy n=220 Soc. Sci + Arts Hum + Economicsn=587 14% 10% About half of them Some of them 15% 41% 21% 40% 25% 50% 43% 39% 48% 53% 54% 21% 19% n=2715 n=121 23% n=147 13% 24% n=224 14% 24% n=374 53% 41% IF NOT ALL TRUSTWORTHY: Which of the following mechanisms do you employ to compensate for any lack of confidence you have in the content you are considering reading/accessing? None of them 48% 14% 16% 11% 18% 21% 29% 22% 8% 14% 18% 24% 22% 54 n=519 n=109 n=92 n=399 n=191 n=522

TRUSTWORTHINESS OF RESEARCH OUTPUTS: Researchers from China, Japan and India are most polarised on whether they trust research outputs. Russian researchers are more likely to only read research from researchers they know Thinking about the various research outputs that you interacted with (or encountered) last week (including. data, code, graphics, video, theses, preprints and articles), what proportion of the outputs would you consider trustworthy (i. e. outputs could be relied upon)? All of them Majority of them Total 14% About half of them Some of them 48% IF NOT ALL TRUSTWORTHY: Which of the following mechanisms do you employ to compensate for any lack of confidence you have in the content you are considering reading/accessing? None of them 15% 21% n=2715 n=3133 China n=363 USA n=558 25% 7% 34% 14% 39% Russia 10% 49% Germany 14% Republic of Korea 12% UK 10% n=137 n=83 n=67 n=153 France 8% India 12% Canada 11% n=76 n=156 n=98 27% 62% Japan n=135 11% 17% 13% 19% 17% 57% 26% 19% 14% 62% 14% 19% n=123 n=71 n=59 14% n=138 17% n=70 32% 17% n=518 n=116 21% 19% 40% 14% 33% 54% 42% n=271 n=137 11% n=88 55

TRUSTWORTHINESS OF RESEARCH OUTPUTS: Latin American researchers most likely to trust research outputs, while APAC most polarised IF NOT ALL TRUSTWORTHY: Which of the following mechanisms do you employ to compensate for any lack of confidence you have in the content you are considering reading/accessing? Thinking about the various research outputs that you interacted with (or encountered) last week (including. data, code, graphics, video, theses, preprints and articles), what proportion of the outputs would you consider trustworthy (i. e. outputs could be relied upon)? All of them Majority of them About half of them Total n=3133 14% 48% Africa n=97 13% APAC n=926 Eastern Europe n=303 Latin America n=236 Middle East n=117 North America n=656 Western Europe n=778 19% 11% 38% 12% 62% 11% 55% n=2715 21% 1% 23% 14% 11% n=270 10% 18% n=191 3% n=100 27% 17% n=84 1% n=759 28% 18% 56% 7% 21% 14% 47% 45% None of them 15% 52% 19% 14% Some of them 14% 16% n=606 n=688 56

TRUSTWORTHINESS OF RESEARCH OUTPUTS: Little difference by age and gender. Early career researchers slightly more likely to read/access research from specific institutes IF NOT ALL TRUSTWORTHY: Which of the following mechanisms do you employ to compensate for any lack of confidence you have in the content you are considering reading/accessing? Thinking about the various research outputs that you interacted with (or encountered) last week (including. data, code, graphics, video, theses, preprints and articles), what proportion of the outputs would you consider trustworthy (i. e. outputs could be relied upon)? All of them Majority of them About half of them Some of them None of them 14% 48% 15% 21% Under 36 n=492 16% 46% 16% 20% 36 -55 n=1581 15% 47% 16% 22% 1% n=1355 52% 14% 22% n=888 48% 15% 22% 1% n=1825 19% n=802 Total n=2715 n=3133 56+ n=1001 11% Male n=2107 14% Female n=932 15% 50% 15% 2% n=421 57

Back to contents TRUSTWORTHINESS OF RESEARCH PROFILES Results by geographic region, country, broad subject area, age group and gender 58

TRUSTWORTHINESS OF RESEARCH PROFILES: XX Engineering and Materials Science researchers most likely to trust all profiles, whilst Mathematicians most likely to think only some/no profiles are trustworthy Now thinking about the various research-related profiles that you interacted with (or encountered) last week (including researcher and institutional profiles), what proportion of the profiles would you consider trustworthy (i. e. confident that the content of the profiles could be relied upon)? All of them Majority of them Total n=2588 Chemistry n=123 Computer Science n=145 Earth & Env. Science n=206 Engineering n=370 Life Sciences n=494 Materials Science n=110 Maths n=95 Medicine and Allied Health n=390 Physics & Astronomy n=165 Soc. Sci + Arts Hum + Economics n=477 About half of them 30% Some of them None of them 45% 34% 10% 14% 45% 10% 9% IF NOT ALL TRUSTWORTHY: Which of the following mechanisms do you employ to compensate for any lack of confidence you have in the research-related profiles you are viewing? n=1825 n=82 29% 45% 13% 12% n=108 30% 44% 13% 12% n=144 35% 30% 36% 30% 27% 32% 26% 41% 9% 49% 35% 38% 46% 42% 51% 15% n=256 8% 13% n=350 16% n=73 9% 8% 22% 9% 11% 9% 17% 14% 13% n=67 n=283 n=110 n=345 59

TRUSTWORTHINESS OF RESEARCH PROFILES: Chinese researchers most likely to trust all profiles (41%), while less than a quarter of researchers in USA, Korea, UK, France and India trust all profiles Now thinking about the various research-related profiles that you interacted with (or encountered) last week (including researcher and institutional profiles), what proportion of the profiles would you consider trustworthy (i. e. confident that the content of the profiles could be relied upon)? All of them Majority of them Total About half of them 14% Some of them 48% None of them 15% 21% IF NOT ALL TRUSTWORTHY: Which of the following mechanisms do you employ to compensate for any lack of confidence you have in the research-related profiles you are viewing? n=1825 n=2588 China n=333 USA n=412 41% 33% 27% 42% Russia 27% 45% n=110 Germany n=69 16% 56% Japan n=113 8% 32% 10% 10% n=82 14% 9% n=80 7% 24% UK 23% 55% 12% 8% n=87 France 22% 58% 11% 6% 3% n=50 India 23% 24% 3% n=102 Canada 25% n=113 n=64 n=133 n=79 41% 9% 54% 23% n=47 Republic of Korea n=58 14% n=315 21% 12% 50% 36% n=197 3% 10% 12% n=44 n=59 60

TRUSTWORTHINESS OF RESEARCH PROFILES: 46% of Latin America researchers trust all research profiles they encounter compared to 24% of North Americas IF NOT ALL TRUSTWORTHY: Which of the following mechanisms do you employ to compensate for any lack of confidence you have in the research-related profiles you are viewing? Now thinking about the various research-related profiles that you interacted with (or encountered) last week (including researcher and institutional profiles), what proportion of the profiles would you consider trustworthy (i. e. confident that the content of the profiles could be relied upon)? All of them Majority of them Total About half of them 14% Some of them 48% 15% None of them n=1825 21% n=2588 Africa n=83 APAC n=811 Eastern Europe n=257 Latin America n=199 Middle East n=100 North America n=491 Western Europe n=631 26% 44% 33% 28% 14% 37% 45% 46% 35% 24% 27% 9% 11% 44% 45% 56% 54% 18% 2% n=61 2% n=557 15% n=185 6% 4% 7% 12% 1% n=110 n=68 10% n=374 10% 9% n=458 61

TRUSTWORTHINESS OF RESEARCH PROFILES: More than a third (35%) of early career researchers more likely to trust all research profiles compared with 25% of those aged 56+ IF NOT ALL TRUSTWORTHY: Which of the following mechanisms do you employ to compensate for any lack of confidence you have in the research-related profiles you are viewing? Now thinking about the various research-related profiles that you interacted with (or encountered) last week (including researcher and institutional profiles), what proportion of the profiles would you consider trustworthy (i. e. confident that the content of the profiles could be relied upon)? All of them Majority of them Total n=2581 Under 36 n=403 36 -55 n=1344 56+ n=794 14% About half of them Some of them 48% 35% 32% 25% Male n=1744 30% Female n=763 31% None of them 15% 43% 49% 44% 47% n=1825 21% 9% 12% 1% n=271 10% 13% 1% n=922 9% 10% 14% 9% n=595 16% 12% 1% n=1231 n=527 62

Back to contents SOURCING TRUSTED CONTENT Results by geographic region, country, broad subject area, age group and gender 63

SOURCING TRUSTED CONTENT: Being able to adjust/choose search algorithm/parameters is considered most effective way to be directed to appropriate trusted content. Being sourceneutral is considered less effective in SSE/AH 64 Please indicate which of the below solutions you believe would be effective in ensuring you are directed to appropriate trusted content? % effective Chemistry You are able to adjust/choose search algorithms and parameters Computer Science 82% When platforms serve you an article to read, they explain why they have selected it 73% The content is inter-operable across databases, i. e. move seamlessly between outputs 80% The information is source neutral (it is not from specific journals or publishers) 71% Earth & Env. Science Engineering 78% 80% 82% 77% 73% 71% 77% 78% Life Sciences 77% 69% 72% 74% Global 78% 71% 56% 58% n=146 n=165 n=257 n=443 n=602 n=3133 Materials Science Maths Medicine and Allied Health Physics & Astronomy Soc. Sci + Arts Hum + Economics Global You are able to adjust/choose search algorithms and parameters When platforms serve you an article to read, they explain why they have selected it The content is inter-operable across databases, i. e. move seamlessly between outputs The information is source neutral (it is not from specific journals or publishers) 70% 68% 70% 63% n=129 18. 09. 2020 74% 69% 81% 65% 77% 75% 62% 78% 66% 72% 67% 70% 71% 61% 73% 65% 66% 71% 58% n=116 66% n=449 57% n=220 50% n=587 62% n=3133

SOURCING TRUSTED CONTENT: Chinese researchers rate all solutions as more effective than average. Researchers from USA and Western European countries think being able to adjust/choose search as effective but rate the other three solutions less effective than average. 65 Please indicate which of the below solutions you believe would be effective in ensuring you are directed to appropriate trusted content? % effective China You are able to adjust/choose search algorithms and parameters USA 84% When platforms serve you an article to read, they explain why they have selected it 81% The content is inter-operable across databases, i. e. move seamlessly between outputs 79% The information is source neutral (it is not from specific journals or publishers) 79% Japan 79% 65% 67% 51% Russia 67% 73% 84% 65% 66% 61% Germany 76% 47% 80% Global 78% 63% 71% 57% 71% 59% 62% n=363 n=558 n=135 n=137 n=83 n=3133 Republic of Korea UK France India Canada Global You are able to adjust/choose search algorithms and parameters 67% When platforms serve you an article to read, they explain why they have selected it 73% 63% 86% The content is inter-operable across databases, i. e. move seamlessly between outputs 70% 58% 86% The information is source neutral (it is not from specific journals or publishers) 65% n=67 18. 09. 2020 75% 53% n=153 75% 53% n=76 86% 70% n=156 78% 52% 68% 63% n=98 78% 71% 62% n=3133

66 SOURCING TRUSTED CONTENT: Researchers in North America and Eastern Europe less likely to consider source neutrality as effective Please indicate which of the below solutions you believe would be effective in ensuring you are directed to appropriate trusted content? % effective North America You are able to adjust/choose search algorithms and parameters When platforms serve you an article to read, they explain why they have selected it The content is inter-operable across databases, i. e. move seamlessly between outputs The information is source neutral (it is not from specific journals or publishers) 79% 63% 67% 53% APAC 75% 68% 79% 63% 76% 56% 71% n=778 n=926 Latin America Middle East Africa 74% 69% 72% 60% Eastern Europe 79% n=656 n=236 18. 09. 2020 Western Europe 75% 71% 75% 66% n=117 74% 72% 74% 62% n=97 80% 65% 73% 47% n=303 Global 78% 71% 71% 62% n=3133

SOURCING TRUSTED CONTENT: Source neutrality is less important among female researchers and those aged 56+ 67 Please indicate which of the below solutions you believe would be effective in ensuring you are directed to appropriate trusted content? % effective Under 36 You are able to adjust/choose search algorithms and parameters When platforms serve you an article to read, they explain why they have selected it The content is inter-operable across databases, i. e. move seamlessly between outputs The information is source neutral (it is not from specific journals or publishers) 18. 09. 2020 84% 77% 36 -55 78% 72% 73% 65% 64% n=492 n=1581 56+ 74% Male 77% 70% 73% 66% 72% 56% n=1001 Female 64% n=2107 79% 68% 72% Global 78% 71% 56% 62% n=932 n=3133

Back to contents WHAT WILL HELP MOST TO IMPROVE COMMUNICATING TO THE PUBLIC Results by geographic region, country, broad subject area, age group and gender 68

COMMUNICATING TO THE PUBLIC: Explanations in lay terms considered most helpful way to communicate research with the public 69 To help people outside the research community judge the quality of research articles they view, which of the following would be most helpful? Chemistry Explain research context, findings and implications in lay terms 60% Computer Science 63% Enable them to ask questions of the authors (e. g. discussion to be posted alongside the article) 43% 39% Clear explanation of peer review and make it easily accessible from content (e. g. how many people involved, their role) 39% Peer reviewed content is clearly marked as such 36% Enable updates to the article after publication (e. g. as more work is done on the topic by author) Provide guidance on statistics (e. g. probability) 40% 28% n=146 Earth & Env. Science 41% 30% n=165 71% Engineering 60% 44% 40% 37% 43% Life Sciences 73% Global 70% 38% 40% 38% 33% 35% 30% 35% 33% 40% 33% 35% 30% n=257 24% n=443 27% n=602 30% n=3133

COMMUNICATING TO THE PUBLIC: Explanations in lay terms considered most helpful way to communicate research with the public 70 To help people outside the research community judge the quality of research articles they view, which of the following would be most helpful? Materials Science Explain research context, findings and implications in lay terms Enable them to ask questions of the authors (e. g. discussion to be posted alongside the article) 58% 36% Peer reviewed content is clearly marked as such 33% Provide guidance on statistics (e. g. probability) 59% 49% Clear explanation of peer review and make it easily accessible from content (e. g. how many people involved, their role) Enable updates to the article after publication (e. g. as more work is done on the topic by author) Maths 40% 23% n=129 Medicine and Allied Health Physics & Astronomy 76% 73% Soc. Sci + Arts Hum + Economics 81% Global 70% 34% 35% 39% 38% 40% 38% 39% 37% 35% 38% 35% 37% 35% 36% 30% n=587 n=3133 41% 28% 27% n=116 37% 33% 40% n=449 42% 29% 22% n=220

COMMUNICATING TO THE PUBLIC: Explanations in lay terms considered most helpful way to communicate research with the public. Researchers from USA and Germany more likely to find flagging up peer review content helpful. 71 To help people outside the research community judge the quality of research articles they view, which of the following would be most helpful? China Explain research context, findings and implications in lay terms Enable them to ask questions of the authors (e. g. discussion to be posted alongside the article) Clear explanation of peer review and make it easily accessible from content (e. g. how many people involved, their role) Peer reviewed content is clearly marked as such Enable updates to the article after publication (e. g. as more work is done on the topic by author) Provide guidance on statistics (e. g. probability) 60% USA 83% 32% 46% 39% 42% 33% 41% 45% 33% 29% 35% n=363 n=558 Japan Russia 61% 55% 36% 33% 35% 29% 25% n=135 Germany 76% 49% 42% 40% 29% 31% 33% 21% n=137 49% 30% n=83 Global 70% 40% 38% 35% 30% n=3133

COMMUNICATING TO THE PUBLIC: Explanations in lay terms considered most helpful way to communicate research with the public. Researchers from UK and Canada more likely to find flagging up peer review content and guidance on statistics helpful. 72 To help people outside the research community judge the quality of research articles they view, which of the following would be most helpful? Republic of Korea Explain research context, findings and implications in lay terms Peer reviewed content is clearly marked as such Enable updates to the article after publication (e. g. as more work is done on the topic by author) Provide guidance on statistics (e. g. probability) 36% 25% 39% 45% 21% 28% France 87% 57% Enable them to ask questions of the authors (e. g. discussion to be posted alongside the article) Clear explanation of peer review and make it easily accessible from content (e. g. how many people involved, their role) UK 41% 50% 41% 43% 46% 34% 40% n=67 n=153 24% n=76 83% 35% 46% 24% n=156 Global 70% 40% 32% 21% 33% Canada 64% 79% 42% 33% India 38% 44% 35% 43% n=98 35% 30% n=3133

COMMUNICATING TO THE PUBLIC: Explanations in lay terms considered most helpful way to communicate research with the public. Researchers from North America and Western Europe more likely to find flagging up peer review content helpful. To help people outside the research community judge the quality of research articles they view, which of the following would be most helpful? North America Explain research context, findings and implications in lay terms Enable them to ask questions of the authors (e. g. discussion to be posted alongside the article) Clear explanation of peer review and make it easily accessible from content (e. g. how many people involved, their role) Peer reviewed content is clearly marked as such Western Europe 83% 32% 80% 39% 38% 33% 31% Provide guidance on statistics (e. g. probability) 36% 32% n=656 n=778 37% 29% n=926 Global 70% 40% 38% 37% 31% Enable updates to the article after publication (e. g. as more work is done on the topic by author) 64% 40% 40% Eastern Europe 60% 42% 36% 45% APAC 31% 35% 21% n=303 35% 30% n=3133 73

COMMUNICATING TO THE PUBLIC: Explanations in lay terms considered most helpful way to communicate research with the public. Latin America researchers less likely to think guidance on statistics and flagging peer reviewed content would be helpful To help people outside the research community judge the quality of research articles they view, which of the following would be most helpful? Latin America Explain research context, findings and implications in lay terms Enable them to ask questions of the authors (e. g. discussion to be posted alongside the article) Clear explanation of peer review and make it easily accessible from content (e. g. how many people involved, their role) Peer reviewed content is clearly marked as such Enable updates to the article after publication (e. g. as more work is done on the topic by author) Provide guidance on statistics (e. g. probability) Middle East 72% 38% 40% 23% n=236 69% 70% 45% 38% 27% Global 64% 47% 44% Africa 40% 38% 24% 36% 28% n=117 38% 30% 43% 26% n=97 35% 30% n=3133 74

COMMUNICATING TO THE PUBLIC: Explanations in lay terms considered most helpful way to communicate research with the public. Interest in a marker for content that is peer reviewed increases with age. Those under 36 more likely than average to think a clear explanation of peer review would be helpful 75 To help people outside the research community judge the quality of research articles they view, which of the following would be most helpful? Under 36 Explain research context, findings and implications in lay terms 43% Clear explanation of peer review and make it easily accessible from content (e. g. how many people involved, their role) 43% 41% Enable updates to the article after publication (e. g. as more work is done on the topic by author) 39% Provide guidance on statistics (e. g. probability) 36% n=492 70% 35% 28% n=1581 Female 69% 72% 38% 33% Male 40% 36% 37% 28% 56+ 71% 64% Enable them to ask questions of the authors (e. g. discussion to be posted alongside the article) Peer reviewed content is clearly marked as such 36 -55 42% 34% 30% n=1001 Global 70% 39% 40% 38% 36% 33% 35% 35% 33% 30% n=932 n=3133 28% n=2107

Back to contents PUBLIC CONFIDENCE IN RESEARCH EVIDENCE Results by geographic region, country, broad subject area, age group and gender 76

PUBLIC CONFIDENCE IN RESEARCH EVIDENCE: Life Scientists and Earth/Environmental scientists slightly more likely to consider misinterpretation of research in media/policy/public discussion a problem Thinking about public confidence in research evidence, how much, if at all, do you believe any of the following are a problem? Chemistry Computer Science Earth & Env. Science Misinterpretation of research outcomes in media, policy or public discussion 84% 78% 86% Increased low quality research available (i. e. research meets minimum technical standard only) 83% 78% Difficulty in distinguishing high quality research Deliberate misrepresentation by media 70% Too many non-peer reviewed research publications Deliberate misrepresentation by researchers/their institutions 66% Volume of information Misinterpretation of research outcomes by researchers/their institutions Contradictory research findings published (e. g. research from different groups have opposing conclusions) 59% 62% 58% 67% 65% 52% 76% 75% 72% 76% 74% 71% 65% 59% 44% 36% Large number of competing platforms providing access to the same research outputs 47% 35% 39% 40% n=165 n=257 n=443 n=146 70% 57% Increased alternative sources (e. g. different versions of articles: preprint, accepted, published) 43% 78% 72% 64% 53% 76% 69% 65% 48% Global 83% 87% 76% 69% 66% Life Sciences 74% 75% 74% 58% Engineering 74% 77% 79% % large/medium problem 81% 72% 77 70% 62% 64% 60% 63% 48% 63% 52% 31% 39% 30% 35% n=602 n=3133

PUBLIC CONFIDENCE IN RESEARCH EVIDENCE: SSE/AH and medical/health researchers more likely to consider volume of research a problem 86% 68% 86% 80% 87% 74% 80% 68% 78% 83% Thinking about public confidence in research evidence, how much, if at all, do you believe any of the following are a problem? 68% 73% 66% 71% 65% 56% 63% 60% 62% 55% 59% 43% 38% 40% 64% 65% 61% 63% 71% 60% 65% 44% 75% 54% 70% 55% 79% 73% 77% 65% % large/medium problem 77% 79% 67% 53% 81% 66% 78 49% 32% 29% 31% 26%

PUBLIC CONFIDENCE IN RESEARCH EVIDENCE: Nearly all researchers in Germany and USA consider misinterpretation of research outcomes in media/policy/public discussion a problem. They are also more likely to think volume of information is a problem. In Japan researchers consider misrepresentation or misinterpretation by institutions/researchers a problem Thinking about public confidence in research evidence, how much, if at all, do you believe any of the following are a problem? China Misinterpretation of research outcomes in media, policy or public discussion 77% Increased low quality research available (i. e. research meets minimum technical standard only) 74% Difficulty in distinguishing high quality research USA Japan 90% 81% Deliberate misrepresentation by media 69% 70% Too many non-peer reviewed research publications 70% 69% Deliberate misrepresentation by researchers/their institutions 70% Volume of information 53% Misinterpretation of research outcomes by researchers/their institutions Contradictory research findings published (e. g. research from different groups have opposing conclusions) 59% Increased alternative sources (e. g. different versions of articles: preprint, accepted, published) 47% Large number of competing platforms providing access to the same research outputs 46% n=363 50% 27% 22% n=558 n=135 80% 75% 83% 74% 62% 68% 81% 59% 53% 41% 44% 72% 56% 76% 44% 78% 65% 43% 53% 84% 67% 34% 26% n=137 83% 74% 67% 73% 75% 67% 95% 62% 45% Global Germany 69% 66% 83% 67% % large/medium problem Russia 80% 79 60% 49% 31% 38% n=83 70% 64% 63% 52% 39% 35% n=3133

PUBLIC CONFIDENCE IN RESEARCH EVIDENCE: Misinterpretation of research outcomes in media/policy/ public discussion and volume of information are more likely to be considered a problem in UK, France and Canada. Researchers in India are most concerned about there being too many non-peer reviewed research publications Thinking about public confidence in research evidence, how much, if at all, do you believe any of the following are a problem? % large/medium problem Republic of Korea UK France India Canada Misinterpretation of research outcomes in media, policy or public discussion Increased low quality research available (i. e. research meets minimum technical standard only) Difficulty in distinguishing high quality research Deliberate misrepresentation by media 80% Volume of information 77% 78% 66% 47% Increased alternative sources (e. g. different versions of articles: preprint, accepted, published) 48% Large number of competing platforms providing access to the same research outputs 46% n=67 62% 50% 31% 27% n=153 75% 43% 64% 45% 31% 60% 32% 57% n=156 74% 72% 70% 60% 71% 49% 71% 78% 63% 77% n=76 88% 54% 70% 30% 22% n=98 83% 78% 84% 57% Global 86% 69% 74% 61% 62% 76% 72% 68% Contradictory research findings published (e. g. research from different groups have opposing conclusions) 85% 84% 92% 74% 82% 85% 83% Misinterpretation of research outcomes by researchers/their institutions 89% 80% 74% Too many non-peer reviewed research publications Deliberate misrepresentation by researchers/their institutions 95% 76% 80 64% 63% 52% 39% 35% n=3133

PUBLIC CONFIDENCE IN RESEARCH EVIDENCE: Researchers in North America and Western Europe more likely to consider misinterpretation of research outcomes in media/policy/public discussion and volume of information a problem. Thinking about public confidence in research evidence, how much, if at all, do you believe any of the following are a problem? North America Misinterpretation of research outcomes in media, policy or public discussion Western Europe 91% 81% 83% Difficulty in distinguishing high quality research 83% 70% Too many non-peer reviewed research publications 70% Deliberate misrepresentation by researchers/their institutions Volume of information Contradictory research findings published (e. g. research from different groups have opposing conclusions) Increased alternative sources (e. g. different versions of articles: preprint, accepted, published) Large number of competing platforms providing access to the same research outputs 50% 27% 22% n=656 78% 72% 75% 73% 69% 74% 71% 70% 65% 54% 57% 69% 48% 53% 32% 47% 31% 46% n=778 76% 71% 74% 54% 83% 68% 63% n=926 Global 74% 75% 68% 75% Misinterpretation of research outcomes by researchers/their institutions Eastern Europe 80% 46% % large/medium problem 78% Increased low quality research available (i. e. research meets minimum technical standard only) Deliberate misrepresentation by media APAC 81 61% 63% 63% 56% 35% 29% n=303 64% 52% 39% 35% n=3133

PUBLIC CONFIDENCE IN RESEARCH EVIDENCE: African researchers more concerned about the number of non-peer reviewed research publications, but less concerned about the volume of information Thinking about public confidence in research evidence, how much, if at all, do you believe any of the following are a problem? Latin America Misinterpretation of research outcomes in media, policy or public discussion 79% Increased low quality research available (i. e. research meets minimum technical standard only) 75% Difficulty in distinguishing high quality research 77% Middle East Too many non-peer reviewed research publications 69% 78% Deliberate misrepresentation by researchers/their institutions 68% Increased alternative sources (e. g. different versions of articles: preprint, accepted, published) 53% Large number of competing platforms providing access to the same research outputs 53% n=236 68% 88% 61% 52% 74% 53% 28% 75% 69% 68% 39% 81% 76% 75% Contradictory research findings published (e. g. research from different groups have opposing conclusions) 83% 70% Misinterpretation of research outcomes by researchers/their institutions Global Africa 76% 79% Deliberate misrepresentation by media Volume of information 65% 52% 40% n=117 % large/medium problem 78% 75% 74% 70% 64% 63% 69% 63% 68% 52% 41% 39% 41% 35% n=97 n=3133 82

PUBLIC CONFIDENCE IN RESEARCH EVIDENCE: Concerns about volume of information and non-peer reviewed research publications increases with age Thinking about public confidence in research evidence, how much, if at all, do you believe any of the following are a problem? Under 36 Misinterpretation of research outcomes in media, policy or public discussion 74% Difficulty in distinguishing high quality research 76% 75% Deliberate misrepresentation by researchers/their institutions 67% 64% Volume of information Misinterpretation of research outcomes by researchers/their institutions Contradictory research findings published (e. g. research from different groups have opposing conclusions) 53% 36% 30% n=492 53% 39% 36% n=1581 76% 82% 78% 76% 74% 80% 75% 74% 70% 64% 57% 50% 40% 37% n=1001 37% n=2107 72% 70% 61% 52% 40% 74% 69% 64% 59% 73% 62% 60% 67% 64% 65% 83% 79% 74% 63% 56% 86% 70% 67% Too many non-peer reviewed research publications Global Female 81% 84% 75% % large/medium problem Male 78% Deliberate misrepresentation by media Large number of competing platforms providing access to the same research outputs 56+ 83% 79% Increased low quality research available (i. e. research meets minimum technical standard only) Increased alternative sources (e. g. different versions of articles: preprint, accepted, published) 36 -55 83 54% 37% 64% 63% 52% 39% 32% 35% n=932 n=3133

Back to contents Demographics 84

N=3133 Demographics Age Subject Chemistry 5% Computer Science 6% Earth & Env. Science 8% 16% Engineering Life Sciences 19% Material Science Maths Under 26 26 -35 36 -45 46 -55 56 -65 Over 65 Prefer not to say China 16% 27% 24% 18% 12% USA Russia 2% Germany Gender Rep. of Korea 5% UK 4% Medicine and AH Male; 70% 12% Physics & Astronomy 17% SSE + Arts Hum 66% Research Institute 16% Hospital / Med. Sch. 5% Commercial / Corp. 4% Government 3% College 2% Other 2% Prefer not to say; 2% Position Organization University France India Female; 27% 8% Head of Dept. / Senior Management Senior Researcher/ Middle Management Researcher/ Staff Member Other Canada 17% 37% 11% Western Europe 21% Eastern Europe 10% 6% 5% 5% 4% Latin America 4% 3% 2% Africa 2% 27% Role Number of articles published Research and/or development 48% 21% Teaching 16% 5 or fewer 18% 16 -25 17% 5% 26 -50 Mgmt/Admin. 4% 51 -75 Advisory/Consultancy 2% Practitioner (eng. /tech. ) 1% 3% 8% 6 -15 Practitioner (clinical) Other APAC 41% Middle East 3% 4% R&D and teaching equally 36% North America 19% 9% Other countries Self-describe; 0% 16% 20% Japan 85 Region Country 1% Back to contents 76 -100 21% 11% 8% 101 -200 Over 200 10% 5%

Back to contents Why journal articles viewed are not trustworthy 86

87 Verbatim analysis – Trust in content “. . . Why do you think all/some of the content you encounter is not trustworthy? ” Poor conclusions drawn "Some of the research outputs were based on assumptions rather than on evidence. " Exaggerated findings / lack of information or detail provided Predatory journals publish whatever they are paid to publish and make themselves seem like regular journals. "Because in some of them the interpretation of the obtained results seems forced to show something that is not clearly supported by the findings. " Random sample of 500 comments coded 18. 09. 2020 “Picking data to fit a particular expected result" “Often results are presented to be more conclusive or generalizable than they are when you read the details of the experiment" Because they are published by predatory journals which are completely untrustworthy

88 Verbatim analysis – Trust in content “. . . Why do you think all/some of the content you encounter is not trustworthy? ” Lack of peer review "Some are anecdotal discussions (e. g. , blog posts, editorials) that have not undergone formal peer review" "Anybody can upload a video or article on Internet. " Lack of proper peer reviewing system or low quality journal "A fairly large proportion of the "research outputs" I encounter are via news or newsletter reports, not the original peer reviewed articles. I don't feel that these summaries / reports are as trustworthy as getting the information directly from the published (and peer reviewed) reports themselves. " Questionable quality of peerreviewing in less "prestigious" journals "Unreviewed preprints on arxiv" Random sample of 500 comments coded 18. 09. 2020 "Not all material that is found has validated methodology and not every reviewer will be able to review all types of material to which it is invited" "The manuscript was not peer reviewed (preprint) and contained false statements. There also peer reviewed articles of bad quality. "

89 Verbatim analysis – Trust in content “. . . Why do you think all/some of the content you encounter is not trustworthy? ” "Why the methodology used in the studies was not appropriate, or the conclusions did not reflect the results" Methodological flaws Reproducibility / Generalisability / incorrect methods used "The quality (conceptual depth) of writing about the methodology and results does not lead to confidence in the competence of the researchers. " "The methodology followed may not be good or there is no clarity in the methods followed for deriving the results. " "Methodology not robust. Insufficient primary data presented. " Random sample of 500 comments coded 18. 09. 2020 "Experiments poorly designed, some analyses seemed suspect, areas that I know well improperly characterized" "I feel sometimes tools and techniques used may divert the results up to some extent(although not too much). "

90 Verbatim analysis – Trust in content “. . . Why do you think all/some of the content you encounter is not trustworthy? ” Bias Peer review / funding source / data falsification / pressure to publish = lower quality / negative findings not published "Some researchers have conflicts of interest and bias results - this should be disclosed" "Ultimately, institutional pressure to publish ('publish or perish'). " Random sample of 500 comments coded 18. 09. 2020 "Political Correctness contaminates some topics and influences the conclusions drawn. For that matter it affects whether the research gets published: try questioning "diversity" or "climate change" and see what happens to a MS. It will vanish and never be published no matter how valid its data may be. Some topics are "forbidden" and this makes some content suspect. Not an issue in many fields but in some it is a real problem. " "There is published Research biased by financial or other Support to the authors and not properly declared. " "Images are prone to manipulation and I don't know if I would spot that. "

91 Verbatim analysis – Trust in content “. . . Why do you think all/some of the content you encounter is not trustworthy? ” Lack of supplementary material Unable to validate conclusions "Some journals do not contain the original data. " "Actual data behind graphs are not always provided, which looks suspect. Some interpretations are repeatedly overemphasised in the text, albeit the experimental evidence for them is rather weak. " "Data not made public, experiments are not well conducted etc. " “Missing raw data, suspicious blots, overstated findings" Random sample of 500 comments coded 18. 09. 2020 "Its hard for me to always believe others based on hypothesis. I prefer to rely on data validation. " "Authors often do not provide data/code/tools/proper description of the scenarios used for the evaluation contained in their papers. In particular, the correctness of code used for simulations reported in papers is often unverifiable. "

92 Verbatim analysis – Trust in content “. . . Why do you think all/some of the content you encounter is not trustworthy? ” Negative perception of source Website / author(s) / predatory journals “A few items were from obscure journals, and at first look the quality was low" "The journal has a shady reputation and the article lacks critical information related with the materials and methods. Raises questions about the reliability of the information provided" "Most of the time published in journals "controlled" by a few like-minded people. " "I don't trust the source who publishes them. " “Because was published online v. gr. arxiv" Random sample of 500 comments coded 18. 09. 2020 "Predatory journals that publish whatever they are paid to publish and make themselves seem like regular journals. "

93 Verbatim analysis – Trust in content “. . . Why do you think all/some of the content you encounter is not trustworthy? ” Errors Grammatical / citation / inflated statistical power / calculations / code "The methodology in the papers is not clear. Problems with the statistical test used (not used the proper one)" "Experiments poorly designed, some analyses seemed suspect, areas that I know well improperly characterized" "Typically poor statistical elements that are obvious to an educated reader but clearly were not obvious to reviewers. One fundamental challenge is that statistical complexity in manuscripts is increasing but the number of statistically competent reviewers is not. " “In some cases errors can appear" "Poor statistical analysis, knowledge gaps, lack of fundamental mechanistic understanding" Random sample of 500 comments coded 18. 09. 2020 "The conclusions are not relevant and the results are not supported by rigorous scientific analysis"

94 Verbatim analysis – Trust in content “. . . Why do you think all/some of the content you encounter is not trustworthy? ” Other Human error / lack of knowledge in field / personal experience / critical thinker “Mainly because I do not have enough skill on the topics" “Because I have seen that in some periods of time and some journals there have been some published data that are doubtful to be true. However always it is almost impossible to find out exactly the trustworthy of works up to hundred percent. " "I read a wide variety of material and I necessarily encounter some material that is of low quality" "Mistakes can always be made" "Just statistically speaking, some work I view is going to be false (with benefit of the doubt to the author, unintentionally)" Random sample of 500 comments coded 18. 09. 2020 “Normal doubt"

Back to contents Why research-related profiles viewed are not trustworthy 95

96 Verbatim analysis – Trust in researcher profiles “. . . Why do you think all/some of the research-related profiles (researcher and institutional profiles) you encounter are not trustworthy? ” Exaggeration / Self promotion Falsification "Depending on where the profiles are posted and whether and by whom they are vetted, it would not surprise me to see credentials exaggerated or outright falsified. In addition, I have personally known of individuals who exaggerated or falsified their or their proteges' profiles. " (Social Science, United States of America, 56 to 65) "Both researchers and institutions tend to exaggerate their achievements. " (Chemistry, Poland, 46 to 55) "Sometimes they are inflated. " (Social Science, Portugal, 36 to 45) "Fabrication of resumes etc can be rampant in some areas" (Environmental Sciences, United States of America, 36 to 45) Random sample of 500 comments coded, of which 176 were not applicable so these were removed from the analysis 18. 09. 2020 "Research profiles are becoming more important than research itself. Therefore, researchers tend to create huge, impressive but "empty" profiles" (Immunology and Microbiology, Spain, 46 to 55) "Because there are institutional pressures to exaggerate and I personally know researchers who do so" (Arts and Humanities, Australia, 36 to 45)

97 Verbatim analysis – Trust in researcher profiles “. . . Why do you think all/some of the research-related profiles (researcher and institutional profiles) you encounter are not trustworthy? ” Quality of research "Weak publication profile in standard research journals" (Agriculture, India, 36 to 45) Cited work / reputation of associated journals / metrics "Some are summaries of recent research where access to the original article is unavailable. " (Pharmacology, Toxicology and Pharmaceutics, United States of America, Over 65) “It depends on their h index and career" (Electrical / Electronic Engineering, Fiji, 56 to 65) "I guess it speaks to the organisation's reputation in the first instance. Also based on previous work that has come out of the relevant research group etc. " (Agriculture, Ireland, 46 to 55) "Lack of habitual research work" (Psychology, Brazil, Over 65) “Unconvinced by past examples of their work" (Physics, Italy, 56 to 65) Random sample of 500 comments coded, of which 176 were not applicable so these were removed from the analysis 18. 09. 2020 "They publish low quality studies" (Agriculture, Sweden, 56 to 65) "Their output shows no sufficient comprehension of the subject" (Engineering and Technology, Germany, 36 to 45) “The low quality of papers published" (Medicine and Allied Health, Indonesia, Over 65)

98 Verbatim analysis – Trust in researcher profiles “. . . Why do you think all/some of the research-related profiles (researcher and institutional profiles) you encounter are not trustworthy? ” Unverified Cannot verify the work / references / endorsements / lack of social network "Healthy criticism of unverified/unverifiable profiles. " (Arts and Humanities, China, 56 to 65) "There is no review on the profile" (Immunology and Microbiology, United States of America, 36 to 45) “They do not have academic social networks" (Arts and Humanities, Argentina, 36 to 45) "If I am contacted by someone with a profile disconnected from everybody, I do not trust it" (Engineering and Technology, Switzerland, 26 to 35) "There is no way to check details" (Social Science, Netherlands, Over 65) Random sample of 500 comments coded, of which 176 were not applicable so these were removed from the analysis 18. 09. 2020 "Because it is created at the discretion of the individual. " (Agriculture, Japan, 46 to 55) "The one I encountered last week was in an application for work, where the number of scientific publications was stated to be about 40, but when checking online only nine publications could be tracked. " (Nursing, Sweden, 56 to 65)

99 Verbatim analysis – Trust in researcher profiles “. . . Why do you think all/some of the research-related profiles (researcher and institutional profiles) you encounter are not trustworthy? ” Out of date or poor profiles Spelling / duplicate profiles / lack of information "Poorly written. " (Engineering and Technology, United States of America, 56 to 65) "Unknown institution, grammatical errors, very diverse content" (Psychology, Canada, 36 to 45) "They tend not to be maintained. " (Medicine and Allied Health, United States of America, Over 65) "Lack of information and/or incomplete information" (Materials Science, France, 46 to 55) “Duplicates" (Earth and Planetary Sciences, Korea, South, 56 to 65) Random sample of 500 comments coded, of which 176 were not applicable so these were removed from the analysis 18. 09. 2020 "Electronically generated profiles need error checking which takes time. " (Biochemistry, Genetics, and Molecular Biology, United Kingdom, 36 to 45) "Many are updated infrequently. Others appear to be updated semiautomatically and contain errors. " (Biological Sciences, United States of America, 36 to 45)

Verbatim analysis – Trust in researcher profiles 100 “. . . Why do you think all/some of the research-related profiles (researcher and institutional profiles) you encounter are not trustworthy? ” Source of profile Not reputable / open to edit "I sometimes search Research. Gate, where some researchers may try to inflate their contributions" (Other, Ireland, 46 to 55) "Research-related profiles can be hacked and modified. " (Biochemistry, Genetics, and Molecular Biology, United States of America, Over 65) “They relate to postings on social media sites or unsolicited emails promoting people or institutions which are self-generated. I am just guessing that these may not all be trustworthy - I don't have time to check all of them. " (Social Science, United Kingdom, 56 to 65) "I tend to be cautious about the research -related profiles in cases of less well known (to me) sources. " (Medicine and Allied Health, Greece, 46 to 55) Random sample of 500 comments coded, of which 176 were not applicable so these were removed from the analysis 18. 09. 2020 "You can post anything, and if a site offers no chance to comment, how can you know. . . and, I can't conduct due diligence on every researcher. . . so, I trust my peer group to vet researchers, articles, sources and peer reviewers, given the flood of journals and research articles" (Environmental Sciences, United States of America, 56 to 65) "Profiles sometimes have more of a "branding" or "commercial" function, which makes me a bit critical about the quality of the content. " (Social Science, Denmark, 36 to 45)

101 Verbatim analysis – Trust in researcher profiles “. . . Why do you think all/some of the research-related profiles (researcher and institutional profiles) you encounter are not trustworthy? ” Reputation associated with the Institution / author / country "Certain countries have reputations for copying or forging data and publications" (Biological Sciences, United States of America, Over 65) "I only trust top ranked outlets" (Earth and Planetary Sciences, Canada, Over 65) "Some profiles were from people/institutions that were very obscure. " (Social Science, Australia, Over 65) "Unknown affiliations, publications etc. " (Engineering and Technology, Italy, 46 to 55) Random sample of 500 comments coded, of which 176 were not applicable so these were removed from the analysis 18. 09. 2020 "They are not affiliated with an organization that produces high quality research. " (Economics, United States of America, 56 to 65) "Profiles based on publications in lowranked and/or possibly predatory journals, and researchers in countries and/or institutions that don't have a significant research track record. " (Physics, Australia, 56 to 65)