Peer Review Evidence experiments opinion Dan Morgan Senior

Peer Review – Evidence, experiments, opinion Dan Morgan – Senior Manager, Global Academic Relations – Access, Elsevier USA

Sources of data • 2009 Peer Review Survey conducted by Sense About Science and Elsevier (www. senseaboutscience. org. uk) (n=4037) • Elsevier Researcher Insights Index/Core Trends ◦ Studies are completed twice a year (Q 1 & Q 3) ◦ Research is conducted anonymously and the respondent does not know that Elsevier is completing the survey until the end. ◦ Respondents are representative of the broader community (i. e. sample is not limited to Elsevier authors) ◦ At least 1500 responses per wave (one has 4000) • Reviewer Feedback Program • Author Feedback Program • Editor Feedback Program • Other sources as noted

3 Highlights • Majority believe that peer-reviewed journals are required (2009 Peer Review Survey) ◦ 84% believe without peer review there would be no control in scientific communication • Majority believe that peer review improves an article (2009 Peer Review Survey) ◦ 91% of authors believe that their last paper was improved as a result of peer review; 82% agreed in general that peer review greatly helps scientific communication • Majority committed to peer review (2009 Peer Review Survey) ◦ 86% say they enjoy reviewing and will continue to review, 89% of authors have reviewed in the previous 12 months. On average 8 papers per year; range: anything between 2 and 30 papers a year

Who is bearing the burden of peer review? Proportion of Global Reviews* % of global reviews vs % global research output *Based on data from Elsevier Proportion of Global Papers (source Scopus) 4

Peer review – known issues Recognition for reviewers Role of the readers? Citation and feedback Transparency of review outcomes Overload of submissions Finding potential reviewers Reviewers Accept/reject invitation Reviewer’s interest & time Publication Editors communication to reviewers and author Author revision Editorial decision Speed of 1 st decision Guidelines and tools provided to reviewers Reviewers recommendations Role of the publisher Quality of peer review

Bearing the burden - finding and retaining good reviewers Main issues for reviewers: • Burdened with too many requests for review • Asked to review papers outside their main expertise • Have to deal with bad quality papers Solutions Elsevier (and other publishers) are working on: Tools that enable journals and Editors to extend pool of reviewers • Reduce the time spent on reviewer identification • Accessible info on new reviewers • Automate things you would ordinarily have to look up/check • Enhance quality of peer review process by finding most appropriate reviewers • Ultimately, increase reviewer satisfaction: more reviewers receiving fewer papers of higher quality of greater relevance

Experimentation: Peer review pilots, initiatives (1) 1. Efficiency: re-using reviewer reports ◦ Re-use of reports in “cascading model’’ (Elsevier ATS/ “Results In”, Plo. S ONE, Nature Communications, BMJ Open) ◦ Re-use of reports in inter-publisher journal consortia (Neuroscience field NPRC) 2. Efficiency: author opt-in possibilities ◦ Fast & light review, versus slow & in depth (Journal of Air Transport Management) ◦ Authors to opt in for skipping 2 nd review at revision (BMC Biology) ◦ Start review after initial acceptance by Editor and online availability (Atm Chem Phys - EGU) 3. Increase transparency of peer review ◦ Show review reports online (EMBO, Agricultural and Forest Meteorology) ◦ Reviewers have the option of revealing their identity

Peer review pilots and initiatives (2) 4. Post-publication commenting ◦ Nature and Cell - trials in 2006 and 2010 ◦ PLo. S ONE, BMJ, Faculty of 1000 Research ◦ Pre-publication commenting/circulation prior to journal submission (ar. Xiv) 5. Increase chances that reviewers accept to review ◦ Monetary incentive, publishing times per named reviewer (Journal of Public Economics) ◦ Reviewers to browse through available submissions (Chemical Physics Letters) 6. Reward or recognition ◦ Publish list of top reviewers in journals and/or at conferences ◦ Provide best-reviewer ‘’certificates’’ ◦ Badges

9

Increase transparency of peer review pilot

11 Post-publication comments

Peer review: Likely to submit/review – by age Q 16. How likely is it you would submit/review a research article to a journal that conducted the following form of assessment (Researcher Insights Index) Single blind Double blind Post-pub Open PR (reviewer assessment, but assessment no PR (reviewer known (reviewer name (reviewer report name and report PR before to author only) next to article) but no name) posted) publication Submit Review Submit Review All 91% 82% 48% 42% 45% 38% 35% 37% 22% 26% 45% 14% 17% 35 and under 88% 90% 84% 83% 57% 44% 54% 42% 39% 37% 29% 50% 51% 14% 19% 36 -55 90% 91% 83% 84% 46% 40% 43% 37% 35% 37% 28% 25% 45% 18% 56 and over 93% 91% 77% 48% 44% 42% 38% 32% 34% 26% 40% 13% Younger researchers were more likely to be in favor of submitting to all variations of open peer review.

Registered reports: Cortex journal

Registered reports: Cortex journal (2) • Experimental methods and proposed analyses are pre- registered and reviewed before data are collected • Agreed publication of their future results providing that authors adhere precisely to registered protocol • Experimental data submitted with Registered Reports made publicly available

Reproducibility considerations • Internal experiments and pilots ◦ E. g. Encouragement/acceptance of replication studies Journal of Research in Personality • Engagement with external pilots ◦ Reproducibility project – Open Science Framework (initial discussions) https: //openscienceframework. org ◦ Reproducibility Initiative – Science Exchange (Mendeley, initial discussions) https: //www. scienceexchange. com/reproducibility • Study/embed the incentives rewards (e. g. Badges? Profiles? )

Summary – what is important? • Stated standards, and adherence to them – full disclosure • Stated processes, and adherence to them – full disclosure • Business model should not matter – the standards and processes should stand separate from model • (Although human nature means that people may always place more scrutiny on any business model where financial viability is based closely on volume - fairly or unfairly) • All need to work together to develop and police these standards • Generational differences will become more evident