3 rd Altmetrics Conference Bucharest What matters where

3 rd Altmetrics Conference, Bucharest What matters where? Cultural and geographical factors in science Dan Penny, Springer Nature September 2016

1 “I really need to understand all these new metrics – I’m sure everyone in my lab gets this better than me” - Principal Investigator, UK 3 rd Altmetrics Conference / 27 th September 2016

2 Today Springer Nature and (alt)metrics Recent happenings The effect of multiple metrics Metrics & China, Metrics & Open Access What researchers say they want 3 rd Altmetrics Conference / 27 th September 2016

3 About me Where do I work? Springer Nature’s Strategic & Market Intelligence team • Usage analysis • Competitor analysis & bibliometrics • Author and user interviews & surveys What is our role? • Asking (and answering) good questions • Reflecting the customer’s perspective • Author & reviewer satisfaction What do I work on? Submission decisions - Open data – Science news – Spending on lab equipment – Drivers for book authoring – Peer review – Reproducibility – Salary & Job Satisfaction – Tools usage – Social networks – Reading behaviour – Submission pain points – Early career issues – Protocols – Author reputation – Press office needs – New Nature launches – Author Services – Article summary content – Deposition services And some ideas that might have been ahead of their time… 3 rd Altmetrics Conference / 27 th September 2016

4 3 rd Altmetrics Conference / 27 th September 2016

5 Springer Nature & Bookmetrix 3 rd Altmetrics Conference / 27 th September 2016

6 What metrics matter to book readers? “Which metric matters the most to you, which is the 2 nd most important, which 3 rd and so on? ” • Downloads • Citations • Book reviews • Online mentionsin news, blogs, social media, wiki, or policy docs • Mendeley readers In progress - low base sizes! 3 rd Altmetrics Conference / 27 th September 2016 Downloads Ranked higher Ranked lower Citations Online mentionsin news, blogs, social media, wiki, or policy Book reviews docs Mendeley readers Other Asia China Europe India N. America S. America

7 3 rd Altmetrics Conference / 27 th September 2016

8 3 rd Altmetrics Conference / 27 th September 2016

9 Goodbye to the Impact Factor? Nature Research journals now display multiple metrics for each of its journals, including the 2 and 5 year Impact Factor, Immediacy Index, Eigenfactor, Article Influence Score and a 2 -year median. Nature has also stopped its email campaigns that would offer Nature personal subscriptions at a $ rate equivalent to the new Impact Factor. “For better or worse, in survey upon survey, our authors cite impact factor as a primary consideration in driving decisions about where to publish their work, and whether to write for a journal. This is despite increasing calls for due caution in using the journal impact factor and a growing recognition that the over-reliance on the journal impact factor as an indicator for the quality of individual articles is damaging to the practice of science. There is no question that decreasing a reliance on the impact factor as an assessment metric is proving to be exceedingly challenging for journals, publishers, funders, research institutions and researchers. ” 3 rd Altmetrics Conference / 27 th September 2016

10 ‘altmetrics’ means different things to different people 3 rd Altmetrics Conference / 27 th September 2016

11 ‘altmetrics’ means different things to different people The REF 2104 impact case studies are the first thing that occurs to several senior editorial staff at Springer Nature when you talk about alternative metrics. This is likely to reflect the findings of ‘The Metric Tide’ report in July 2015, which: - expressed concern that some indicators can be misused or ‘gamed’: journal impact factors, university rankings and citation counts being three prominent examples; - concluded that in assessing research outputs in the REF, it is not currently feasible to assess research outputs or impacts in the REF using quantitative indicators alone; - recommended that metrics should support, not supplant, expert judgement. http: //www. hefce. ac. uk/media/HEFCE, 2014/Content/Pubs/Independentresearch/2015/The, Metric, Tide/2015 _metric_tide_executive_summary_and_recommendations. pdf 3 rd Altmetrics Conference / 27 th September 2016

12 A year later, the wheels are turning - in the UK, at least • A new forum for responsible metrics is being set up as a partnership between Higher Education Funding Council for England (HEFCE), Research Councils UK (RCUK), Wellcome, Universities UK and JISC to advance the agenda set out in The Metric Tide. • The forum will develop a programme of activities to support the responsible use of research metrics in higher education institutions and across the research community in the UK. This will include advice on, and work to improve, the data infrastructure that underpins metric use. In this way, the whole research community can benefit from the more judicious use of metrics. • The Association for Research Managers and Administrators (ARMA) is offering a course to its members on the Responsible Use of Metrics in May 2017. • This is great – but Stacy Konkiel’s Altmetrics blog from last year: “The Metric Tide calls for governments and funders to offer more investments in the research information infrastructure, especially to increase interoperability with systems like ORCID. We’d add to that and call for the private sector (including other metrics aggregators) to kill information silos in favor of building or allowing technologies that “play well” with others even if they’re commercial. ” 3 rd Altmetrics Conference / 27 th September 2016

13 The librarian’s valuable role Jenny Dela. Salle & Elsevier Library Connect Set of cards that can be printed as a poster or as cards for training sessions. She suggests asking users to think about is each metric one for authors, one for documents or one for journals? Set is intentionally not comprehensive: the ones chosen are perhaps those that librarians will most frequently come across. 3 rd Altmetrics Conference / 27 th September 2016

14 But the tide is still rising… “There are powerful currents whipping up the metric tide. These include growing pressures for audit and evaluation of public spending on higher education and research; demands by policymakers for more strategic intelligence on research quality and impact; the need for institutions to manage and develop their strategies for research; competition within and between institutions for prestige, students, staff and resources; and increases in the availability of real-time ‘big data’ on research uptake, and the capacity of tools for analysing them. ” The Metric Tide, July 2015 3 rd Altmetrics Conference / 27 th September 2016 Because of these currents, there is a resulting increase in the sheer visibility of metrics – whether to measure journal, article or author performance: • Research. Gate’s RG Score • Relative Citation Ratio – NIH-introduced metric being adopted by Digital Science • Elsevier’s Cite. Score is under development – Scopus is currently visiting publishers, libraries and researchers

15 3 rd Altmetrics Conference / 27 th September 2016

16 3 rd Altmetrics Conference / 27 th September 2016

17 Awareness of (alt)metrics? Regional variance? In st Re se ar Ot he ch Ga te Op en A Em cc es ai s l t he au th or Pa y p er vi ew Re se ar ch 4 L ife De ep dy ve ib r. . . r L n o itu tio r What tools/sites do you use to get access to literature etc. ? 100% 80% 60% 40% 20% 0% Europe N. America Asia (inc. Middle East) Rest of World r he Ot ry to ct S pa ub g P Ha r W Europe From this it is clear that beyond the incumbent tools, awareness and usage varies quite considerably by region – the data suggest that Research. Gate has lower uptake in N. America, and Altmetric has lower uptake in Asia. 3 AM conference blog by Utrecht: Librarians demonstrate a higher awareness than researchers for all tools. This may be because they are more aware of the criticisms of traditional metrics tools and are in a good position to explore alternatives due to their knowledge of the information landscape. On the other hand, the stakes for librarians are lower as they are not the ones that are dependent on metricsbased criteria for funding and evaluation. zin Im lis h o r. . PL o. S tm et ric us Al ie n f S c eb o Sc op ce R JC . What tools/sites do you use to measure impact? 70% 60% 50% 40% 30% 20% 10% 0% Plenty of studies taking place, including Utrecht’s recent survey of over 20, 000 individuals, carried out by Bianca Kramer and Jeroen Bosman. N. America Asia (inc. Middle East) 3 rd Altmetrics Conference / 27 th September 2016 Rest of World Does the gap between the what researchers use and what librarians recommend mean that librarians are out of touch with what researchers deem important? Or is there still a long way to go in making researchers aware of altmetrics tools?

18 The effect of multiple metrics 3 rd Altmetrics Conference / 27 th September 2016

19 Further studies – and caveats! Springer Nature surveys its audience from time to time – we try to minimise this, believe it or not. . . Country comparisons are a challenge – it’s important to ensure that a sample has a similar profile in terms of role, seniority and disciplinary coverage. And due to language and cultural differences, both qualitative and quantitative research is at risk of misunderstandings or other bias – we are doing our best to provide the option to offer local languages in online surveys. 3 rd Altmetrics Conference / 27 th September 2016

20 What’s the result of this growth in metrics? Confusion? ("Beyond a certain point are more metrics useful? ” bit. ly/2 c. KCg 1 m – Euan Adie, Altmetric. com) “I really need to understand all these metrics – I’m sure everyone in my lab gets this better than me”- PI, UK “I don’t take any notice of my Research. Gate score, because I cannot work out how it is calculated”- Professor, USA “It’s not my job to understand all this – my supervisor recommends which are the best journals for me to read”- Postdoc, Japan “We don’t really need to look at these kinds of metrics – we know that we are producing high quality research”- VC, Cold Spring Harbor What awareness is there? Is there actually much demand amongst researchers to better use metrics? 3 rd Altmetrics Conference / 27 th September 2016

21 Awareness of metrics - recent Springer Nature survey findings “How familiar are you with each of the following metrics? ” n=2, 734 c. f. Rodrigo Costas and Htet Aung – following this! I use this I am aware of this, but have not used it before I do not know what this is Impact Factor 95% h-index 18% Eigenfactor 10% Relative Citation. . . 8% Source Normalise. . . 6% HITT Score 22% 30% 52% 43% 19% 3 rd Altmetrics Conference / 27 th September 2016 10% • Awareness of h-index is encouragingly positive • HITT Score is an invented metric, intended in part to gauge reliability of the answers • Approximately 50 -60% of respondents seem to be unaware of all but the most common metrics listed here. 48% 32% 31% We recently asked a broad range of Springer, Nature and Biomed Central authors about their views on metrics 5% 67% SCImago Journal. . . • 59% 63% 79%

22 Awareness of metrics – variance by region “I do not know what this is” “How familiar are you with each of the following metrics? ” n=2, 734 • • US culture of research assessment is less developed? Does India have a strong culture of bibliometric analysis? 0% Impact Factor h-index SCImago Journal Ranking 100% China India Japan Europe Relative Citation Ratio (RCR) Eigenfactor Source Normalised Impact per Paper (SNIP) 3 rd Altmetrics Conference / 27 th September 2016 50% USA

23 But yes, there is demand – e. g. when deciding where to submit “When deciding where to submit, to what extent would you like more factual data from databases, publishers or other sources on the following? ” Yes, slightly more South America North America Europe Australasia Asia 84% Yes, lots more 82% The readership of the journal 76% 78% 87% 38% 43% 33% 87% 81% Quality of peer review 85% 81% 91% 48% 37% Quality of peer review The readership of the journal 3 rd Altmetrics Conference / 27 th September 2016 42% 81% Relevance of a journal to your discipline 71% 70% 73% 88%

24 Follow up on Nature’s 2015 Author Insights study 3 rd Altmetrics Conference / 27 th September 2016

25 In 2015, we found there were some significant differences by region Peer review quality – “Very important” or “Important” 97% 89% USA 90% Europe Journal readership – “Very important” or “Important” 95% 94% 91% India China Japan 91% Brazil USA 86% Europe India 81% 79% China Japan Speed of decision – “Very important” or “Important” 89% 83% 84% 77% USA Europe India 3 rd Altmetrics Conference / 27 th September 2016 China Brazil 92% 87% 95% 89% Impact Factor – “Very important” or “Important” 86% 93% Japan Brazil 73% 74% USA Europe India China Japan 79% Brazil

26 Metrics & China, Metrics & Open Access 3 rd Altmetrics Conference / 27 th September 2016

27 Metric usage: China The way in which authors in China vs. Ro. W are communicating their research varies. Authors from Ro. W seem to be more likely to communicate their work online, for example on a professional profile, personal website, or through social media , science specific or not. The findings show that, 21% fewer authors from China said that they promoted their most recently published work on their professional page. Authors from China are however slightly more likely to email peers, colleagues or specific mailing lists than authors based in Ro. W. (n=403 China, 1, 152 Ro. W) 3 rd Altmetrics Conference / 27 th September 2016 % authors who used listed method to communicate most recent research to peers and wider audience Presenting at a conference 46% 45% 47% Emailing peers and colleagues Professional page, e. g. university page 35% 14% Scientific social networking site 25% 7% Institutional press office promoting research Uploading findings to a repository Social media 2% Personal website, e. g. blogs 23% 19% 15% 10% 15% 8% 7% Public engagement activities 60% 15% 16% 4% 7% 4% Promoting metrics around research 1% 2% Podcast or video 1% 2% Other 0% 11% Do not usually promote research 14% Emailing a subject-specific mailing list 0% Ro. W China 50% 100%

28 China: Turning Point http: //www. nature. com/press_releases/turning_point. pdf Qual (32 interviews) & quant (1690 respondents) by Nature Publishing Group, 2015 1) Chinese assessment policies are driving scientists to target high-impact journals. Unlike in Europe, funders and institutes are not widely seeing the need to ensure this trend does not hamper the discovery and innovation process overall. - 87% of the surveyed scientists indicated that they are likely to publish relatively fewer papers each year in future, but target these to higher profile journals. - Almost all surveyed PIs indicated that they feel pressure to pursue hot topics, with approximately half identifying the lure of publishing in high-impact journals as the most important reason. Pressure to produce extremely high quality output may make researchers reluctant to share their data or delay publishing until they can get their paper accepted in a highprofile journal. This is not unique to China. But China needs to do more to share of science and improve positive societal impacts from research. 3 rd Altmetrics Conference / 27 th September 2016

29 3 rd Altmetrics Conference / 27 th September 2016

30 China: Turning Point http: //www. nature. com/press_releases/turning_point. pdf 2) Communication by the scientific community to wider society is increasingly important. When the public begins to question spending on science, the importance of communication is even more evident. - The researchers surveyed by NPG almost unanimously agreed that scientists should play a role in communicating to the public. However, only around half had experience of some type of science communication in the past three years. This lack of engagement is compounded by the low level of scientific literacy of China’s general populace. - Chinese scientists are afraid to engage outside of their community because they fear being judged by peers or misunderstood. China’s culture of modesty thwarts interactions between scientists and the media, and the many administrative hurdles in the Chinese academic culture represent another important factor. - Chinese institutes and funding bodies could incentivize good science communication by tying it to researcher assessment. For instance, a more-nuanced rather than pure publication driven assessment scheme that gives credit to good science communication activities, as well as collaborations with industries, would help. 3 rd Altmetrics Conference / 27 th September 2016 “(Science communication with public media) This may be all right in the western countries as they respect diversity and individuality. But in China, as a young scientist, this may cause trouble and I don’t want to do that. I’m happy to do that when I become a senior heavyweight in my field. ” –PI, age 35 -44, Shanghai

31 A ‘culture of measurement’ may be connected to other behaviour – such as open access publication The University of Utrecht’s 2015/16 tools survey was sent out to Nature authors with unique links for each respondent so that the data could be analysed alongside the answers they had given in a previous survey. The next slides show the data from Utrecht’s ‘survey of scholarly communication’ analysed alongside the data on how many open access papers authors had published over the last 3 years from Nature Publishing Group’s ‘ 2015 author survey’. For this analysis respondents were split into three groups, based on how frequently they chose to publish via an OA model (this was based on the mean number of OA papers being determined as 3): • Frequent: Authors that have published more than 1 OA paper a year (n=218) • Infrequent: Authors that have published 1 OA paper a year or fewer (n=510) • Traditional only: Authors that have published no OA papers at all (n=428) From our initial analysis it seemed to suggest that there was a link between certain tool usage and how frequently an author published via an OA model. 3 rd Altmetrics Conference / 27 th September 2016

Some researcher profiles, such as ORCID, seem to be used more by authors who publish frequently via an OA model 32 “What researcher profiles do you use? ” 90% 80% Research. Gate % of authors saying they use 70% Google Scholar Citations 60% 50% 40% ORCID 30% Profile page at own instituion Academia. eu 20% Research. ID 10% Other 0% +1 OA article per yr 1 or less OA articles per yr Academia. eu Mean Standard Deviation T Test 2. 57 3. 89 0. 4351 3 rd Altmetrics Conference / 27 th September 2016 No OA articles Google Scholar Research. Gate Research. ID ORCID Citations 2. 44 3. 91 0. 1590 2. 52 3. 58 0. 0995 3. 28 4. 63 0. 0006 2. 98 4. 35 0. 0020 (Google Scholar Citations n=791, Research. Gate n=872, ORCID n=518, Academia. eu n=258, Research. ID n=291, Profile Pages at own Institution n=310 Other n=97) These significance levels do suggest that there is a relationship between publishing OA and use of these tools

Tools used to measure impact have a greater uptake amongst authors who publish frequently via an OA model 33 “What tools/sites do you use to measure impact? ” 80% 70% 60% Web of Science JCR 50% 40% Scopus 30% 20% Altmetric 10% Others PLo. S 0% +1 OA article per yr 1 or less OA articles per yr JCR Mean OA papers per user Standard Deviation T Test 3 rd Altmetrics Conference / 27 th September 2016 Scopus 2. 45 3. 74 0. 0435 No OA articles Altmetric 2. 62 4. 19 0. 0220 3. 02 3. 97 0. 0032 Web of Science PLo. S 2. 61 4. 04 0. 0004 (JCR n=518, Wo. S n=589, Altmetric n=184, Scopus n=379, PLo. S n=131, Other n=122) 3. 74 4. 86 0. 0001

34 What researchers say they want 3 rd Altmetrics Conference / 27 th September 2016

35 2016: Impact Factor is still most important for China Speed of decision still important for China & India “If available, how important would the following types of information about journals be in helping you decide where to submit in the future? ” China India Japan Europe USA 95% 85% 75% 65% 55% 45% 35% Impact Factor Average time to Information on Satisfaction of Alternative peer review the subject areas publication who readers are press coverage for published authors citation metrics covered by papers previously (e. g Eigenfactor published in the Score) journal 3 rd Altmetrics Conference / 27 th September 2016

36 But information on who the readers are is perhaps where there is most potential to add – and altmetrics can play a part here China India Japan Europe USA 95% 85% 75% 65% 55% 45% 35% Impact Factor Average time to Information on Satisfaction of Alternative peer review the subject areas publication who readers are press coverage for published authors citation metrics covered by papers previously (e. g Eigenfactor published in the Score) journal 3 rd Altmetrics Conference / 27 th September 2016

3 rd Altmetrics Conference / 27 th September 2016 al d ec. . llo w -re se ar c up er y s f m of in i on s o ip sh er Re ad nd vi so r a l b na ur jo a ea pr ph ic s ra Ge og fe us try y i nd sh ip ad er re of d p w hi rs de ea f r o ze Si Op USA Europe India China Japan Brazil Less influence he r en t er nm ov r g o in e ip l isc y d ith in de ea y r ar Cr os s-d isc ip Base sizes • USA 6, 191 • Europe 1, 757 • India 498 • China 1, 391 • Japan 779 • Brazil 211 Influence on reputation Impact Factor was high, but the aspects about readership are interesting m rs hi p More influence lin 2015: “We’d like you to think about a journal in your field which you consider to have a good reputation. What is it about this journal that gives it a high reputation? ” . 37

It’s nearly 11 o’clock 38 https: //www. dunkindonu ts. com/DDBlog/2011/09/ new_dunkin_donuts. html #sthash. k. Ww. NE 9 TZ. dpbs 3 rd Altmetrics Conference / 27 th September 2016

39 It’s nearly time for coffee IMHO book and journal authors have historically not thought very much about the readers of their research – perhaps because this information has been unknowable. Making reader data available will encourage the question: “Who is my research for? Who will this benefit? ” The value of a wide variety of metrics, made easy to engage with, is not necessarily about better assessment – rather, it can be about exploration, and how it helps everyone to reach a better question. I’m looking forward to - Talking about how we can better help authors and readers with metrics - Hearing about what’s going on outside the UK and Europe - Exploring! 3 rd Altmetrics Conference / 27 th September 2016

40 When multiple metrics can be A Good Thing “Britain easily beat Japan at the Rio Olympics” 3 rd Altmetrics Conference / 27 th September 2016

41 “Yeah, but per capita Britain didn’t do that well” 3 rd Altmetrics Conference / 27 th September 2016 http: //www. medalspercapita. com/#about: 2016

42 “Yeah - but by GDP Japan came 70 th!” 36 Great Britain 64 USA 70 Japan 3 rd Altmetrics Conference / 27 th September 2016

43 “Yeah - but by team size, Great Britain only just beats North Korea…<pause> …so how does Great Britain’s Olympic Committee decide how big its team should be…? ” 3 rd Altmetrics Conference / 27 th September 2016

44 What’s the result of multiple metrics? Through having multiple metrics readily available, easily explored and clearly defined, it is possible to explore a topic and reach a better question: which here might be “Does Great Britain’s Olympics performance represent success…? ” “The success of the sporting model holds wider lessons for government. The [Olympic] model shows that it pays to play to your strengths. In economic terms, Britain’s strongest areas are financial services, higher education and very specialised manufacturing. It would be nice to win at beach volleyball, but then, it would be nice to make steel at competitive prices. Neither warrants throwing good and scarce money after bad. ” https: //www. ft. com/content/0 ba 195 cc-608 a-11 e 6 -b 38 c-7 b 39 cbb 1138 a 3 rd Altmetrics Conference / 27 th September 2016

45 Mulţumesc! (Thank you) Dan Penny, Springer Nature d. penny@nature. com Some of these raw data are available (or will soon become available) at https: //figshare. com/authors/Nature_Publishing_Group_NPG_/6 11499 …or feel free to contact me 3 rd Altmetrics Conference / 27 th September 2016