New roles for research libraries in performance measurement
New roles for research libraries in performance measurement? OCLC Changing support/supporting change, 10 -12 June 2014, Amsterdam Paul Wouters, Centre for Science and Technology Studies (CWTS)
CWTS Profile • Innovative Contract Research Organization • Research Performance- and Benchmark studies • Tailor-made solutions • Input for strategic decision making • Based in Leiden, the Netherlands 1
• Backbone: the Center for Science and Technology Studies (CWTS) of Leiden University • Recognised leader in the field for more than 20 years: bibliometrics, scientometrics, and research evaluation • Strong expertise: science and technology indicators & network analyses and mapping 2
Clients face key questions • How should we monitor our research? • How can we profile ourselves to attract the right students and staff? • How should we divide funds? • What is our scientific- and societal impact? • What is our area of expertise? • How is our research trans-disciplinary connected? 3
Products and Services CWTS B. V. Monitoring & Evaluation • Performance Analysis • Leiden Ranking • Benchmark Reports • Researcher Profiles • Journal Profiles & Indicators 4
Products and Services CWTS B. V. Advanced Analytics Tailor-made analysis based on network analysis, text mining and visualisation techniques • Research strengths analysis • Find blind spots/hot spots • Identification of partners/potential new staff • Enhanced collaborative network analysis 5
Products and Services CWTS B. V. Advanced Analytics Mapping & Network Analysis • VOSviewer 1. 5. 5 Readily available via www. vosviewer. com 6
Products and Services CWTS B. V. Training & Education • CWTS Course 'Measuring Science and Research Performance' 7
New roles for research libraries • Increased bibliometric services at university level available through databases • Increased self-assessment via “gratis bibliometrics” on the web (h-index; publish or perish; etc. ) • Emergence of altmetrics • Increased demand for bibliometrics at the level of the individual researcher • Societal impact measurements required • Career advice – where to publish? 8
Altmetrics projects 9
Promise and potential • Overcome limitations peer review • Be faster than citation analysis (“real-time”) • Be able to capture non-traditional influence and activity • Various projects at CWTS: – Open Access Laboratory (eg editorial board membership indicators) – Usage, readership, and download analysis (eg Mendeley) – What do altmetrics indicators signify or represent? – Google Scholar for SSH 10
Extensive comparison of altmetric indicators with citations (Costas, Zahedi, Wouters, 2014 – with Altmetric. com) • 15%-24% of the publications present some altmetric activity 11
Be e fo r 20 13 20 12 20 11 20 10 20 09 20 08 20 07 20 06 20 05 20 04 20 03 20 02 20 01 20 00 19 99 19 98 19 97 19 96 19 95 19 94 19 93 19 92 19 91 19 90 90 19 350000 30% 300000 25% 250000 20% 200000 15% 150000 10% 100000 5% 0 0% 12
Extensive comparison of altmetric indicators with citations (Costas, Zahedi, Wouters, 2014 – with Altmetric. com) • 15%-24% of the publications present some altmetric activity • Most active: social sciences, humanities and the medical/life sciences • Positive correlations but relatively weak correlations with citations • Altmetrics do not reflect the same concept of impact as citations 13
14
The ACUMEN project 15
Academic Careers Understood through Measurements and Norms European 7 th Framework collaborative project Capacities, Science in Society 2010 Grant Agreement: 266632 9 institutional partners, in 7 countries
ACUMEN research assessment of scientometric indicators in performance evaluation Common Data Strategy 15 European countries assessment of webometric (and altmetric) indicators comparative analysis of peer review systems in Europe analysis of gender dimension in researcher evaluation ethnographic study of important evaluation events Bulgaria Czech Republic Denmark Estonia Finland France Germany Hungary Israel Italy Netherlands Poland Slovenia Spain United Kingdom 4 Academic Disciplines (a) astronomy and astrophysics (b) public and occupational health (c) environmental engineering (d) philosophy (including history and philosophy of science) Tatum & Wouters | 14 November 2013
portfolio aim is to give researchers a voice in evaluation narrative e ex pe rtis influenc e ➡evidence based arguments ➡shift to dialog orientation ➡selection of indicators ➡narrative component ➡Good Evaluation Practices ➡envisioned as web service ut p t ou
ACUMEN Portfolio Evaluation Guidelines Career Narrative Links expertise, output, and influence together in an evidence-based argument; included content is negotiated with evaluator and tailored to the particular evaluation - aimed at both researchers and evaluators - development of evidence based arguments (what counts as evidence? ) - expanded list of research output - establishing provenance Expertise Output Influence - scientific/scholarly - technological - communication - organizational - knowledge transfer - educational - publications - public media - teaching - web/social media - data sets - software/tools - infrastructure - grant proposals - on science - taxonomy of indicators: bibliometric, webometric, altmetric - on society - guidance on use of indicators - on economy - on teaching - contextual considerations, such as: stage of career, discipline, and country of residence Tatum & Wouters | 14 November 2013
Publications related to career path 14 Website launched, advisory committees joined Lic Phil 12 10 Project 2000 -3 Project 2004 -6 1 yr Project PHD 8 3 m project Publications 1 yr project 6 4 FR UK DK DE USA IL DE 2 0 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
Portfolio & Guidelines ➡ Instrument for empowering researchers in the processes of evaluation ➡ Taking in to consideration all academic disciplines ➡ Suitable for other uses (e. g. career planning) ➡ Able to integrate into different evaluation systems © 2014 Paul Wouters. This work is licensed under a Creative Commons Attribution 3. 0 Unported License. Suggested attribution: “This work uses content from "New roles for research libraries in performance measurement? " © Paul Wouters, used under a Creative Commons Attribution license: http: //creativecommons. org/licenses/by/3. 0/” Tatum & Wouters | 14 November 2013
- Slides: 23
