The MultiDimensional Research Assessment Matrix Henk F Moed

The Multi-Dimensional Research Assessment Matrix Henk F. Moed Elsevier, Amsterdam, the Netherlands Seminar Research Evaluation in Practice, National Geographic Society, Washington DC, 17 October 2012

Short CV Henk F. Moed Years Position 1981 -2009 Staff member at Centre for Science and Technology Studies (CWTS), Leiden Univ. 2009 Professor of Research Assessment Methodologies at Leiden University 2010 – Elsevier, Sci. Val Dept. Senior Scientific Sept 2012 Advisor As from Elsevier, AGRM Dept. Head of Informetric Sept 2012 Research Group

Contents 1 Boundaries of the playing field; rules of the game 2 The multi-dimensional research assessment matrix 3 Towards more sophisticated indicators: combining big data sets

(o) Bibliometrics makes sense

Which country has these USA main collaborators? Main collaborators UK China Germany Canada

Which country has Brazil these main collaborators? Main collaborators Argentina USA Portugal France Chile

Which country has these main collaborators? Malaysia Main collaborators Thailand India Singapore Iran UK

Which country has these main collaborators? Romania Main collaborators France Hungary Germany Italy Bulgaria

Which country South has these main collaborators? Africa Main collaborators UK China USA Nigeria Australia Netherlands

Contents 1 Boundaries of the playing field; rules of the game 2 The multi-dimensional research assessment matrix 3 Towards more sophisticated indicators: combining big data sets

(i) Bibliometric research assessment is potentially a proper tool to consolidate academic freedom

(ii) Bibliometric tools help establishing a longer term perspective in research funding

(iii) One must be cautious using “societal benefit” as an assessment criterion of basic research: it can not be measured in a politically neutral way

(iv) Citations measure scientificscholarly impact rather than quality or validity

(v) Citation counts in social sciences and humanities may be influenced by political ideologies

Citation impact and ideology Fall of the Berlin wall in Nov. 1989

(vi) The future of research assessment lies in the intelligent combination of metrics and peer review

(vii) Case study on funding policies of a National Research Council reveals biases in peer review

Affinity Applicants – Evaluation Committee 0 Applicants are/were not member of any Committee 1 Co-applicant is/was member of a Committee, but not of the one evaluating 2 First applicant is/was member of a Committee, but not of the one evaluating 3 Co-applicant is member of the Committee(s) evaluating the proposal 4 First applicant is member of the Committee(s) evaluating the proposal

% SUBMITTED APPLICATIONS For 15 % of SUBMITTED applications an applicant is a member of the evaluating Committee (Affinity=3, 4) AFFINITY APPLICANT - COMMITTEE

% GRANTED APPLICATIONS Probability to be granted increases with increasing affinity applicants-Committee AFFINITY APPLICANT - COMMITTEE

Logistic regression analysis: Affinity Applicant-Committee has a significant effect upon the probability to be granted MAXIMUM-LIKELIHOOD ANALYSIS-OF-VARIANCE TABLE (N=2, 499) Source DF Chi-Square Prob ------------------------------INTERCEPT 1 18. 47 0. 0000 CITATION IMPACT APPLICANT 3 26. 97 0. 0000 ** Rel transdisc impact applicant 1 0. 29 0. 5926 AFFINITY APPLICANT-COMMITTEE 2 112. 50 0. 0000 ** Sum requested 1 45. 47 0. 0000 ** Institution applicant 4 25. 94 0. 0000 ** LIKELIHOOD RATIO 199 230. 23 0. 0638

(viii) Data must be accurate and verifiable

(ix) Assessed researchers must have the opportunity to check data and comment on outcomes

(x) A framework is needed to characterize and position bibliometric indicators and products

Contents 1 Boundaries of the playing field and rules of the game 2 The multi-dimensional research assessment matrix 3 Towards more sophisticated indicators: combining big data sets

The Multi-Dimensional Research Assessment Matrix Expert Group on the Assessment of University-Based Research (AUBR, 2010)

Multi‐dimensional Research Assessment Matrix (Part) Unit of Purpose assessment Individual Allocate resources Research Improve group performance Output dimensions Research productivity Quality, scholarly impact Innovation and social benefit Sustainability & Scale Department Monitor research programs Institution Increase regional engagement Research Promotion, Research field hiring infrastruct. Bibliometric Other indicators Publications Peer review Journal citation impact Actual citation impact Internat. coauthorship citation ‘prestige’ Patents, licences, spin offs Invitations for conferences External research income Ph. D completion rates

Multi‐dimensional Research Assessment Matrix (Part) Unit of Purpose assessment Individual Allocate resources Research Improve group performance Output dimensions Research productivity Quality, scholarly impact Innovation and social benefit Sustainability & Scale Department Monitor research programs Institution Increase regional engagement Research Promotion, Research field hiring infrastruct. Bibliometric Other indicators Publications Peer review Journal citation impact Actual citation impact Internat. coauthorship Read columnwise citation ‘prestige’ Patents, licences, spin offs Invitations for conferences External research income Ph. D completion rates

Multi‐dimensional Research Assessment Matrix (Part) Unit of Purpose assessment Individual Allocate resources Research Improve group performance Output dimensions Research productivity Quality, scholarly impact Innovation and social benefit Sustainability & Scale Department Monitor research programs Institution Increase regional engagement Research Promotion, Research field hiring infrastruct. Bibliometric Other indicators Publications Peer review Journal citation impact Actual citation impact Internat. coauthorship citation ‘prestige’ Patents, licences, spin offs Invitations for conferences External research income Ph. D completion rates

Multi‐dimensional Research Assessment Matrix (Part) Unit of Purpose Output Bibliometric Other assessment dimensions indicators Individual Allocate resources Research group Research Publications Peer review productivity Improve Quality, performance scholarly impact Department Monitor research programs Innovation and social benefit Journal citation impact Patents, licences, spin offs Actual citation impact Invitations for conferences Institution Increase Sustainabiregional lity & Scale engagement Internat. co- External authorship research income Research field Promotion Research infrastruct. & hiring citation ‘prestige’ Ph. D completion rates

MD-RAM: Example 1 q. Publications in international jrnls; q. Individual q. Hiring/promotion q. Productivity & impact q. Actual citation impact q. Ph. D date, place, supervisor; q. Invitations for conferences

Multi‐dimensional Research Assessment Matrix (Part) Unit of Purpose assessment Individual Allocate resources Research Improve performance group Department Monitor research programs Institution Research field Increase regional engagement Promotion, hiring Output dimensions Research productivity Quality, scholarly impact Innovation and social benefit Bibliometric Other indicators Publications Peer review Sustainability & Scale Internat. co- External authorship research income citation Ph. D com‘prestige’ pletion rates Research infrastruct. Journal citation impact Actual citation impact Patents, licences, spin offs Invitations for conferences

MD-RAM: Example 2 q. Publications in international jrnls; q. Research group q. Monitoring q(Trend in) actual citation impact research program q. Scientific impact q. Collaborations q. Topicality

Research assessment methodologies: Important considerations • Methodology must be fit-for-purpose • What is the primary “problem” to be solved? • Be aware of unintended effects • Change a methodology every 5 -10 years • What is an acceptable “error rate”? • Wrong in individual cases benificiary for the system as a whole

Contents 1 Boundaries of the playing field and rules of the game 2 The multi-dimensional research assessment matrix 3 Towards more sophisticated indicators: combining big data sets

Journal full text data Journal articles + citations Journal usage data Unit of assessment Books Conference Procs Newspapers Patents Trade jrnls Social media

(a) Downloads vs. Citations What do full article downloads measure?

Authors vs. readers ? Authors Readers

Hypothesis on degree of correlation between downloads and citations Authors Readers Strong Weak

Usage vs. citations per main field Scientific PSYCHOL ? Societal

(b) Patent citations to journal articles: The technological impact of research

The Technological Impact of Library Science Research: A Patent Analysis [Halevi et al, 2012] PATENTS (Total. Patent) Citations by patent examiners and inventors 42 LIBRARY SCIENCE JOURNALS (Scopus)

Cited articles: keywords in titles Citing patents: keywords in titles The articles feature information retrieval and indexing, information and documents management systems which pertain to electronic and digital libraries development The patents focus on electronic information administration, navigation, and products and services management in commercial systems.

(c) International scientific migration

International migration vs. co-authorship Relationship Definition International co-authorship Authors from institutions located in different countries jointly publish a paper International migration Comment Country relates to where authors work, NOT A scientific author moves from to their one country to another nationality

Map of countries with Ratio migration/collaboration > 1. 2

TO COUNTRY PAK PRT IND NLD USA NLD CHN USA BRA MYS No. Co. No. FROM authored Migrating Country papers authors IND Language 276 similarity 118 BRA 1, 971 423 drives migration PAK 276 it 96 stronger than IRN 492 80 drives co-authorship IND 12, 013 3, 307 PAK 145 21 TWN 3, 979 1, 048 Political tensions IRN 3, 039 affect migration less 780 PRT than they 1, 971 affect co- 352 NGA 31 authorship 122 Ratio % Migration / % Coauthorship 3. 6 3. 4 3. 3 3. 1 2. 8 2. 6 2. 4 2. 1

(d) Citation context analysis Combining citation data from Scopus with full text article data from Science. Direct

The use of contextual citations analysis to disclose thematic and conceptual flow of cross- disciplinary research: the case of the Journal of Informetics 2007 (Gali Halevi et al. , to be published, 2012)

Emerging sectional themes (OUTSIDE DISCIPLINE) INTRODUCTION CONCLUSIONS FINDINGS & DISCUSSION The themes sequence in the word clouds below might suggest that the individual output evaluation done by structured peer review leads to an acknowledgment of the importance and evolution of networks rather than individuals

(e) Book Citation Index Citation flows between books and journals

A Scholarly Book Citation Index: Approaches a) Add selected book series b) Add books from selected publishers c) Add selected individual book titles

Thank you for your attention!

Elsevier Bibliometric Research Program: www. ebrp. elsevier. com