BIBLIOMETRICS USE AND LIMITATIONS Wolfgang Glnzel KU Leuven

BIBLIOMETRICS – USE AND LIMITATIONS Wolfgang Glänzel KU Leuven, Belgium ISPR, HAS, Hungary Wolfgang Glänzel Bibliometrics. Use and limitations

STRUCTURE OF THE PRESENTATION 1. 2. 3. 4. 5. 6. Introduction Scope and structure of bibliometrics Data sources of bibliometric research “Perspective shift” in use of bibliometrics Bibliometrics as a tool in research evaluation Bibliometric research and services at the Steunpunt O&O Statistieken Wolfgang Glänzel Bibliometrics. Use and limitations 2

1. INTRODUCTION Wolfgang Glänzel Bibliometrics. Use and limitations 3

1. Introduction What is bibliometrics? The terms bibliometrics and scientometrics were introduced by Pritchard and Nalimov & Mulchenko in 1969, and are nowadays used almost synonymously. Pritchard “bibliometrics is the application of mathematical and statistical methods to books and other media of communication”. Nalimov & Mulchenko Scientometrics is “the application of those quantitative methods which are dealing with the analysis of science viewed as an information process”. Wolfgang Glänzel Bibliometrics. Use and limitations 4

1. Introduction Gorkova (1988) Informetrics stands for a more general subfield of information science dealing with mathematicalstatistical analysis of communication processes in science. A quite broad definition of the scope of bibliometrics given by Glänzel & Schoepflin (1994) includes “all quantitative aspects and models of science communication, storage, dissemination and retrieval of scientific information”. Wolfgang Glänzel Bibliometrics. Use and limitations 5

1. Introduction These notions also integrate many presently existing orientations such as applications to science policy, library science, technometrics, Web analyses and information retrieval. Prominent methodological issues are related to • • • subject delineation subject evolution and structure publication activity/scientific productivity scientific collaboration, co-authorship networks citation impact and citation networks bibliographic coupling, co-citation analysis computational linguistics; text mining, co-word analysis visualisation techniques data matching science-technology linkage Wolfgang Glänzel Bibliometrics. Use and limitations 6

1. Introduction Common misbeliefs on bibliometrics • Main task of bibliometrics should be the expeditious issuing of “prompt” and “comprehensible” indicators for science policy and research management. • Bibliometrics is only an auxiliary tool, research on methodology is unnecessary. Instead, bibliometricians should elaborate guidelines explaining the use of their indicators. • Bibliometrics might be reduced to simple counting activities in order to replace/supplement qualitative assessment by quantitative indicators and to set publication output off against funding. Wolfgang Glänzel Bibliometrics. Use and limitations 7

1. Introduction Facts about bibliometrics Bibliometrics is a powerful, multifaceted endeavour encompassing sub-areas such as - Structural scientometrics came up with results like the re-mapping of the epistemological structure of science. - Dynamic scientometrics constructed sophisticated models of scientific growth, obsolescence, citation processes, etc. - Evaluative scientometrics developed arrays of indicators to be used to characterise research performance at different levels of aggregation and come up with benchmarking solutions. Wolfgang Glänzel Bibliometrics. Use and limitations 8

2. SCOPE AND STRUCTURE OF BIBLIOMETRICS Wolfgang Glänzel Bibliometrics. Use and limitations 9

2. Structure of Bibliometrics Three “components” of present-day bibliometrics according to its three main target-groups Bibliometrics for bibliometricians The domain of bibliometric “basic research”. Bibliometrics for scientific disciplines A large but also the most diverse interest-group. Due to the scientists’ primary scientific orientation, their interests are strongly related to their speciality. Bibliometrics for science policy and research management The most important field of application. Here the assessment and comparative analysis of research performance are in the foreground. Wolfgang Glänzel Bibliometrics. Use and limitations 10

2. Structure of Bibliometrics Links of bibliometrics with related research fields and application services Science policy Research management Scientific information Librarianship Services for Research in Economics Sociology of science applied Library and Information Science Scientometrics basic Wolfgang Glänzel Life sciences Informetrics Webometrics History of science Mathematics/Physics Bibliometrics. Use and limitations 11

3. DATA SOURCES OF BIBLIOMETRIC RESEARCH Wolfgang Glänzel Bibliometrics. Use and limitations 12

3. Sources of Bibliometrics Data sources of bibliometrics are bibliographies and bibliographic databases. Large scale analyses can only be based on bibliographic databases. Despite objections to journal coverage and data processing policy of the ISI, the databases of the Institute for Scientific Information (Thomson - ISI), above all, the Science Citation Index (Expanded) have become the most generally accepted data source. Recently several countries have developed their own national citation indexes which are also used (as supplementary) data sources. Wolfgang Glänzel Bibliometrics. Use and limitations 13

3. Sources of Bibliometrics Bibliographic databases are generally designed for the retrieval of information. However, their versatility is often limited in bibliometric use. Originally, bibliometrics was developed for studying communication in basic research. Present-day tasks: • How to extend bibliometrics towards application to engineering, social sciences and humanities? • How to measure and evaluate aspects of technology linkage? Since non-periodicals are important conveyors of information in engineering, social sciences and humanities, journal based data-sources are accepted by scientists only with certain reservations. Wolfgang Glänzel Bibliometrics. Use and limitations 14

4. “PERSPECTIVE SHIFT” IN USE OF BIBLIOMETRICS Wolfgang Glänzel Bibliometrics. Use and limitations 15

4. Perspective Shift Bibliometrics deals with the quantitative analysis of the (social) network of scientific communication. From the historical perspective, information scientists and bibliometricians proceeded from the same approach. First citation analyses were designed to aid the decision which periodicals should best be purchased by small libraries. Many tools and indicators developed by the ISI were designed for use in scientific information, information retrieval and libraries. Þ The application to science policy has brought a new perspective, and resulted in re-interpretation of bibliometric conceptions. Wolfgang Glänzel Bibliometrics. Use and limitations 16

4. Perspective Shift Example: The notion of citations in information science, bibliometrics and science policy Citations became a widely used measure of the impact of scientific publications, but the notion of what citations actually express sheds light on different aspects. These different views might results in conflicts caused by reinterpretation and “perspective shift”. Wolfgang Glänzel Bibliometrics. Use and limitations 17

4. Perspective Shift Some examples: According to Merton, citations are intimately connected with the reward system of science. Cozzens: “Citation is only secondarily a reward system. Primarily, it is rhetorical-part of persuasively arguing for the knowledge claims of the citing document. ” Cronin: Citations are "frozen footprints in the landscape of scholarly achievement … which bear witness to the passage of ideas“. Glänzel and Schoepflin: Citations give “a formalised account of the information use and can be taken as a strong indicator of reception at this level. ” Wolfgang Glänzel Bibliometrics. Use and limitations 18

4. Perspective Shift The process of re-interpreting the notion of citation and its consequences interpretation citation Bibliometrics/ Information science Information use repercussion re-interpretation (possible distortion of citation behaviour) Research evaluation/ Science policy Wolfgang Glänzel uncitedness: unused information frequent cite: good reception self-cite: part of scient. communication Reward system (quality measure) uncitedness: low quality frequent cite: high quality self-cite: distortion of impact Bibliometrics. Use and limitations 19

5. BIBLIOMETRICS AS A TOOL IN RESEARCH EVALUATION Wolfgang Glänzel Bibliometrics. Use and limitations 20

5. Research Evaluation Present-day use of bibliometrics The evaluation of scientific research performance is the most important application of present-days bibliometrics. • Growing, often controversial, policy interest to use bibliometric techniques as output measures of research productivity and efficiency. • Thus bibliometrics evolved to a standard tool in science policy and research management. • Bibliometric components are used, among others, in national science reports, bibliometric indicators appear in the daily press, benchmarking results are used for research management and for allocation of research funding. Wolfgang Glänzel Bibliometrics. Use and limitations 21

5. Research Evaluation What can bibliometrics not be responsible for? • Bibliometrics can be used to develop and provide tools to be applied to research evaluation, but is not designed to evaluate research results. • Bibliometrics does not aim at replacing qualitative methods by quantitative approaches. • Consequently, bibliometrics is not designed to correct or even substitute peer reviews or evaluation by experts but qualitative and quantitative methods in science studies should complement each other. Wolfgang Glänzel Bibliometrics. Use and limitations 22

5. Research Evaluation Distorted behaviour based on policy use and misuse of bibliometric data One issue concerns the changes in the publication, citation and collaboration behaviour of scientists (both positive and negative) that the consistent policy use of bibliometric indicators might potentially induce. Uninformed use and misuse are not always beyond the responsibility of bibliometricians. Unfortunately, bibliometricians do not always resist the temptation to follow popular, even populist, trends in order to meet the expectations of the customers. Any kind of uninformed use or misuse of bibliometric results involves the danger of bringing bibliometric research itself into disrepute. Wolfgang Glänzel Bibliometrics. Use and limitations 23

5. Research Evaluation The problem of inappropriate use ranges from uninformed use, over selecting and collecting ‘most advantageous’ indicators to the obvious and deliberate misuse of data. Uninformed use • incorrect presentation, interpretation of bibliometric indicators or their use in an inappropriate context caused by insufficient knowledge of methodology, background and data sources Misuse • intentionally incorrect presentation, interpretation of bibliometric indicators or their deliberate use in inappropriate context • tendentious application of biases or tendentious choice of (incompatible) indicators Wolfgang Glänzel Bibliometrics. Use and limitations 24

5. Research Evaluation Þ But even correct use might have undesired consequences • Author self-citations are shown in an unfavourable light. Authors might thus feel urged avoiding self-citations – a clear intervention into the mechanism of scientific communication. • Indirect repercussions can be observed when bibliometric tools are used in decision-making and the scientific community recognises the feedback in terms of their funding. • Butler (2004) showed on the example of Australia what might happen when funding is linked to publication counts. She found that this has stimulated an increased publication activity in the lower-impact journals. Wolfgang Glänzel Bibliometrics. Use and limitations 25

5. Research Evaluation Schematic visualisation of the feedback of policy use of bibliometrics on the scientific community Wolfgang Glänzel Bibliometrics. Use and limitations 26

5. Research Evaluation Possible positive effects • Scientists might recognise that scientific collaboration and publishing in high-impact or even top journals pays off. Also their publication activity might be stimulated. Possible negative effects • Exaggerated collaboration, even trends towards hyperauthorship, inflating publication output by splitting up publications to sequences, inflating citation impact by selfcitations and forming citation cliques, etc. • Trend towards replacing quality and recognition by visibility at any price or towards preferring journals as publication channels in social sciences and humanities might be among these effects. • A certain “champions league” mentality is spreading among scientists (“Shanghai ranking”, H-index, “IF Filter”). Wolfgang Glänzel Bibliometrics. Use and limitations 27

6. BIBLIOMETRIC RESEARCH AND SERVICES AT THE STEUNPUNT O&O STATISTIEKEN Wolfgang Glänzel Bibliometrics. Use and limitations 28

6. The Steunpunt O&O Statistieken (Research Centre for R&D Statistics – SOOS) was created in January 2002 after open calls for proposals. SOOS is a unique and dedicated research center to support its science and innovation policy in Flanders. This center has to develop an appropriate system of quantitative indicators to quantify R&D efforts at Flemish universities, research institutes and industry that can be used by policy makers to support the development of appropriate regional science and innovation policy for Flanders. Wolfgang Glänzel Bibliometrics. Use and limitations 29

6. The Steunpunt O&O Statistieken ACTIVITY OVERVIEW The activities of SOOS can be classified into four categories: 1. Developing an appropriate IT-infrastructure to support S&T indicator development; 2. Providing the Flemish government with appropriate S&T indicators on which to base its science and technology policy resource allocation decisions; 3. Executing specific studies on science and technology domains or science policy relevant topics for the Flemish government; 4. Doing original research in the field of S&T studies. Wolfgang Glänzel Bibliometrics. Use and limitations 30

6. The Steunpunt O&O Statistieken LONG- AND MEDIUM-TERM SERVICES FOR THE FLEMISH GOVERNMENT I. The BOF-key One of the major funding mechanisms for basic science in Flanders is the BOF-mechanism. The Bijzonder Onderzoeksfonds is a research fund of the Flemish government that is distributed to the Flemish universities on the basis of a specific allocation model. This allocation key comprises both input and output measures on the research performance of the universities involved. Part of the key is based on publication and citation data as derived from Wo. S by SOOS. Wolfgang Glänzel Bibliometrics. Use and limitations 31

6. The Steunpunt O&O Statistieken BOF-funding for Flanders amounts to 100 million EUR annually. This amount is distributed in basic science funding amongst the six Flemish universities (LUC/UHasselt, KUB, KUL, VUB, UA and UGent). II. The IOF-key is a distribution key to determine the annual allocation of 12 million EUR (to be augmented in the year 2006) amongst the six Flemish universities in order to support applied research. The IOF-key is partially based on patent statistics derived from the USPTO, EPO and PCT databases. SOOS has been instrumental both in the development of and the data delivery for those distribution instruments. Wolfgang Glänzel Bibliometrics. Use and limitations 32

6. The Steunpunt O&O Statistieken III. The Flemish Indicatorbook on Science, Technology and Innovation appears biannually, and does not only support government questions on R&Dproficiency in Flanders, but also aims to supply adequate and upto-date information on the European R&D-potential of the Flanders region. Two editions of the Indicatorbook have been published by SOOS so far: 2003 and 2005. Wolfgang Glänzel Bibliometrics. Use and limitations 33

6. The Steunpunt O&O Statistieken IV. Domain Studies (here bibliometrics only) Aim of these studies is to conduct and to provide analyses of Flemish research and technology activity in the international context in highly important areas. Expected outcomes are relevant information about performance and competitiveness in Flanders. Three domain studies have been prepared and published by Soo. S: Nanoscience and –technology, Biotechnology and molecular biology (both in 2003) and Stem-cell research (2004). The studies are based on both publication and patent data extracted from the Wo. S, the USPTO and EPO databases. Wolfgang Glänzel Bibliometrics. Use and limitations 34

6. The Steunpunt O&O Statistieken BIBLIOMETRIC RESEARCH PROJECTS AT SOOS Methodological and applied research is closely linked to the service tasks, for instance, in the context of methodological validation, of improvement of the efficiency of indicators or of the subject classification. ‘Basic research’ at the Steunpunt is structured in a systematic way to support policy-relevant research and services and to contribute to the advancement of the field of bibliometrics. Similarly to the services, research is done at the national/regional, institutional and research-group level. Wolfgang Glänzel Bibliometrics. Use and limitations 35

5. The Steunpunt O&O Statistieken 1. Development of a hierarchic subject classification scheme for evaluation purposes 2. The role of author self-citations in scientific communication 3. The myth of delayed recognition 4. Analysis of possible extension of bibliometric methods to social science and humanities 5. Studying the network of science communication 6. “Indicator research” (improving journal and subject impact measures and science indicators) Wolfgang Glänzel Bibliometrics. Use and limitations 36

6. The Steunpunt O&O Statistieken 7. Bibliometrics beyond the Wo. S (e. g. , proceedings literature) 8. Bibliometric approach to social sciences and humanities 9. Combining text-mining and bibliometrics 10. Mapping academic mobility and scientific collaboration in Europe 11. Research group dynamics and their impact on performance and visibility 12. Bisociation analysis of scientific communication Wolfgang Glänzel Bibliometrics. Use and limitations 37