IF Only Pushing the Limits of Impact Factor

IF Only: Pushing the Limits of Impact Factor Rebecca Kennison School of Social Work’s Faculty Development Seminar April 27, 2009 CDRS. COLUMBIA. EDU

Brief Background • 1962: Institute for Scientific Information (now Thomson Scientific, a division of Thomson Reuters) founded by Eugene Garfield • Impact factor invented as way for librarians to easily compare journals (via database for sale) • Journal inclusion based on Bradford’s Law – Index covers 7500 journals of the more than 110, 000 being published CDRS. COLUMBIA. EDU

Calculation A = the number of times articles published in indexed journals in 2006 -2007 that were cited in indexed journals during 2008 B = the number of “citable items” (usually articles, reviews, proceedings or notes, but not editorials or letters to the editor) published in 2006 -2007 2008 impact factor = A/B Note: The 2008 impact factor will actually be published in Summer 2009, because it cannot be calculated until all of the 2007 publications had been received by Thomson Scientific, which can take months. CDRS. COLUMBIA. EDU

Advantages • Seen as objective measure • Highly ranked journals cited more often • Shortcut for determining quality of research Researchers prioritize their reading Librarians easily compare journals within disciplines Authors decide where to submit papers first Tenure/promotion committee has no need to understand work – Deans and provosts measure output quality of faculty – Editors and funding agencies determine quality of researchers submitting their work – – CDRS. COLUMBIA. EDU

Major Criticisms • Focus on publishing in “high impact” journals influences research direction (e. g. , “trendy”) and publishing options (e. g. , general rather than specialty journals) • Not all journals are included, so not reflective of entire publishing landscape – Thomson privileges American journals and those published in English, excluding much of international publication world, both for journals indexed and for citations appearing in those journals – List of journals indexed varies from year to year • Journal editors manipulate the system (e. g. , reviews, self-citations, direct negotiations with Thomson) – and feel forced to do so • Thomson’s data are often faulty and irreproducible (and problem is getting worse) and criteria for what “counts” changes from year to year • IF shows only the average of citations across articles (many never cited at all) within a journal and says nothing about quality or citability of any given article • “Negative” citations count in IF every bit as much as “positive” ones • IF only looks at short term (but not immediate) impact, not long term • IF cannot be compared across fields (i. e. , high IF score in one field may look low in comparison with another field) and focuses solely on journal publications, so is useless as institutional measure CDRS. COLUMBIA. EDU

IFS in Social Work • IFS in social work journals = 0. 06 to 1. 62 [in 2006] • Often best papers in social work go elsewhere because of pressure to publish in “high impact” journals (Information from Jenson & Fortune, 2008) CDRS. COLUMBIA. EDU

Thomson Reuters clearly has created a great business for themselves … … but is it good for us? CDRS. COLUMBIA. EDU

“We are slaves to the impact factor. ” – Marty Frank, Executive Director, American Physiological Society CDRS. COLUMBIA. EDU

“I don’t think two years is an adequate representation of anything. I think if you went and found the impact factor of Einstein’s theory of relativity, you would get zero. And to me that’s an important paper, so I just don’t agree with that. ” – Berkeley researcher in study on scholarly communication conducted in 2006 CDRS. COLUMBIA. EDU

“Academia is being held ransom by these citations. ” – Alan Nevill, University of Wolverhampton CDRS. COLUMBIA. EDU

“[EASE] recommends that journal impact factors are used only … for measuring and comparing the influence of entire journals, but not for the assessment of single papers, and certainly not for the assessment of researchers or research programmes …. ” – European Association of Science Editors, 2008 CDRS. COLUMBIA. EDU

“Any selection or promotion committee that asks you for impact factors is probably a second-rate organization. A good place will want to know about the quality of what you have written, not where you published it — and will be aware that the two things are uncorrelated. ” – David Colquhoun, University College London CDRS. COLUMBIA. EDU

“[A]gencies often wish to bypass the work involved in obtaining citation counts for individual articles and authors. . Thus, the [journal impact factor] is used to estimate the expected count of individual papers, which is rather dubious…. ” – Eugene Garfield, inventor of the Impact Factor CDRS. COLUMBIA. EDU

“The practice of using the IFS [impact factor score] as a proxy indicator of article impact merits significant concern as well as further empirical investigation. ” – Gary Holden, New York University Silver School of Social Work CDRS. COLUMBIA. EDU

Impact of IF • Criteria for tenure and promotion determined by scholars outside institution, not by peers • Best journal for paper often not best ranked, but pressure is to publish in only a few journals • Focus on IF reinforces traditional reward system CDRS. COLUMBIA. EDU

Alternatives • H-Index (a. k. a. Hirsh number) • Page. Rank algorithm (used by Eigenfactor) • MESUR (MEtrics from Scholarly Usage of Resources) • Citation feature from Google Scholar CDRS. COLUMBIA. EDU

CDRS. COLUMBIA. EDU

Scholar Factor • Suggested by Phil Bourne, UCSD • Goal: Create reward system that is fair reflection of entire academic contribution CDRS. COLUMBIA. EDU

Feds Reviews Curation Research [Grants] Journal Article Publishers Poster Session Conference Paper Community Service/Data Societies Blogs CDRS. COLUMBIA. EDU

Scholar Factor Formula SF = (H Factor) + (Grant/Manuscript Review Factor/20) + (Annotations/Software/Datasets Factor/5) + (Web Factor/50) CDRS. COLUMBIA. EDU

H Factor • H factor = Number of papers cited more than H times • H factor of 20 indicates that an author has 20 papers cited more than 20 times • Derived from Google Scholar data (assuming an author could be uniquely identified) • Much broader and deeper literature indexed CDRS. COLUMBIA. EDU

Grant/Manuscript Review Factor • Accumulative number of authenticated grant and paper reviews • 20 reviews increases SF by 1 CDRS. COLUMBIA. EDU

Annotations/Software/Datas ets Factor • Accumulative number of authenticated entries made in a public database or software entries added to an open-access archive • If n researchers involved in making the entry, then 1/n of an entry per researcher • 5 entries increase SF by 1 • Amount of work performed also needs to be weighed, but you get the idea CDRS. COLUMBIA. EDU

Web Factor • Number of authenticated blog posts, wiki postings, etc. , made that show x or more links to them (a measure of their value), where x is to be determined • 50 entries increases SF by 1 CDRS. COLUMBIA. EDU

Impact of SF • Criteria for tenure and promotion determined by broad range of academic contribution • Replaces primacy of journal impact factor with article (or other formal publication) citation index • Allows reward system to change An idea perhaps ahead of its time, but one to consider CDRS. COLUMBIA. EDU

Questions? CDRS. COLUMBIA. EDU