Granting and Evaluation of Research in Humanities and

Granting and Evaluation of Research in Humanities and Social Sciences at the European Level – and especially at ERC and ESF Alain PEYRAUBE CNRS – EHESS Membre of the ERC Scientific Council │1

European Reports on Evaluation • AUBR Report « Assessing Europe’s University-Based Research » by Mackiewicz (Summer 2009, 130 pages). Detailed lists on the evaluation system in Australia, Belgium, Finland, France, Germany, Hungary, Ireland, Italy, The Netherlands, Norway, Spain, Sweden, United Kingdom Suède, on Webometrics, on ARWU (Shanghai rankings, on THES, on PRSP (Taiwan), on CWTS (Leiden) • SPRU report « Towards a Bibliometric Database for the Social Sciences and Humanities – A European Scoping Project » (ANR, DFG, ESRC, AHRC, NWO, ESF) – January 15, 2010 │2

European Reports on Evaluation (2) • Article « Journal Base – A comparative international study of scientific journals in the social sciences and the humanities » , M. Dassa, C. Kosmopoulos, D. Pumain Cybergeo, January 2010 • Article « Research Quality Assessment and the Metrication of Social Sciences » (Myriam E. David, European Political Science 2008. 7, 13 pages) • Bibliométrie et évaluation en SHS (Ghislaine Filliatreau, Revue d’histoire moderne et contemporaine 55 -4, 2008, 4 pages) │3

FP 7 (EC) 7 th Framework Programme (FP 7) 50. 52 Billion Euros, 4 Specific Programs “Family” of FP 7 Specific Programmes Co-operation 33 B Euros (8 priorities) Ideas 7. 5 B Euros [ERC] Capacities 4 B Euros (Infrastructures, etc. ) People 5 B Euros (Marie Curie fellowships) Ideas: complementary to other FP 7 support policy vs. science-driven, bottom-up vs. targeted research │4

The Scientific Council Members & Role 22 most respected researchers reflecting the full scope of European research and scholarship proposed by an independent identification committee appointed by the Commission (for 4 years, renewable once) Role: Establishes overall scientific strategy establishes annual work programmes (incl. calls for proposals, evaluation criteria); defines peer review methodology; ensures selection and accreditation of experts Controls quality of operations and management Ensures communication with the scientific community │5

The Executive Agency Executes annual work programme as established by the Scientific Council Implements calls for proposals and provides information and support to applicants Organises peer review evaluation Establishes and manages grant agreements Administers financial aspects and follow-up of grant agreements │6

Strategic Aims Overview • ERC : first pan-European funding agency for frontier research • Funding directed to individual teams and projects selected solely on the criteria of excellence • Operates according to the principles of scientific excellence, autonomy, efficiency and transparency Autonomous scientific governance Simple, user-friendly delivery │7

Strategic Aims (2) • Reinforce excellence, dynamism and creativity in European research • Improve the attractiveness of Europe for the best researchers from both European and third countries, as well as for industrial research investment • ERC grants can complement the efforts of national funding bodies and host institutions to build or reinforce excellence across Europe

ERC Grants: Open to individual teams anywhere in the world • ERC is the only component of the FP 7 -family to widely open its programs and schemes to individual teams anywhere in the world, • under only one condition, i. e. that funded projects will have to be located in EU or its associated countries • Thus, PIs and team members can be of any nationality, working in almost any country • Team may consist solely of the PI (mathematics, humanities) │9

Two types of Grants • ERC Starting Grants, supporting excellent early stage independent investigators (2 -9 years since completion of Ph. D) – 2007 and 2009. Starting Grant divided into two categories from 2010 on : Starters (Ph. D : +2, - 7) and Consolidators (Ph. D : + 7, - 12). Grants : up to 1. 5 M euros for a period of 5 years (+ additional Euros 500, 000 for eligible ‘start-up’ costs) • ERC Advanced Grants, supporting investigators at all subsequent stages – 2008 and 2010. Grants up to 2. 5 M euros (+ additional Euros 1 M for eligible ‘start-up’ costs) │ 10

ERC Grant schemes Budget Allocation For operational reasons the Sc. C agreed on 3 main research domains: ─ Domain 1: Mathematics, Physical Sciences, Information and Communication, Engineering Sciences, Universe and Earth Sciences (PE) ─ Domain 2: Life Sciences (LS) ─ Domain 3: Social Sciences and Humanities (SH) Pre-allocation of call budget for ERC Grants per domain currently as follows: 45% PE - 40% LS - 15% SH 2009 : 39% PE – 34% LS – 14% SH + 13% Interdisc. 2011 : 40% PE – 35 % LS – 15% SH + 10% ID │ 11

ERC Grant schemes Panel Structure Panels have one Panel Chair and 10 -12 Panel Members (now between 12 and 15), all selected by the Scientific Council members 20 Panels for the 1 st Call : 8 PE, 7 LS, 5 SH 25 Panels now: 10 PE, 9 LS, 6 SH Panel Chair oversees evaluation process for the proposals assigned to his/her Panel in collaboration with ERC staff Panel Chair gives high level stamp of credibility and visibility to the whole evaluation process │ 12

Panels composition • Panel Members and Chairs Make decisions and document the decisions • Panel chairs Ensure quality of the evaluation process Chair panel meetings (steps 1 & 2) Participate in panel chair meetings • ERC Panel Teams (2 Scientific Officers) Ensure that rules are respected Ensure comparable process is followed in all panels Transition: composed of ERC staff and RTD colleagues │ 13

Evaluation criteria 1. Principal Investigator CV of 2 pages, and Early Achievements track-record of 2 pages for Starting Grant; CV of 2 pages, and 10 -year track record of 2 pages for Advanced Grant 2. Research Project Extended Synopsis of 5 pages, Full scientific proposal of 15 pages maximum for both Starting Grant and Advanced Grant │ 14

Evaluation criteria (2) 3. Research Environment Step 1 of evaluation criteria 1 and 2 Starting Grant PIs invited for interview for the 2 nd step of evaluation Step 2 of evaluation all criteria Referees and panels evaluate and score criteria 1 and 2 numerically, which will result in the ranking of the proposals Criteria 3 is considered as "pass/fail" and commented upon but not scored. │ 15

Evaluation criteria (3) • Each proposal marked on a scale of 1 to 4 for each of the 2 evaluation criteria: 4 : Outstanding, 3 : Excellent, 2 : Very Good, 1 : Non -competitive • A proposal marked below the quality threshold of 2 on either of the two headings after the 1 st stage of evaluation not retained for the second step • Proposal marked below the quality threshold of 2 at the end of the 2 nd step not funded

Co-investigators • • • Exceptional For interdisciplinary proposals only PI can include 1 or more co-investigators (co-I) Co-I submitted to same resubmission rule as PI Higher budget (up to 3, 5 M€ for Advanced Grant) Scientific added value of including co-investigators to be assessed by panel │ 17

New Panel structure and description Social Sciences and Humanities (18 March 2008) • SH 1 Individuals, institutions and markets: economics, finance and management • SH 2 Institutions, values and beliefs and behaviour: sociology, social anthropology, political science, law, communication, social studies of science and technology • SH 3 Environment and society: environmental studies, demography, social geography, urban and regional studies • SH 4 The Human Mind and its complexity: cognition, psychology, linguistics, philosophy and education • SH 5 Cultures and cultural production: literature, visual and performing arts, music, cultural and comparative studies • SH 6 The study of the human past: archaeology, history and memory │ 18

ERC-2007 -St. G Submission (stage 1): Massive response! Domain Number of proposals % of budget pre-allocated* Life Sciences 3, 396 37, 0 40 Physical Sciences & Engineering 4, 408 48, 1 45 Social Sciences & Humanities 1, 363 14, 9 15 Total 9, 167 *Indicative budget established by Sc. C (Work programme 2007) │ 19

Country of host institution Number of proposals by domain and country of host institution (head quarters) 34 countries │ 20

ERC-2007 -St. G Granted projects Domain Number of proposals % of budget pre-allocated* Life Sciences 108 36 40 Physical Sciences & Engineering 137 45 45 Social Sciences & Humanities 58 18 15 Total 303 3. 3 *Indicative budget (Scientific Council, ERC work programme 2007) │ 21

Distribution of the 303 granted • 303 proposals granted (3. 30%): 137 PE (45%), 108 LS (36%), 58 SH (18%) • Distribution of the 303 by host institution: 19% UK, 13% FR, 11% DE, 8. 5% NL, 8. 2% IT, 8% SP, 8% IL, 1 proposal from CZ • The 58 SH: 18 UK, 10 NL, 6 FR, 4 DE, 4 ES, 4 IT, … │ 22

ERC Advanced Grant – 1 (2008) (ERC Advanced Investigator Researcher Grant) Targeting researchers who have already established their independence as team leaders and are exceptional leaders in terms of significance of their research achievements (in the last 10 years) up to 5 years, up to € 3, 5 million per grant ~2000 Starting Grants over 7 years of FP 7 (2007 -2013) │ 23

ERC Advanced Grant 1 Evaluation Criteria: Excellence is the sole award criterion 1. Principal Investigator - Quality of research output/track-record ( 10 -year track-record) Intellectual capacity and creativity ( CV + leadership profile) 1. Research Project - Ground-breaking nature of the research Potential impact Methodology 1. Research Environment (assessed in Step 2 only) - Contribution of the research environment to the project Contribution of the project to research environment Participation of other legal entities (if clear/ substantial added value) │ 24

Distribution / Domain - Applications Domain Number of proposals % of budget pre-allocated* Life Sciences 766 35 34 Physical Sciences & Engineering 997 46 39 Social Sciences & Humanities 404 18 14 Total 2167 │ 25

Distribution / Domain - Granted Domain Number of Proposals % of proposals % of budget pre-allocated* Life Sciences 84 33. 6 34 Physical Sciences & Engineering 117 46. 8 39 Social Sciences & Humanities 49 19. 6 14 Total 250 + 29 INTERDISC. (11 PE, 11 LS, 7 SH) = 13% of the budget = 279 (12. 9%) │ 26

Advanced Grant 1 – Distr. Ibution / Host Institution of the granted projects • 63 UK, 37 FR, 28 CH, 28 DE, 20 IT, 1 NL, 15 IL, 13 SE, 13 ES, … 2 CZ • PE Domain, out of the 117: 20 UK, 15 FR, 12 CH, 12 IT, 11 DE, 10 NL, 9 IL, 9 SE … • LS Domain, out of the 84: 18 UK, 11 DE, 11 FR, 9 CH, 5 ES, 5 NL, 4 IT, 4 IL … • SH Domain, out of the 49: 12 UK, 9 FR, 5 DE, 5 NL, 5 ES, 4 IT, 3 CH … │ 27

ERC Starting Grant 2 - 2009 Submitted and mainlist proposals by domain Submitted proposals Mainlist Life Sciences 927 (37%) 69 (31. 5%) Physical Sciences & Engineering 1112 (44. 4%) 93 (42. 5%) Social Sciences & Humanities 464 (18. 6%) 41 (18. 7%) Interdisciplinary Domain 16 (7. 3%) ∑ = 2503 ∑ = 219 │ 28

ERC Starting Grant 2009 The 238 granted proposals (9. 50%) • Distribution by host institution: 43 UK, 31 FR, 28 DE, 18 ES, 17 CH, 17 NL, 15 BE, 15 IT, 15 NL, 14 BE, 14 IL, … • In the SH domain : 12 UK, 6 DE, 6 NL, 5 BE, 5 ES, 5 IT, 2 DK, 2 FR, … │ 29

Advanced Grant 2 - 2009 Number of submitted and short-listed proposals Submitted proposals Short List Life Sciences 512 198 Physical Sciences & Engineering 736 247 Social Sciences & Humanities 335 109 Interdisciplinary research ? ∑ = 1583 ∑ = 554 │ 30

Main Lists : 236 (14. 9%) • • • PE : 99 proposals SH : 42 proposals LS : 81 proposals ID : 14 proposals (6 PE, 8 LS, 0 SH) Host Institution : 58 UK, 34 FR, 31 DE, 29 CH, 16 NL, 15 IT, 15 NL, 12 SE, 11 IL, 10 ES, … │ 31

Distribution by domain • 105 PE : 23 UK, 15 CH, 15 DE, 14 FR, 8 SE, 6 IL, 6 IT, 5 ES, 4 NL, … • 89 LS : 21 UK, 15 FR, 13 CH, 12 DE, 6 NL, 5 IT, 4 ES, 3 IL, 3 SE, … • 42 SH : 14 UK, 5 FR, 5 NL, 4 DE, 4 IT, 2 IL, 2 NO, 1 AT, 1 CH, 1 ES, 1 IE, 1 HU, 1 SE │ 32

Research Assessment in the Humanities and the ESF-ERIH project │ 33

ESF-SCH workshop in 2001 We can no longer allow ourselves the indulgence of attributing the lack of appropriate tools to a peculiarity of the Humanities and Social Sciences Specialists in the Humanities should take into consideration its special characteristics and develop the corresponding tools It is essential for the future of the Humanities to furnish itself with reliable reference tools, following the example of other scientific disciplines. │ 34

ESF-SCH workshop in 2001 (2) • AHCI (ISI-Thomson Scientific Web of Knowledge) not appropriate • Urgent need for a European Reference Index for the Humanities (ERIH) as an additional tool for research evaluation, and not the exclusive means • Request the ESF-SCH to go ahead and try to compile lists of reference journals (Stage 1) • Develop methodology for including other formats: monographs, book chapters, edited volumes, etc. (Stage 2) │ 35

ESF-SCH > ERIH Steering Committee Overall responsibility with the ESF Standing Committee for the Humanities (SCH) SCH nominates ERIH Steering Committee in Spring 2004: F. Kiefer (HU), A. Mustajoki (FI), A. Peyraube (FR, Chair), Gudrun Gersmann (DE), Marc Waelkens (BE), Michael Worton (UK). ERIH Steering Committee responsible for: Identification of the disciplinary structure Definition of methodology including the definition of categories of journals Approval of membership of Expert Panels (members suggested by MO’s, SCH, ERIH St. Comm) Validation of journal lists proposed by Expert Panels │ 36

ERIH: current disciplinary structure • • Archaeology Art and Art History Classical Studies Anthropology Gender Studies History Hist. & Phil. of Science Linguistics • • Literarature Musicology Oriental & African Studies Pedagogical & Educ. Research Philosophy Psychology Religious Studies & Theology Disciplines under considertation • Archives, Library & Museum Studies • Film, Media & Cultural Studies • Area Studies 37│ 37

Peer review: the basis of methodology • Peer review recognised as the only practicable method of evaluation in basic research (standard method used in evaluation of scientific production) nomination of panel members • ESF Mos approached – with guidelines – to provide lists of reference journals discipline by discipline │ 38

ERIH – Journal categories: A category (expected: 5 -20% of all titles): High-ranking, international level publications; Very strong reputation among researchers in the field; Regularly cited all over the world B category: Standard, international level publications; Good reputation among researchers in the field in different countries, occasionally cited all over the world │ 39

ERIH – Journal categories (2): • C category: • Important local / regional level publication; • Mainly local / regional readership, but occasionally cited outside the publishing country • Only European publications are considered in this group │ 40

Process of accreditation • Input: National panels / scientific communities provide lists of journals • Selection: Expert Panels define scope, analyse and assess input, produce lists • Consultation: MOs, subject associations (European level and some national), specialist research centres • Calibration/harmonisation: ERIH Steering Committee; • After approval by ESF-SCH, publication of ‘initial lists’ took place in 2007 • Open feedback process via on-line questionnaire for editors and publishers • ‘Revised lists’ to be published in 2010 │ 41

Some examples • Anthropology: 752 journals proposed by Mos 242 journals selected: 40 A (16. 5%), 112 B (46. 3%), 90 C (37. 2%) • Archaeology: 1312 (Mos) 425: 91 A (21. 4%), 170 B (40%), 164 C (38. 6%) • History: 1419 (Mos) 934: 135 A (14. 45%), 380 B (40. 7%), 419 C (44. 9%) • Linguistics: 1093 (Mos) 585: 96 A (16. 4%), 92 B (36. 5%), 275 C (47%) │ 42

Some examples (2) • Literature: 1453 (Mos) 976: 255 A (26. 1%), 481 B (49. 2%), 240 C (24. 6%) • Music and musicology: 300 (Mos) 166: 21 A (12. 6%), 86 B (51. 8%), 59 C (35. 5% • Philosophy: 658 (Mos) 305: 44 A (14. 4%), 130 B (42. 6%), 131 C (42. 9%) • Relig. Stud. &Theology: 745 (Mos) 371: 76 A (20. 5%), 203 B (54. 7%), 92 C (24. 8%) │ 43

ERIH: challenges and criticism • Misunderstandings about the character of the currently used A/B/C categories: ranking or assessment of audience, distribution and reach? • Misunderstandings around category C seen as ‘low quality’ when the idea is to identify quality European journals with, mostly linguistically, limited circulation; identification of quality national journals is the main innovation of ERIH • Following discussions in the research community, process of renaming ERIH categories is underway 44│ 44

More misunderstandings • Some research councils and research organisations have used ERIH as a tool for assessment of individual research production / productivity • ERIH “initial lists” were used when they were still under revision │ 45

New categorization of Journal categories • National Journals (former category C): European publications with a recognised scholarly significance among researchers in their respective research domains in a particular (mostly linguistically circumscribed) readership group in Europe; occasionally cited outside the publishing country, though their main target group is the domestic academic community • International Journals (former categories A and B): both European and non-European publications with an internationally recognised scholarly significance among researchers in their respective research domains, and which are regularly cited worldwide │ 46

New Journal categories (2) Differentiation between current categories A and B is based on a combination of two criteria: influence and scope: Category A • international publications with high visibility and a very strong reputation and influence among researchers in the various research domains in different countries Category B • international publications with significant visibility and a good reputation and influence in the various research domains in different countries. │ 47

Next steps • Reorganisation of the expert panels by using a panel rotation mechanism and inclusion of new experts • Expert Panel meetings revising ‘initial lists’ based on received feedback (adding in new journals, eliminate some journals, changing and revising categories) • Consolidation of the lists in 2009 (First update) > New lists (Revised lists) to be published in 2010 • Workplan for inclusion of other publications (monographs, edited books, proceedings, etc. ) │ 48

ERIH - contact www. esf. org/erih │ 49

Merci Thank you 谢谢 │ 50