Estonian experience in evaluating research proposals Madis Saluveer

Estonian experience in evaluating research proposals Madis Saluveer Head, Department of Research Funding Estonian Research Council Regional workshop "Science evaluation as a prerequisite for promoting excellence in research", Chisinau 6 November 2012 1

Outline • Institutional restructuring in Estonia in 1990 s and its outcomes • International evaluations of Estonian R&D Changes in the institutional financing system • Investing into the development of R&D system • Evaluating research - issues for discussion Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 2

R&D and higher education reform in 1990 s • Legislative acts to support the functioning of the new system were adopted • Research financing system reformed • Institutional reform launched • International evaluations of research areas and the R&D funding system carried out • Higher and post-graduate education brought into accordance with international criteria Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 3

Institutional reform in 1990 s • 1989 -1990 - the Estonian Academy of Sciences was re-organised into a classical (personal) type of Academy -> former Academy research institutes were merged with universities Nature 1 Oct 2009: “With the exception of Estonia, the basic research in the region (CEE) is still controlled by the national academies of science. . . ” • 1990 -1991 All Soviet Union ministerial branch institutes dissolved • 1990 - Estonian Science Foundation and Estonian Innovation Foundation founded • 1997 – Research Competence Council (=>Estonian Research Council) established • 1997 - Archimedes Foundation founded (management of EU framework programmes + single NCP network) Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 4

Establishing the R&D legal „backbone“ • 1994 – „Organization of Research and Development Act“ adopted by the Estonian Parliament; latest amendments approved by the Parliament in February 2011 (researcher career path, financing instruments, re-organisation of funding agencies) • 1995 – Universities´Act adopted • 1997 – Estonian Academy of Sciences Act • 2001 - Estonian R&D Strategy 2002 -2006 “Knowledge-based Estonia” approved by the Estonian Parliament • 2006 - Higher Education Strategy for 2006 -2015 approved • 2007 - Estonian R&D and Innovation Strategy 2007 -2013 “Knowledgebased Estonia II” adopted by the Estonian Parliament Regional workshop "Science evaluation as a prerequisite for promoting excellence in research", Chisinau 6 November 2012 5

Rational of performance-based funding The rationale of performance funding is that funds should flow to institutions where performance is manifest: ‘performing’ institutions should receive more income than lesser performing institutions, which would provide performers with a competitive edge and would stimulate less performing institutions to perform. Output should be rewarded, not input. Regional workshop "Science evaluation as a prerequisite for promoting excellence in research", Chisinau 6 November 2012 6

International evaluations of Estonian R&D system 1 Evaluation of all R&D areas by Royal Swedish Academy of Sciences and the Swedish Research Councils 1991 -1992 • Approach: broad and deep evaluation of the whole system; no broad legislative background existed • Goals: to receive an unbiased opinion on the level of research according to the standards used in Scandinavia; to introduce the international norms; to get advice what to do • Outcome: More than 400 reports presented by all research groups in all fields of research; 8 expert panels paid site visits; 19 reports produced Observations and recommendations: • Restructure the policies for decision-making • Integrate research and education • Unequality across the fields • Stop publishing in local papers without peer review-> promote peer review • Lack of young scientists -> normalise age structure, increase the number of research students • Establish/ promote international collaboration Regional workshop "Science evaluation as a prerequisite for promoting excellence in research", Chisinau 6 November 2012 7

International evaluations of Estonian R&D system 2 2000 -2003 – the first systematic evaluation based on established legal basis (ORDA) Approach: full and deep evaluation by the fields of science – 46 expert groups (approx. 200 foreign experts from 18 countries) Outcome: Very good job done, a great number of substantial recommendations put forward; however, the results were heavily underused. Process costly, slow, involved a lot of bureaucracy; assessment methodology changed/improved over the evaluation period; units of assessment varied from small research groups to whole institutes; rapid change of R&D landscape over the evaluation period Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 8

International evaluations of Estonian R&D system 3 2009 – new regulation in ORDA: introduction of 2 types of evaluations 2010 - regular evaluation ( “wide and shallow”, every 7 years): Approach: voluntary, covers the whole R&D system, fast, low cost, low bureaucratic burden - based mostly on existing information, full usage of Estonian Research Information System ETIS; peer-review carried out by foreign experts (16 peers); strict rules; simple outcome (yes/no) Consequences: • financial: access to state research funding instruments – base-line funding, targeted funding • for HEI-s – prerequisite for the Ph. D studies Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 9

International evaluations of Estonian R&D system 4 Targeted evaluation (“narrow and deep”; „tailor-made“ : topic, subjects of evaluation, terms of references, evaluators and the detailed mechanism of evaluation may vary) Goals may vary: • gaining a profound insight of a research field • gaining information for outlining new research policy measures, e. g. R&D strategies, research programmes, major research policy decisions, funding mechanisms, etc, and/or for elaborating the existing ones • may be carried out in single or in multiple recearch areas Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 10

International evaluations of Estonian R&D system 5 2011 – first targeted evaluation carried out (plant and soil science) Approach: based on „tunnetatud vajadus“; self-assessment reports; international peer review (5 experts from 5 countries), site visits; no grades but policy recommendations Outcome: evaluation committee report with observations and recommendations http: //www. etag. ee/wp-content/uploads/2012/05/SEV_PMT_report_2012. pdf Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 11

Estonian R&D expenditure 2001 -2010 Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 12

Number of publications of Estonian researchers indexed in Thomson Reuters Wo. S Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 13

Estonian participation in EU framework programmes Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 14

FP 7 budget per researcher (thousand €) Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 15

Estonian Research Information System ETIS: some facts • ERIS (predecessor of ETIS) 1999 -2006 • Need for large scale developments (real-time multi-partner dynamic system) • Spring 2006: launch of ETIS with pre-installed data (users, publications, projects, etc) Evolution of ETIS as data repository: • 2008 - 78 000 publication records • 2010 - 100 000 publication records • 2012 - 134 000 publication records • 2012 - 39 different electronic forms in use • September 2012: more than 76 000 visitors (Estonia 71 393, Finland 1 212, Germany 860, USA 581, UK 569, Sweden 547, Latvia 312, Italy 300, France 273, Spain 243) Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 16

ETIS functionalities • an information channel and a tool for researchers and R&D institutions • a gateway where researchers can submit grant applications and reports • an instrument for funding bodies to process applications, reports and reviews • a state register of R&D institutions, researchers, financiers and research activity • a gateway to Estonian science landscape • a means to get fast and adequate R&D statistics • a trusted data repository Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 17

ETIS classification schemes • Research Classification Scheme • Classification of publications in ETIS • List of internationally acknowledged publishers Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 18

Classification of publications in ETIS Articles in journals 1. 1. Scholarly articles indexed by TR Web of Science or published in journals indexed by ERIH categories INT 1 and INT 2; 1. 2. Peer-reviewed articles in other international research journals with an ISSN code and international editorial board, which are circulated internationally and open to international contributions; articles of ERIH category NAT; 1. 3. Scholarly articles in Estonian and other peer-reviewed research journals with a local editorial board; scientific articles in journals important for Estonian culture Akadeemia, Looming, Vikerkaar. 1. 2. Books/monographs 2. 1. High-level peer-reviewed research monographs; 2. 2. Other monographs; 2. 3. Dissertations published in a series of dissertations (excluding manuscripts). 3. Articles in proceedings/a chapter in a book or in a collection 3. 1. Articles/chapters in books published by the publishers listed in Annex (incl collections indexed by the TR Conf Proc Citation Index); 3. 2. Articles/chapters in books published by the publishers not listed in Annex; 3. 3. Specific research publications (dictionaries, lexicons, sets of maps, (field) guides, text-critical publications); 3. 4. Articles/presentations published in conference proceedings not listed in Section 3. 1; 3. 5. Articles/presentations published in local conference proceedings 4. Editing scientific publications 5. Published meeting abstracts 6. Other publications 6. 1. Full articles in encyclopedias; 6. 2. Text books and other study materials; 6. 3. Popular science articles; 6. 4. Popular science books; 6. 5. Essays accompanying books of fiction; 6. 6. Articles in other journals and newspapers; 6. 7. Other creative activities Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ 19 Chisinau 6 November 2012

Classification of research fields in ETIS Biosciences and Environment (12 sub-fiels): Biochemistry, Microbiology, Genetics, Ecology, Biosystematics and –physiology, Forestry, Agricultural Sciences, Food Sciences, State of the Environment and to Environmental Protection, Substances Hazardous to the Environment, Geography and Regional Studies, Environmental Policy, Environmental Economy and Environmental Law, Biotechnology, Molecular Biology, Cell Biology, Biophysics and Economic and Technological Research relating to the above fields Culture and Society (14 sub-fields): Philosophy, Theology, History and Archaeology, Cultures Research, Aesthetics and Arts Research, Philology and Linguistics, Law, Psychology, Special Education, Educational research, Social Sciences, Economics, Political Science and Administration, Communication and Information Sciences Health (11 sub-fields): Biomedicine, Veterinary Medicine, Pharmacy, Dental Science, Nursing Science, Public Health Science, Clinical Medicine, Sport Sciences, Nutrition, Occupational and Environmental Medicine, Biochemistry, Genetics, Microbiology, Biotechnology, Molecular Biology, Cell Biology, Biophysics and Bioinformatics relating to the above fields Natural Sciences and Engineering (17 sub-fields): Architecture and Industrial Design, Geosciences, Space Research and Astronomy, Mathematics, Statistics, Information Processing Sciences, Telecommunications, Electrical Engineering and Electronics, Medical Engineering, Physics and Technical Physics, Chemistry and Chemical Technology, Process Technology and Materials Science, . Mechanical Engineering, Automation Technology and Manufacturing Technology, Industrial Engineering and Management, Construction and Municipal Engineering, Energetics, Biotechnology relating to the above fields Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 20

Evaluating research proposals - issues for discussion • • Types, scope and frequency of evaluations Peer-review and/or „metrics“ Scope of use of bibliometrics Impact of research evaluations on funding Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 21

Evaluation of research proposals 2 1994 -2012: Estonian Science Foundation (personal) grants • Assessment scheme: set requirements for PI (presence of mandatory publications), international peer review (2 peers), expert panels => low competition, very small grants (~10, 000 €/year), high number of grants (total 500 – 600) 2012 replaced by Estonian Research Council’s personal grants • Assessment scheme: international blind peer review (2 -3 peers), expert panels, final assessment by the Evaluation Committee => very high competition, average grants (50 – 70 000 €/year), small number of grants (2012: ~20, total ~100) 1997 – 2012: targeted financing of research topics (multi-annual team funding) • Assessment scheme: set requirements for PI and team (presence of mandatory publications), international (blind) peer review (2 -3 peers), expert panels, final approval by MER => medium competition, small grants (~100 000 €/year), average number of financed topics (~200) 2012 replaced by Estonian Research Council’s institutional funding (multi-annual team funding of institutions) • Assessment scheme: international blind peer review, expert panels, final approval by MER => high competition, average grants (~200 000 €/year), low number of grants (2012: 31; total ~100) Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 22

Evaluation of research proposals 3 MER Centres of excellence programme 2001 -2007 (2 calls) • Assessment scheme: peer review by local experts ; final approval by MER => 4 – 11 research groups per Co. E; average competition, total 10 Co. E, total programme 6. 4 M€ Centres of excellence in research programme 2008 -2013 (2 calls) • Assessment scheme: international peer review (30 peers); final approval by Programme Steering Committee => 4 -8 research groups; high competition, total 12 Co. E, total programme 55. 3 M€ Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 23

Evaluation of national programmes MER national programmes (limited domains, e. g Estonian language & literature, history, folklore; national collections; language technology) • Assessment scheme: peer review by local experts; final approval by the Steering Committee => small teams, low competition, small grants (15 -50 000 €/year) EU-financed national thematic R&D programmes 2007 -2013 (6 programmes, total 51. 8 M€) • Assessment scheme: peer review by local and international experts; panel meeting and final approval by the Steering Committee => complementary funding, targeted R&D domains, low competition, grant sizes vary by domain (e. g. Health technologies 200 -500 000 €, environmental technologies 130 – 400 000 €, materials and nanotechnology 200 – 850 000 €, etc. ) Regional workshop "Science evaluation as a prerequisite for promoting excellence in research", Chisinau 6 November 2012 24

OP 2 – Operational Programme for the Development of Economic Environment Measure 2. 1: Thematic R&D programmes aimed at long-term economic development SUPPORT 51, 8 mil eur Priority 2: Improving the competitiveness of Estonian R&D activities through the research programmes and modernisation of higher education and R&D institutions SUPPORT 346, 7 mil eur Measure 2. 4: Modernising research apparatus and equipment SUPPORT 110, 7 mil eur Measure 2. 2: Centres of excellence in research and participating in research cooperation programmes of the EU and the Baltic Sea region SUPPORT 55, 3 mil eur Measure 2. 3: Modernising the general infrastructure of R&D institutions SUPPORT 128, 7 mil eur Modernising the general infrastructure for higher education SUPPORT 42, 1 mil eur research Modernising the educational environment of institutions of professional higher education and universities SUPPORT 14 mil eur Modernising the general infrastructure for research SUPPORT 86, 5 mil eur higher education research and higher education Research infrastructures of national importance (Roadmap, investment plan) SUPPORT 29, 4 mil eur Research equipment for R&D institutions (R&D institutions, open call) SUPPORT 46, 3 mil eur Small-scale research equipment for research groups (Research groups, open call) SUPPORT 14, 5 mil eur E-resources programme SUPPORT 6, 4 mil. Copyright eur 25

Estonian RI Roadmap • RI roadmap – a long term (10 -20 years) planning instrument that lists research infrastructures of national importance, either new or in need of upgrading • The roadmap will be updated regularly (at an interval of 3 to 5 years) to take into account the changing circumstances and opportunities • Being part of the roadmap does not mean that a RI will be funded but the roadmap will be used as an input for the investment decisions • Budget 29. 4 M € for 2010 -2015 • Financed from EU ERDF (85%), state budget (10%) and own contribution (5%) • https: //www. etis. ee/Portaal/infrastruktuur. aspx Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 26

Evaluation of research infrastructure proposals Support for small-scale research equipment (total: 14. 5 M€) • Assessment scheme: annual open calls; research groups; 2 step review: local peer review and approval by ETAg Evaluation Committee; Support for R&D institutions (total: 46. 3 M€) • Assessment scheme: open calls; R&D institutions; 2 -step review: local peer review and approval by programme Steering Committee; RIs of national importance (total: 29. 4 M€) • Assessment scheme: call for expressions of interest; call for full proposals (investment plans); R&D institutions; main and reserve list, final approval by Estonian Government; Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 27

Thank you for your attention! Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 28