Evaluation of Science Foundation Ireland A Case Study

Evaluation of Science Foundation Ireland: A Case Study of a Programme Designed to Enhance National Research Quality Dr. Jim Ryan Michael Fitzgibbon www. circa. ie

CIRCA Group Europe CIRCA is an international consulting company, established in 1991 in Dublin, Ireland. Member of ETAN (European Technology Assessment Network) and the European S&T Observatory (ESTO). 12 Consultants in different areas of S&T, all with significant experience of European S&T issues and systems. The major Irish consultancy company specializing in S&T policy and management issues. www. circa. ie 2

About Forfás • Forfás is the national board responsible for providing policy advice to Government on enterprise, trade, science, technology and innovation in Ireland. • Legal responsibility for the promotion and development of these sectors is vested in Forfás by the State, through the Department of Trade, Enterprise and Employment. www. forfas. ie Michael. Fitzgibbon@forfas. ie 3 9/15/2020 www. circa. ie 3

Presentation Outline § Irish infrastructure – the context § Background to Science Foundation Ireland § Evaluation Process www. circa. ie 4

Ireland in Summary § § § Population 3. 98 million Total at work 1. 81 million (was 1. 1 million in 1990) GDP per capita € 33, 919 § A GROWING ECONOMY - 2003 § § § GDP Exports Imports Trade Surplus € 135 bn € 113 bn € 92 bn € 21 bn - 15. 5% of GDP The average annual rate of increase in Irish GDP has been 9% over the 7 year period 1994 - 2000, the highest in the EU and the OECD www. circa. ie 5

S&T Context: Some notable features 1. Ireland has historically had very low S&T investment 2. Universities & Institutes of Technology are almost the sole public source of technology & RTD services: Only Public RTD institute is in Food/Agric. 3. S&T was a low priority, and the agency structure was under-developed and staffed, until very recent times www. circa. ie 6

Profile of Enterprise in Ireland Most of the products we manufacture are designed elsewhere Bulk of our exports are marketed /sold by organisations outside Ireland www. circa. ie 7

Changing the Capability Profile of Enterprise in Ireland § Need to Build World Class Expertise § International Sales & Marketing § Operations § R&D/Technology § Focus on Sectors & Niches www. circa. ie 8

Technology Foresight Ireland: Conducted 1998; published by Forfás 1999 “A world class research capability in selected niches of … enabling technologies is an essential foundation for future growth. These key technologies require new skills, and in particular the development of a cadre of world class S&T personnel. ” www. circa. ie 9

Ireland –S&T & the Economy 1. Foreign Direct Investment is important to economy (~130, 000 jobs; 14% of GDP) 2. Innovative capacity seen as important to the attraction and retention of FDI firms 3. Also vital to the growth of indigenous companies Investment in RTD was seen as important: o o To provide an incentive for Foreign firms To develop indigenous high-technology industries ENTERPRISE IRELAND www. circa. ie 10

“. . There will be a major accelerated increase in Research, Technological Development and Innovation (RTDI) investment with the objectives: § to strengthen the capacity of Irish third level institutions and other research establishments to conduct research relevant to the needs of the Irish economy; § to strengthen the capacity of Irish firms to assimilate the results of R&D into their products and processes, and; § to provide support for sectoral research in agriculture, food, marine and the environment. . ” www. circa. ie 11

RTD and Innovation Funding in National Development Plan 2000 – 2006 Total : € 2. 5 billion PRTLI Capital Facilities & Research Infrastructure in Universities € 690 m Other Sectoral & Industry Supports Science Foundation Ireland. Quality and Scale of Basic R&D € 1, 160 m € 650 m www. circa. ie 12

Research Community view: SFI v PRTLI provides infrastructure for research performance Bricks & Mortar Science Foundation Ireland provide researchers Grey Matter www. circa. ie 13

Programme for Research in Third Level Institutions (PRTLI) Forced Universities to develop Strategic Plan for RTD Built 33 new Research Centres Provided an additional 90, 000 m 2 research space Funded 62 new or expanded research programmes Increased undergraduate Post-grad places by approx 1, 500 Encouraged inter-disciplinary research and inter-institutional collaboration Encouraged greater interaction between research and teaching Conway Institute – Dublin Nanofabrication facility – Cork Materials & Surface Science Institute – Limerick www. circa. ie 14

‘Helping Ireland Recruit and Retain Research Groups’ § SFI was established in 2000 and became operational in 2001 § ‘SFI is investing € 646 million in academic researchers who are likely to generate new knowledge, leading edge technologies, and competitive enterprises in the fields underpinning two broad areas: § Biotechnology § Information & Communications Technology’ Vision Statement Through strategic investments in the people, ideas and partnerships essential to outstanding research in strategic areas, SFI will help build research of globally recognised excellence and nationally significant economic importance. www. circa. ie 15

§ Grants are allocated solely on the criterion of excellence as determined by external, international peer review: relatively unusual in EU § Have established 163 research groups led by principal investigators of whom 34 have come to Ireland from laboratories abroad § SFI groups employ over 1150 research staff and 450 Ph. D students § Established 6 major R&D Centres in collaboration with industry Schemes Committed €m 2001 -2005 (April) No. of Awards 2001 -2005 Investigators and Fellows 254 128 CSETs 108 6 Research Professorships 30 10 Research Frontiers 12 107 Young Researcher 5 4 Other 46. 5 182 Total 462. 5 437 www. circa. ie 16

Six Centres for Science, Engineering and Technology (CSETs) have been established in core thematic fields and with considerable industry involvement. The CSETs are in the following areas : § § § Adaptive Nanostructures and Nanodevices Regenerative Medicine Digital Enterprise Research Alimentary Pharmabiotics Telecommunications Value Chain Human Proteomics (CRANN in Trinity College Dublin) (REMEDI in NUI Galway) (DERI in the NUI Galway) (APC in University College Cork) (Trinity College Dublin) (College of Surgeons, Dublin) www. circa. ie 17

Review of the performance and impact of SFI to date. § Requested by Department of Enterprise, Trade & Employment at end ‘ 04 § Managed by Forfás. § Department ‘Accept that SFI is a long-term investment. . . § …. but also know that the early stages are critical’ § it is therefore ‘not too early to form an initial impression about the operational performance of SFI to date’ www. circa. ie 18

Review of the performance and impact of SFI to date. Evaluation Methodology: 1. International review panel led the evaluation. Small size of Ireland 2. Consulting groups were commissioned (by public tender) to provide support and information inputs to the panel. makes it difficult to find local independent panel-members for such evaluations. External panels are therefore the norm. § CIRCA Group conducted the Peer Review Assessment and the Bibliometric analysis ( with CWTS, Leiden, NL) § Technopolis (UK) conducted a survey of industry views www. circa. ie 19

Members of International Panel § Professor Sir Richard Brook (Chairman) Director of the Leverhulme Trust (UK) § Prof. Karin Markides Deputy Director General of Vinnova, Swedish agency for Innovation (Sw) § Dr. David Clark Chair of Computer Science & Telecomm Board, US National Academies (US) § Dr. Wilhelm Krull Secretary General of the Volkswagen Foundation (D) § Prof. David Finnegan Institute of Cell & Molecular Biology, Univ. of Edinburgh (UK) § Mr. Pat Toole Retired Senior VP of Corporate Manufacturing & Technology of IBM (US) www. circa. ie 20

Terms of Reference Appropriateness or Efficacy: Are the objectives still consistent with the Irish research system and with national research and innovation policies? Is the investment in SFI appropriate to meet its objectives? Are there sufficient and appropriate complementary measures to SFI which would allow the desired economic effects to materialise? Effectiveness: Is SFI effective in meeting its objectives? Are its programmes and activities likely to lead to the desired outcomes? What are the outputs and impacts of its activities? What impact is SFI having on the research system as a whole? Efficiency: Examine the operational efficiency of SFI in relation to a range of relevant issues (e. g. application and review procedures, monitoring). www. circa. ie 21

Panel Methodology • Conducted discussions with a representative cross-section of relevant groups: § SFI-Funded Researchers (44) , post- doctoral staff (33) and students (34) funded under SFI awards; § SFI management and administration (5) § Management staff of institutions housing www. circa. ie 22

Evaluation of the Peer Review system CIRCA Group Process Design § Peer selection: qualifications, nationality*, experience, gender, etc § Peer process: criteria used, information provided to peers; ranking and rating processes; § Internal SFI Process: mechanisms and speed of decision-making; rejection rate etc § Supports provided to successful and unsuccessful applicants § Oversight and monitoring: conflict-of-interest rules; appeals process; internal audit/evaluations Process Operation § Random Case files were examined to assess the practical operation of the review process. These were balanced by discipline, university, successful/unsuccessful applicants, and by type of award. * No Irish reviewers are used www. circa. ie 23

Bibliometric Analysis CIRCA Group and CWTS (Leiden) Two studies were commissioned to assess: § Quality of the SFI-funded researchers from non-Irish institutions prior to their coming to Ireland. This was to determine whether the SFI system had selected excellent researchers § The publication record of researchers who had been working in Irish institutions prior to funding. This was to assess whether the receipt of funding had affected their publication output and/or quality Note that appointments of researchers by SFI commenced in 2001 www. circa. ie 24

Percentage of SFI-funded researchers with a higher than average number of papers in the top 10% and Top 5% of papers in their respective fields Biotech Top 10% ICT Top 5% Top 10% % Top 5% % Foreign* Researchers 82 64 73 80 Resident Researchers 68 64 75 50 * i. e. researchers who were working outside Ireland when receiving SFI award www. circa. ie 25

Industry View of SFI Technopolis (UK) In-depth face-to-face or telephone interviews with 24 companies were conducted in January/ February 2005. The interviewees were from a mix of large foreign-owned multinationals, large indigenous firms and small high technology startups, all operating in business sectors with an interest in biotechnology or information and communications technology research. § § § Generally very positive about SFI approach and activities Some concern about lack of incentives in HE sector re IP protection etc Recognise that time is required for policy to bear fruit www. circa. ie 26

Overall View of Panel § SFI is a ‘bold and well-designed path to transform Irish research’ § There are ‘persuasive indications of the suitability and effectiveness of its process’ § The defining aspect of SFI is its ‘demand for excellence in research quality’ § Continued success will need ‘ongoing commitment to sustenance of a research culture by government and universities’ Report to be formally published by Forfás in November 2005 www. circa. ie 27