The Making of the FP 6 Expost Evaluation

The Making of the FP 6 Ex-post Evaluation Erik Arnold American Evaluation Association, Orlando 12 November 2009

The expert group … don’t blame them for what I say! • • • • Ernst Th. Rietschel (Germany) - Chairman Erik Arnold (United Kingdom) - Rapporteur Antanas Čenys (Lithuania) Andrew Dearing (United Kingdom) Irwin Feller (United States of America) Sylvie Joussaume (France) Aris Kaloudis (Greece/Norway) Lene Lange (Denmark) Jerzy Langer (Poland) Victoria Ley (Spain) Riitta Mustonen (Finland) Derek Pooley (United Kingdom) Nicoletta Stame (Italy) 2

What’s changed in Framework Programme goals? • • From FP 5 “A successful and scientifically strong European industry and high quality of life for citizens” To FP 6 as an instrument for reaching Lisbon, Barcelona, ERA introduction of (shifting) policy goals 3

Structure of the Framework Programme (€ 20 bn 2002 -6) • 6 FP for RTD and Demonstration (93%) 1. Focusing and integrating Community research (76%) 1. Thematic priorities (65%) 2. Specific activities covering a wider field of research (7%) 3. Non-nuclear activities of the JRC (4%) 2. Structuring the ERA (15%) 3. Strengthening the foundations of the ERA (2%) • 1. 2. 3. EURATOM (7%) Priority thematic areas (5%) Other activities in the field of nuclear technologies & safety (0. 3%) Nuclear activities of the JRC (2%) 4

The evaluation • • The first evaluation of a single FP (after 3 ‘Five-Year Assessments’: Davignon; Majo; Ormala) The most solidly evidence-based evaluation of the FP yet undertaken Hence, broke the mould in moving from high-level discussion to an analysis with empirical ‘bottom’ Evaluation issues • • • Rationale Implementation Achievements Recommendations + Vision 5

The Commission’s FP 6 evaluation has focused on the new • • • New member states New instruments (In toto, No. Es, ERANETs x 2, ETPs, OMC …) Activities of DG-ENTR Behavioural additionality International standing of FP 6 (+China + Bilaterals) Bibliometric profiling Network formation in FP 6 Gender equality Ethical review • Plus the traditional ‘participation survey’ across the board 6

The Member States and others’ evaluations • • • Continuation of traditional impact studies in some countries Those who need evaluation the most do the least … From snapshot to video: Sweden and the ‘Gang of 4’ (Sweden, France, Spain, Czech Republic) Associated States: Norway, (Switzerland) ‘Third Countries’: China 7

Achievements of FP 6 • High volume of R&D at high quality • Quality-assured assessment processes • Participation by excellent researchers • • Thematic priorities – two thirds of the FP; appear productive but evidence is general, unsegmented – we’d like to know more EURATOM – captured the ITER global fusion facility. Fission? Indirect but positive evidence on industrial competitiveness Lack of institutional strategies has limited the FP’s strategic influence over the Knowledge Infrastructure No ‘great leap forward’ in ‘structuring the ERA’– probably because we haven’t fully agreed what the ERA actually is Integrated the New Member States Gender, Joint Research Centre – unfinished business 8

Expert group recommendations 1. Better, more transparent design 2. Bigger role for the MS; avoid the tendency towards a Commission monopoly of initiative (eg ERC) 3. Develop focused strategies for ‘Third’ countries: OECD; BRICS, poor countries 4. Add bottom-up experimentation (compare NEST) 5. SMEs matter but stop the silly targeting of lower-capability firms 6. More research infrastructure; more ESFRI 7. More women 8. More young people and mobility 9. Radical overhaul of administration - no more tinkering 10. Broader evaluation approach - we know more but far from enough 9

Vision • • A new, confident and outgoing Europe A proactive partner in the global knowledge society Much stronger global links and collaboration Top down and bottom up combined • Grand Challenges • Great Ideas 10

Self-criticism: not enough policy or process • • • No discussion of the instrument mix at EU level Research institutes omitted from the picture No discussion of the division of labour between the Member States and the Framework Did not take on the changed character of the ‘self-organised’ instruments or their implications for future R&D policy In many places, failed adequately to differentiate among the instruments – too much momentum from the old Framework evaluation tradition Inadequate treatment of the FP design process and the influence of internal incentives in the Commission (not entirely our fault … ) 11

Evaluation, the Commission and the FP • • • Massive changes in extent and visibility of evaluation since the 1999/2000 reforms Early signs of an ‘evaluation culture’ in places Use of programme theory an improvement over the 1990 s • Input - Output - Results - Impacts a useful corrective to the old “we can’t measure the impacts so let’s talk about the programme management” approach • Is there a risk of losing sight of the processes and the political science? • Still opportunities to improve the link between evaluation and the evolution of the FP 12

Thanks for listening erik. arnold@technopolis-group. com 13