The evaluation of publicly funded research activities An

The evaluation of publicly funded research activities An overview Luis Sanz Menéndez OECD Symposium Madrid 3 July 2008

Issues for debate • Implications of the use of evaluation as a management tool • Changing evaluation "objects". – systems and policies, – programmes, – research 'collectives', – individuals and – institutions.

Evaluation as a management tool • Characterizing evaluation activities - objects - aims - approaches - articulation with decision making processes • Different objects (Chabbal's terminology (1987)) – Actors: individuals & research collectives, – Operators: institutions, procedures, programmes and agencies/services, Policies

Evaluation objects: changing focus • From the beginning: actors at the core of the process Researchers and research teams, research results and projects Organizing the Peer review process • OECD and the evaluation of national systems Peer reviews and indicators • The eighties and the focus on research operators Programmes at the core of new methodological developments • The nineties (1) and the fashion of privatisation Focusing on the performance of research institutions • The nineties (2) up to … : the fashion of 'excellence' and the growing between multiple missions and one sided criteria of performance

Evaluation as a management tool • Different aims 1 - Audits Objective: compliance to pre-established rules, often administrative. Main effect: sanctions. 2 - Assessment of Performance Problems: measures, differentiating outputs from outcomes. Approaches: "Summative"/"ex-post". Effects: hierarchy/positioning, gratifications/ rewards. 3 - Relevance of action/activities Objective: the adequate "implementation structures". Approach: "pro-active" (focusing on the definition of future action). Effects: changes/adaptations in the course of action 4 - Appropriateness (overall strategy) Objective: discussing the aims (against the changing environment) and/or their translation into goals and objectives. Effects: redefinition of the action and of its course.

Evaluation as a management tool • Approaches to evaluation - evaluation process versus tools mobilized - "characterization" (of a situation) versus "judgment" - relations between approaches and couples "object/aim" --> typical articulations • The articulation with decision making processes - 2 different approaches: support to the “boss” or as a means for collective learning. - 4 major constraints --> - Timing ---> - Relevance ---> - Robustness ---> - Credibility ---> to feed back in the decision making process to address issues at stake to resist critics from evaluees to be taken up by stakeholders & decision makers

Objects & approaches: Typical articulations Object/ Target Actors Projects Aims/ Issues Performance Quality Approach/ Process Scientometrics Peer Reviews System Performance Indicators Appropriateness Advisory councils Operators Performance Audits Relevance Renewed schemes Appropriateness & approaches Source: Callon, Laredo and Mustar (eds), 1997, the Strategic Management of Research and Technoloy, Paris: Economica International

Issues for debate related to 'objects' • National policies (OECD role) • The evaluation of operators • Approaches adopted and problems raised by the evaluation of 'research collectives' (labs, groups, centers, institutes…). RAE in UK; INSERM in France, CSIC in Spain, etc. • Problems raised around the evaluation of individuals: CNRS in France; CNAEI in Spain; SNI in México, etc • The specific case of the strategic management of research institutions: how to go from the micro to the meso level?