Basic questions of policy evaluation design and tuning

Basic questions of policy evaluation design and tuning: a quick reminder presented by Vincent Spenlehauer – Ecole des Ponts Paris. Tech - IFRIS Discussion Mireille Matt INRA - GAEL

Outline of the discussion Objective: to discuss the similarities and specificities of the basic questions related to public policy evaluation vs. research result or S&T policy evaluation • Brief summary of paper • Similarities with assessing impact of public research/S&T policy • Specificities of assessing impact of public research/S&T polic

Brief summary (1) • Design and implementation of the interministerial evaluation of the national policy for road safety (control and sanction for departing from the highway Code)=> drastically reduce the number of deaths on the road • Very detailed chronological story with a socioanalytical dimension (1999 -2003) • Narrates how, with his social scientist position, he was able to orient and structure an uncertain and chaotic process of policy evaluation and influence the subsequent reforms

Brief summary (2) • Phase 1: struggle with different departments in French Ministries (Dpt for modernization and deconcentration, Dpt for security and road circulation…) and the National council for evaluation – back-and-forth => definition of the object to evaluate, the objectives, the perimeter of the evaluation, public authorities and actors concerned • Phase 2: implementation of the evaluation project: design and specifications of the evaluation, methods to be used, external scientific expert – managing the interaction between the scientific hemisphere (the questionning of the evaluation) and the politico-administrative hemisphere (governance of the evaluation process) – Ministries = major stakeholders

Brief summary (3) • Phase 3: Epilogue – Change of the head of the National Instance of Evaluation (budget, definition of specifications, call for proposals…) - choice of winner – steering committee – aim = define a new policy of control-sanction and document all possibilities – wise and persuasive entrepreneur for public policy design=> changes in public policies towards road safety in 2003 • The results of this evaluation is due to the head of National Instance of Evaluation – political strategies – involvement social scientists in the evaluation process – • Political dimension of an evaluation along the whole process. How should social scientists involved in this type of process behave in front of this omnipresent political dimension

Similarities with assessing impact of public research

Evaluation at different steps of the process of public actions

THE DIFFERENT LEVELS OF EVALUATION: a complex process analysed by VS E 1 - E 4 : Relevance (content) and "quality" of conception (decision process) E 1 : Relevance of objectives E 2 : Coherence of objectives E 3 : Relevance and coherence of the "institutional arrangements" E 4 : Coherence between objectives and institutional arrangements E 5 - E 7 : Implementation and results E 5 : Programme management (cost-timing-quality…) = monitoring E 6 : Effects / outputs / impacts = effectiveness E 7 : Match between effects and objectives = efficacy (1) E 8 - E 9 : Efficiency and legitimization E 8 : Match between objectives - institutional arrangements and effects-outputs-impacts : do the same / do better another way = assessing adequate funding, management, contractual behaviour in order for objectives to be achieved in a cost-effective manner = efficiency E 9 : Ex-post relevance of objectives, given the results of the other evaluations = assessing whether initial objectives are still valid in the light of evolving RTD, societal and environmental conditions = efficacy (2)

General issues towards a « good » impact evaluation system • Impact assessment is only one of the levels of evaluation • Complexity : no single best method ; multi-disciplinarity /quantitative and qualitative information; WITH alignment between (implicit or explicit) theoretical basis and which impacts are to be evaluated • Guaranteing the scientific value of evaluation methods (Robustness, Repetability, Appropriability, Transparence, Independance of the evaluators, Confidentiality, Sampling…) • Balance between systematic / standardized /simple approaches and exploratory studies => systematic issues: Evaluation vs monitoring, time fit between R&D activity and evaluation, Availability of data, Evaluation “fatigue“, Cost of evaluation

General issues towards a « good » impact evaluation system • Ensuring that evaluation takes place on a programmed and properly resourced basis • Providing « easy to understand » , « usable » , « credible » results while avoiding meaningless list of indicators / scoreboards • Providing a mechanism for feedback of the results into policy making (learning policy maker) • Interactions between academics, practitioneers, policy makers and research actors for better understanding of scope, relevance and needs => evaluation is a « social process »

General issues towards a « good » impact evaluation system Evaluation as a « social process » (cf. L. Georghiou ) Motives / interest of the actors involved in th evaluation process § Those « being » evaluated ü justification/legitimation ü learning at operational level ü gain new supports for public sources § Those who are the audience of the evaluation ü accountability ü resources allocation ü learning at policy level (pro-active evaluation) § Those performing the evaluation ü academic interest ü consultant business Þ Make purpose and context-dependancy clear before choosing an approach

Specificities of assessing impact of public research

Challenges for an « ideal » evaluation system Trends / challenge for Trends in S&T policy making Multi-level multi-agent decision Variety of goals Variety of Inst. Arrang. Coupling with other policies International collaboration / integration (EU) evaluators Complexity/Separability/Attribution Multiple stakeholders –Coherence of objectives –Weight of different dimension of the evaluation –Handling/using the results Evaluation of policy mix Mix of « evaluation cultures » Flexibility / adaptive learning policy makers Evaluation of shifts Development of competition-based programmes Legitimacy, project fallacy

Challenges for an « ideal » evaluation / challenge for system Trends in research activity evaluators Globalization Solving problem orientation Interdisciplinarity Cooperation S-I linkages Increasing knowledge content IPR regulations International dimension Benchmarking Bias toward short term market-type output Limits of peer reviews Separability/Complexity/ Attribution Network evaluation Evaluation of knowledge/competences /capabilities / capacity Limits of bibliometrics

ASIRPA Challenges • Evaluation of an organization with multiple missions/objectives • Case study approach with quantification • Identifying the cases within the organization • Considering a variety of impacts (measure) • The observed impact is the result of a complex process involving a heterogeneous set of actors evolving over a long period of time => impact pathway

ASIRPA Challenges • Attribution vs Contribution • Problem of project fallacy • Develop a « standardized » method for presenting and anlysing the case (impact pathway, chronology, impact vector, transversal analysis)

Questions? • What could we learn from your experience ? Ø In terms of method used to evaluate the impact of the road safety policy (by consultant)? ØIn terms of actors performing the evaluation of impacts? Consultant vs Academic actors ØIn terms of mechanisms used to feedback the results into policy actions? Ø In terms of transferring the method developped?

THANK YOU FOR YOUR ATTENTION !