Advantages and limitations of transnational benchmarking in policy
Advantages and limitations of trans-national benchmarking in policy evaluation Patries Boekholt Technopolis Group Amsterdam patries. boekholt@technopolis-group. com 1 Technopolis Group
This presentation • • 2 Background on benchmarking Methodological issues Case: ‘centres of excellence’ Some considerations on policy benchmarking Technopolis Group
Policy benchmarking en vogue in Europe • Joint ambition: increase S&T expenditures to 3% of GDP by 2010 • EU Member States: common understanding to benchmark policy performance • Motivations for policy benchmarking: • • • To understands where improvements have to be made Understand factors involving performance of policies Learning from ‘best practices’ or ‘good practices’ Setting standard and targets for performance Taking part in the process is already helping to learn: ‘naming and shaming’ • Larger demand to integrate cross-border benchmarking in evaluations of RTD policy instruments 3 Technopolis Group
Methodological issues (1) • RTD policy instruments operate in a specific national/ federal or regional institutional setting and governance structure • The effectiveness of policies depends on their role in a national innovation system • R&D instruments part of a ‘policy mix’: their effectiveness and relevance depend on other policy measures • Transferability /Diffusion of policies limited 4 Technopolis Group
Methodological issues (2) • Comparability of data is limited due to many differences • • • Objectives of instruments differ even within ‘classes of instruments’ Time frames of instrument Financial investments Implementation models ‘Zero base’ levels are not similar Influence of socio-economic context (economic climate, …. ) • Important part of the effects are behavioural • Networking effects • Change in culture 5 Technopolis Group
Example: ‘Centre of Excellence’ programmes • • Netherlands: Evaluation of Leading Technological Institutes (LTIs) Typical example of ‘centres of excellence programme’ Inspiration came from Engineering Research Centres in US (1995) Benchmarking of LTIs programme and LTI centres as part of the evaluation • Benchmark partly based on previous study for Norwegian Research Council 6 Technopolis Group
Competence Centre programmes are spreading 7 Technopolis Group
Problems encountered • Despite comparable ‘centres of excellence programmes’ in various countries, large differences in the detail • Policy objectives determine the positioning of Competence Centres on the ‘scale’ of excellent science and industry oriented research • This effects the performance and evaluation criteria • On the individual centre level: scientific/technological scope and focus do not match e. g ‘Telematics’ Institute • Particularly configuration of academia - competence centre relation had many different models • Different emphasis on changing the academic research culture versus providing useful results for industry 8 Technopolis Group
Competence centres on Stokes’ Quadrants Yes Quest for fundamental understanding No Pure basic research (Bohr) Not a place you’d want to be … Competence centre focus 9 Use inspired basic research (Pasteur) Pure applied research (Edison) Increasing market failure Yes No Considerations of use Technopolis Group
Positioning of the Benchmarks Centres 10 Technopolis Group
Some lessons • Cross-border benchmarking of RTD programmes is particularly beneficial in the early design phase of RTD policies: to learn from success factors and pitfalls in other countries and adapt to own situation • Benchmarking as part of ex-post programme evaluations has many methodological pitfalls • Context dependency hampers the transferability of ‘best practices’ • Beware of simplistic use of results by politicians • Neverthelessons can be learnt using benchmarking: • Failures and pitfalls in the design of existing instruments • Policy decisions on strategic positioning of RTD programmes 11 Technopolis Group
- Slides: 11