The Dutch Innovation voucher Two Examples of Dutch

The Dutch Innovation voucher Two Examples of Dutch Policy Evaluation: randomization and regression discontinuity Hungary May 18, 2011 Marc Van der Steeg CPB Netherlands Bureau for Economic Policy Analysis The Hague, The Netherlands m. w. van. der. steeg@cpb. nl

The evaluation problem The Dutch Innovation voucher n A two-way relation Hungary May 18, 2011 ► causal: policy leads to more innovation ► correlation: innovative firms make more use of innovation policy instruments n How to disentangle these two relations? ► add covariates to the regression equation ► do highbrow econometrics (e. g. matching) ► or. . . n Controlled experiment ► experimental group and control group ► random allocation ► difference is causal impact

The Dutch Innovation voucher The Dutch innovation voucher pilot 2004 Hungary May 18, 2011 n Goal: “Lead them to water and pay them to drink” (Angrist et al. , 2006) ► Introduce SMEs to public research institutes ► Market-oriented incentives for research institutes

The Dutch Innovation voucher Characteristics voucher program Hungary May 18, 2011 n n n credit note, worth max EUR 7500. for SME’s only no own contribution required application-oriented research questions placed with a defined group of institutes no restrictions on level of question or technology n valid for half a year n 100 vouchers available n LOTTERY if demand exceeds supply

Example of voucher project The Dutch Innovation voucher n “Biodiesel from Africa” Hungary May 18, 2011 ► Seeds of tropical plant Jatropha can be used for production of biodiesel ► However, current oilpresses less efficient than for production of biodiesel from coleseed: 40 % lost ► Diligent Energy Systems used an innovation voucher and asked Technical University of Eindhoven to improve efficiency of process – PHD-student carried out experiments – Plans to do further research on adaptation of coleseed press for Jatrophra

The Dutch Innovation voucher Research questions Hungary May 18, 2011 n To what extent did the voucher affect the degree of: 1. Science-industry interaction ► Direct effect ► Persistence effect 2. Innovation ► Product innovations ► Process innovations

Application process The Dutch Innovation voucher n 1044 applications on September 17 th, 2004 Hungary May 18, 2011 n Lottery: ► 100 winners ► 944 ‘losers’

Data (1) The Dutch Innovation voucher n First round of telephone interviews May, 2005 Hungary May 18, 2011 ► 100 winners ► 500 randomly selected losers ► questions about actual and counterfactual behaviour n Response rate ► 71 winners (71%) ► 242 losers (48%) n Second round of telephone interviews in September 2006, with questions on: ► interaction after voucher period ► realized innovations after voucher period

The Dutch Innovation voucher Data (2) Hungary May 18, 2011 n No significant differences between winners and losers in background characteristics: ► firm size ► region ► sector ► previous interaction with public research institutes n We can confidently attribute any differences in outcomes to the voucher policy instrument ► however, we still add all available covariates to regression equation ► this raises precision of effect estimates

Analysis (1): Direct effect The Dutch Innovation voucher n Data on behaviour during voucher period: Hungary May 18, 2011 ► 62 out of 71 (= 87%) winners commissioned a project ► 20 out of 242 (= 8%) losers commissioned a project n Effect estimates ► 13% of the vouchers not used (= (71 -62)/71) ► 8% crowding out (= 20/242) ► 79% additional assignments (= 62/71 - 20/242) n Main conclusion: 8 out of every 10 vouchers additional ► Voucher generates substantial more science-industry interaction

Stated preferences versus revealed preferences The Dutch Innovation voucher n Survey questions: Hungary May 18, 2011 ► Losers: what would you have done if you would have won a voucher ► Winners: what would you have done if you would not have received a voucher ► 76% winners say: without voucher, fewer projects ► 86% losers say: with voucher, more projects n Conclusion: in this case, stated counterfactual behaviour gives good proxy for real causal effect ► But need not always be the case!!!

Other findings The Dutch Innovation voucher n No indications for effect on size of project Hungary May 18, 2011 ► Most SME’s commissioned project of more or less voucher value n Indication for small timing effect (of 1 out of 10) ► Voucher winners claimed that voucher did not affect number of projects, but only led them to carry out a project earlier in time. n Same analysis on two 2005 voucher pilots shows robustness of results ► Van der Steeg et al. (2007) – 2005 -1: 7 out of 10 – 2005 -2: 5 out of 10

Analysis (2): Persistence and innovation output effect The Dutch Innovation voucher 1. Persistence: no effect Hungary May 18, 2011 ► Voucher winners do not carry out more assignments than voucher losers in 1½ year after voucher period. ► Reasons for not carrying out new assignment: too expensive, no new questions, own research 2. Innovation output: mixed evidence ► positive effect on process improvements ► no significant effects on realization of new or better products and new processes 3. Indications for crowding out of own R&D

Note on importance of method The Dutch Innovation voucher n Randomization was crucial Hungary May 18, 2011 n Applicants were not at all representative for Dutch SME’s!!! ► average SME (11 -50 workers): 14 % realized product innovation (CIS data) ► Voucher applicants (11 -50 workers) : 75 % n Failing to control for differences in innovation capacity would lead to huge overestimation of effects ► Unobservable factors can be crucial, e. g. timing of having an innovative idea may determine decision to apply for an innovation subsidy

The Dutch Innovation voucher Crucial issues for design and evaluation of voucher scheme Hungary May 18, 2011 1. 2. 3. 4. Identify measurable goals of voucher policy Randomization is crucial for evaluation! Sufficient vouchers and size of control group Collect pre-treatment characteristics / behaviour of firms ► via application form; administrative datasets 5. Make cooperation to evaluation obligatory ► also for control group of losers in lottery! 6. Avoid abuse ► e. g. print assignment to copy shop of university 7. Do not let losers of lottery apply in next wave ► or large enough time span between waves

The Dutch voucher instrument after 2004 pilot The Dutch Innovation voucher n Two new larger pilots in 2005: 1000 vouchers Hungary May 18, 2011 n Definitive instrument since 2006: ► around 6000 vouchers annually ► Split in small (E 2500) and large (E 7500) vouchers n 2010: introduction of “private” vouchers ► Possibility to commission question to private knowledge provider n 2011: new government stopped voucher program (as part of large budget cuts)

The Dutch Innovation voucher Dutch cohesion policy (1) Regression discontinuity Hungary May 18, 2011 n 2007 plans by government to improve 83 most disadvantaged neighborhoods n Substantial budget of 350 million euro per year n Neighborhood action plans with goals to improve housing, jobs, education, integration and safety.

The Dutch Innovation voucher Dutch cohesion policy (2) Setup and evaluation design Hungary May 18, 2011 n Ranking of neighborhoods on various socioeconomic indicators: worst 83 have been selected n Evaluation design: ► Exploit ranking with cut-off at neighborhood 83 ► Exploit availability of before and after information on range of relevant outcome indicators n Diff-in-diff: compare development in 1 -83 with that in 84 -183 n ‘ Local’ diff-in-diff design: 50 -83 versus 84 -117

The Dutch Innovation voucher Same pre-trends supports common trend assumption: Liveability index (scale 1 -7) Hungary May 18, 2011

The Dutch Innovation voucher Common pre-trends also for wide range of other characteristics Hungary May 18, 2011 % of social housing % of non-western immigrants

The Dutch Innovation voucher No effect on primary school test scores after one year Hungary May 18, 2011 But we need to measure later-year effects…:

Other evaluation plans: Innovation and Science policy The Dutch Innovation voucher n Innovation loans Hungary May 18, 2011 ► Selection of proposals on basis of certain criteria / judgements ► Compare performance of just selected versus just not selected companies n Scholarships for talented researchers “Veni & Vidi grants” ► Selection of researchers on basis of ranking of proposals by comittee ► Compare scientific output & careers of just selected versus just not selected applicants

Contact The Dutch Innovation voucher n Report on innovation vouchers: Hungary May 18, 2011 http: //www. cpb. nl/eng/pub/cpbreeksen/discussie/ 58/ n Contact: m. w. van. der. steeg@cpb. nl