Counterfactual impact evaluation What it tells us and
- Slides: 29
Counterfactual impact evaluation What it tells us… and what it doesn't Daniel Mouqué DG Regional Policy, European Commission Regional Policy
Reminder: Counterfactual = comparison Regional Policy
In practice, comparison group Regional Policy
This imposes conditions… • Similar intervention over large "n" • (law of large numbers) Regional Policy
… which only hold for certain measures • Interventions which target individuals or enterprises • Not infrastructures (exception: impact of infrastructures on individuals) • Perhaps for area based initiatives (provided similar goals/means) Regional Policy
There also data needs 1. Good data on the intervention (nature, scale, dates) 2. Good data on target indicators (before and after, including for non-beneficiaries) 3. The ability to link 1 and 2 Regional Policy
Lessons learned from enterprise support studies • DG Regional Policy doing & encouraging since 2008 • What are we learning? And what would we like to know? In terms of: 1. Investment, capital constraints and other market failures (and how vary by firm and support size) 2. Impact of support on the enterprise (productivity, innovation, employment) Regional Policy
What do we learn… … ABOUT INVESTMENT, CAPITAL CONSTRAINTS AND OTHER MARKET FAILURES? Regional Policy
Impact on investment in Eastern Germany (GEFRA 2010) Regional Policy
E. Germany not an isolated example 2 Extra investment/public support 1. 5 1 0. 5 0 R&D Germany Grant Eastern Germany SME Piemonte Regional Policy SME Poland Law 488, Italy
Small is beautiful 1 – the firms Scheme Study Finding on large enterprises Investment grants, IT ASVAPP (2012) No or negative impact firms >250 employees Comparison between SME size classes Thorough exam: no difference RSA invest Crisculo et No impact for firms > Impacts slightly grants, UK al (2012) 150 employees higher for firms < 50 employees? DK CEBR, No impacts for firms > Not examined Innovation Denmark 150 employees Consortia (2010) Innovation Czarnitzki Small much better, but Smaller may do support, DE et al (2011) firm or grant size? better Invest GEFRA/IAB (Did not study large No difference by support, E. (2010) enterprises) SME size class Regional Germany Policy
Small is beautiful 2 – the support • ASVAPP (2012) even controlling for firm size, smaller grants more effective (cpj € 79, 000 for smallest grants, rising to € 489, 000 for largest). • ASVAPP (2012) outright grant to SMEs similar effect to soft loan of same size • Czarnitzki et al (2011) presence or absence of a grant was the crucial factor - smallest grants had almost the same innovation impact as the largest • Comparing across studies: schemes of smaller support tended to have better results (eg RSA, UK) Regional Policy
Business advice can be cost effective • Better survival rates 2 -4 years later in North Jutland. • € 7500/net firm € 1500/net job • (Rotger and Gørtz, 2009) Regional Policy
What do we learn? • • • Capital rationed for SMEs, but only partially Grants help – do not substitute private money This argument applies to small enterprises and (probably) to medium sized but not large firms Less support and/or financial instruments would still work Capital constraints not the only market failure: success of advice => information failures more serious, at least for smallest and newest firms? Regional Policy
What would we like to know? • The mechanism for capital constraints? Knowing this would help for… • Targetting by firm? And what too big for support? • More effective solutions than direct financial support? (E. g. change capital market) • What is the optimal level and form of support? • What information failures? • What is good soft support (incl. business advice)? • How to target/tailor by context and firm? => Need more CFs and other types of evaluation Regional Policy
What do we learn… … ABOUT IMPACTS ON THE FIRM? PRODUCTIVITY, INNOVATION & JOBS Regional Policy
Broader more often than deeper Scheme Law 488 invest grant, IT SME support in Piemonte Enterprise support NI SME grants, PL in Poland RSA invest grant UK DK Innovation Consortia Study ASVAPP (2012) Hart & Bonner (2011) Trzciński (2011) Criscuolo (2012) CEBR (2010) > empl ++ > productivity None ++ For loans, not for grants Small but statistically significant None Small but stat. sig. ++ ++ Statistically insignificant Statistically Supported profits grew insignificant 12% more over 10 yrs Regional Policy
A closer look at some exceptions • CEBR (2010) in DK: innovation consortia increased profitability 12% vs controls over a 10 year period (adds up to € 260, 000 extra profits per firm). • Czarnitzki (2007): R&D subsidies in Germany had a significant effect on research and innovation where the firm also benefitted from networking • Czarnitzki (2007): in Finland both financial R&D support and networking effective, and additive Regional Policy
CIS indics, Germany (Czarnitzki, 2011) Regional Policy
But innovation is not a panacea • GEFRA (2010) investment impact of R&D grants < modernisation grants (leverage 0. 91. 0 vs 1. 4 -1. 5). Innovation benefits worth loss in impact? • De Blasio, Fantino & Pellegrini (2009) No additional impact from investment scheme: less tangible nature => more possibilities for deadweight Regional Policy
Jobs created, but < monitoring data Scheme Study Jobs supported Jobs created (monitoring) (from CF) Investment GEFRA/IAB 107, 000 "created", support, E. (2010) plus 439, 000 Germany "safeguarded" Law 488 ASVAPP invest (2012) support, IT SME invest Trzciński grants, PL (2011) 27, 000 82, 000 "gross 12, 000 created" 36, 000 "net" (beneficiary survey) 25, 000 "created" 10, 500 Regional Policy
Job quality good • ASVAPP (2012) average firm salary and productivity same or slightly greater • Trzciński (2011) jobs created in SMEs received similar pay rises to those in the control group – and that jobs were maintained five years after support. Regional Policy
What do we learn? • • • Relatively easy to make firms proportionately bigger (e. g. with grants) More difficult to make firms more innovative/productive (soft support better? ) Measures with less tangible targets eg innovation can be abused (maybe we knew this already? ) Regional Policy
What is left unanswered? And how would we answer this? • Is soft support really the key to innovation? • What is the mechanism for productivity and innovation? What types of innovation influenceable, how to target by firm etc? • What constitutes a "smart" support package? What soft support, what mix with financial support? • How to avoid abuse of innovation and networking measures? => Need more CFs and other types of evaluation Regional Policy
In conclusion … Regional Policy
In summary • New lessons from CFs about impacts (partial capital constraints for SMEs, scaling up effect, importance of information failure) • More to learn about impacts from CFs (e. g. soft support, financial instruments) • Need other evaluation methods to open "black box" of mechanisms (targetting most effective solutions, best investments) • Some factors too intangible for quantified approach? (innovation) Regional Policy
• Beware the man whose only tool is a hammer… • … for every problem comes to resemble a nail • - Abraham Maslow Regional Policy
Regional Policy
For further information Info. Regio: ec. europa. eu/inforegio Impact evaluation centre: http: //ec. europa. eu/regional_policy/information/ evaluations/guidance_en. cfm#2 Regional Policy
- Counterfactual thinking
- Counterfactual thinking
- International initiative for impact evaluation
- Impact evaluation
- Impact evaluation
- Collective impact evaluation
- A lyric poem that tells a story
- During the _____ of the play, romeo is banished.
- Haiku poem examples
- Songlike poem that tells a story
- Title for questionnaire
- Romeo and juliet act 5 scene 3 summary
- The views of ____ comprise the generalized other
- Poetic narrative definition
- Noun pronoun verb adverb
- Stated and implied main idea examples
- Among the four students of class v radha meera
- Topographic map interpretation
- Ophelia tells her father in scene i that hamlet is behaving
- Explanation purpose
- An intricate dance that tells a story
- Adverb tells when
- Adverb tells when
- A song transmitted orally which tells a story
- What if it finishes ppt
- Michael tells sara he weighs 104 pounds
- Narrative writing tells a story
- Koran the alchemist
- Subjects and predicates
- The standard error tells biology