EXPOST AND ONGOING EVALUATIONS OF EU DEVELOPMENT PROGRAMMES

EX-POST AND ONGOING EVALUATIONS OF EU DEVELOPMENT PROGRAMMES International Evaluation and Methodology Conference Budapest 6 -7 May Evaluation Tools for Programme Management Experience from Previous Ongoing Evaluations IRELAND Dr Jim Fitzpatrick Managing Director Fitzpatrick Associates Economic Consultants www. fitzpatrick-associates. com 122 Ranelagh Village Dublin 6 Ireland E-mail: jfitzpatrick@fitzpatrick-associates. com 1

CONTENTS v context v nature/focus of ongoing evaluations v Irish approach 1994 -99, 2000 -06 v examples of impact on management/implementation v some lessons for 2007 -13? 2

IRISH CONTEXT v participation in four periods: 1989 -93, 1994 -99, 2000 -06, 2007 -13 v limited evaluation in first round, a lot of ongoing evaluation 1994 -99, 2000 -06 v organisational model changed v ongoing evaluation not new, but new emphasis 3

FITZPATRICK ASSOCIATES’ EXPERIENCE 2007 -13 Type of Evaluation Ex Ante Evaluation Ongoing Evaluations 2000 -2006 Mainstream CSF/OPs Border, Midland (Romania) NSRF (Romania) Mid-term Evaluations Community Initiatives Mainstream OPs 1. Local Urban and Rural LEADER + (Irl) Feoga Rural Western Regional OP (Irl) Economic Competitiveness OP (Hungary) Agriculture and Rural Development OP (Hungary) PEACE III/Interreg (Irl) Competitiveness 1994 -99 Development (Irl) North-South 1989 -93 Community Initiatives Mainstream OPs Community Initiatives 1. LEADER II (Irl) Development (Irl) 2. Tourism (Irl) INTERREG Ireland-Wales INTERREG Roads Investment (Irl) Human Resources Investment (Irl) Water, sanitary services (Irl) Border, Midlands, West Regional Operational Programme (Irl) EQUAL Ireland-Wales INTERREG LEADER+ 1. Industry (Irl) 2. Tourism (Irl) 1. LEADER II (Irl) 1. Industry (Irl) 2. Tourism (Irl) 3. CSF and the 1. LEADER II (Irl) 2. Employment and Regions (Irl) 4. Environment (Irl) 5. Agriculture and Rural Development (Irl) Adapt (Irl) 3. Small Business (Irl) LEADER+ Ireland-Wales Mid-term Updates INTERREG (Irl/Wales) Regional Operational Programme (Irl) Community Support Framework (Irl) Ex-post Evaluations CAP Rural Development Objective 1 in Plan (Irl) Management/Imp Systems (Irl) Ireland (EU-wide) ESF in Ireland (EU-wide) * Inc. experience as either lead or subcontractor. 4 LEADER II (Irl) Employment and Adapt (Irl) 1. Water and 1. STAR Sanitary Services (Irl) 2. Rural Development (Irl) 3. Farmyard Pollution (Irl) (telecoms) (Irl) 2. Envireg (environment) (Irl)

MULTI-DIMENSIONAL NATURE OF EVALUATION Stakeholder Involvement Managing the team EVALUATION A Balancing Act Client, user relations Research, Analysis Time, resources, budget 5

FOCUS OF EVALUATION AT DIFFERENT STAGES Stage Ex ante Ongoing Ex Post Evaluation Planning Implementation Accountability 2006 2007 -13 2013+ Rationale *** * * Relevance *** * Effectiveness * ** Efficiency * ** *** Purpose Timing Impact 6

EU REGULATORY REQUIREMENTS 2007 -13: ONGOING EVALUATION v Regulation*: Member States should carry out evaluations linked to the monitoring of operational programmes in particular where: Ø monitoring reveals a significant departure from the goals set Ø proposals are made for the revision of operational programmes v Guidelines**: Ø encourage evaluation which meets programming needs Ø indicate intentions in Evaluation Plans Ø shift from regulation-driven Mid-term Evaluation towards “a more flexible, demand-driven approach…” Ø process related focus *Regulation 1083/2006 Article 48. 3 ** DG Regio, The New Programming Period 2007 -13, Indicative Guidelines on Evaluation Methods: Evaluation During the Programming Period, Working Paper No. 5, April 2007 7

IRISH MODEL 1994 -99 Ongoing Evaluation 1994 -99 Operational Programme v ongoing evaluation supported by ongoing evaluators v mix of internal/external v focus at mainly OP level, varied by OP (exc. ESF) v general focus on issues arising v technical assistance to programming v “watch-dog” role in some cases 8 Internal Unit External Evaluators Human Resources _* _ Industry Agriculture/Rural Development _ Tourism _ Transport _ Local development _ Environment _ Fisheries _ *ESF (Fund-wide) _ CSF (NSRF) _

IRISH MODEL 2000 -06 v became highly centralised at NSRF level Ongoing Evaluation 2000 -06 Internal v single NDP/CSF Evaluation Unit v emphasis on commissioning one-off external evaluations (ongoing, mid-term update) v some internal evaluation also External v school-leaving survey v road investment v childcare v water services v social inclusion co-ordination v human resource targeting 9

MAKING A DIFFERENCE? v Roads investment Ø identified that apparent cost rises were due to under-estimates: contributed to more careful costings v Tourism Ø big projects had unrealistic expectation about absorption rate: led to reallocation of funding v Horizontal Themes Ø identified variable relevance of these: led to operationalising treatment of these in a more selective way v Indicators/Indicator Systems Ø Internal and external evaluations helped develop these 10

ISSUES/LESSONS FOR THE 2007 -13 PERIOD v Organisational arrangements – use ongoing support v Topics – should be problem-solving, real issues in implementation, timely v Partners – involve OPs/IBs (ongoing evaluation normative). Evaluations need “champions” v Terms of Reference – should be specific, not “set piece”, doable scope v Tendering/choosing evaluators – ongoing resource with flexibility v Evaluation Plans – leave room for responsiveness v Skills – may be wider than conventional professional evaluators v Monitoring data – a key precondition v Internal v external – a mixture best v Management – don’t substitute evaluation for management decision 11