Theory Driven Evaluation tracing links between assumptions and

Theory Driven Evaluation: tracing links between assumptions and effects Sixth European Conference on Evaluation of Cohesion Policy Warsaw, 30 November – 1 December 2009 Karol Olejniczak, k. olejniczak@uw. edu. pl EUROREG – University of Warsaw: www. euroreg. uw. edu. pl

Theory-Driven Evaluation in a nutshell • It relates evaluation research with the scholarly socio -economic theories • It treats programme as optimistic assumption about causal relations: If we DO…. than we GET… and than… • Programme is a set of theories… – Underlying Theories (knowledge, experience, influences) – Theory of Change (assumption about strategic change) – Theory of Implementation (the way every-day work is organized) • . . . that works in a certain context & circumstances • TDE is and approach, it is method-neutral – Logic models – 5 stages procedure

5 stages procedure STAGE 1 Understanding the context STAGE 2 Defining Theory of Change and tracing its Underlying Theories STAGE 3 Reconstructing detailed Theory of Change and its indicators STAGE 4 Tracing the real change outputs and effects STAGE 5 Explaining the obtained effects

Theory Driven Evaluation in Practice Case Study of Ex post Evaluation Neighbourhood Program INTERREG/TACIS CBC Poland-Belarus-Ukraine 2004 -06

Programme • Border regions from Poland, Belarus, Ukraine • Two programmes – two pots of money: INTERREG 37, 8 mln EUR vs TACIS 8 mln EUR • Beneficiaries – local communities, local services, NGO • 173 major projects + 300 micro-projects Contracting Authority: • Polish Ministry of Regional Development, Territorial Cooperation Unit Evaluator: • EGO – Evaluation for Government Organizations s. c. Contract: • 5 monhts (July-Nov 09)

Stage 1: Understanding the context Questions: Answers: • What theories tell us about cross -border cooperation? • CBC depends on permeability of the border & partners complementarities • What type of border it is? • Highly peripheral, underdeveloped area, border as barrier • What was the scale of intervention? • Minor financial impulse, could be visible only on a local scale Methods: • Literature review, review of earlier empirical research • Analysis of general statistics • Review of socio-political situation

Stage 2: Defining Theory of Change & tracing its Underlying Theories Questions: Answers: • What strategic objectives have been set? • Dual objective: IF we act on 5 thematic fields THAN quality of life AND socio-economic integration will be improved in a border area • What were the assumptions and inspirations for these decisions? • Reasonable fields of intervention but no measures for border permeability • Stakeholders choices, intuition, earlier experiences of Poland. Germany CBC Programme Methods: • Desk research (programme documents) • Interviews with key programme stakeholders involved in the programme design

Stage 3: Reconstructing detailed Theory of Change and its indicators Questions: Answers: • How should we define the change in terms of indicators & assessment criteria? • Effects both planned and sideeffects have to be measured on 2 scales: quality of life & integration • How programme designers defined these changes in terms of programme & projects indicators? • They have to be traced in 3 dimensions: thematic (projects topics), relational (partnerships) and territorial (local communities) Methods: • Analysis & assessment criteria base on logic models, they differ in details but scale stays the same • Logic models for each thematic group and partnerships • Assessment system – two scales: quality of life vs integration • Review of the programme indicators

Stage 4: Tracing real changes – outputs & effects Questions: Answers: • What was the funds spatial & thematic distribution? • What was the number & structure of partnerships? • What were the effects of thematic group of projects? • What were the effects of partnerships? • What were the impact on local communities? 1. High disproportion between 3 sides of the border 2. Focus on improving quality of life 3. Local and close-to-border effects 4. The main integration effect was brought by soft-projects, it was institutional, limited integration of local societies 5. Balanced effects (quality + integration) brought by tourism & border security projects Methods: • Local visits of all project sites (different observation tools applied, depending project’s topic) • Survey of project beneficiaries, survey of partnerships • Social survey of twin communities • Interview with local stakeholders, review of local press

Stage 5: Explaining the obtained effects Questions: 1. Why disproportion? 2. Why focus on quality of life? 3. Why close-to-border and local effects? 4. Why institutional integration? 5. Why tourism & border security projects had best effects? Methods: • • Answers: 1. Unbalanced money & procedures; higher experience of Polish teams 2. High peripheriality & local needs; unclear demarcation line 3. Border as a sealed barrier, small funds spread spatially 4. Selection criteria, limited trust – focus on smaller projects, expert-type projects, micro-projects as top-down not bottom up initiatives 5. Tourism easy to combine with next initiatives (multiplier effects), security required official cooperation agreements Brainstorming with experts Second review of qualitative & quantitative data again Interviews with programme managers Survey of unsuccessful and potential applicants

Summing up the case study Main message: • Close to border and local effects. • In a given context every joint project was a success How TDE helped us? • Dealing with complexity - packing & unpacking issues • Focusing exploration on the right level • Making the fair judgement – understanding contextual limitations • Writing the concise report - clear narrative

TDE for cohesion programmes Advantages Challenges • Using too rigid model can • Articulates rationality of the lead to tunnel vision and programme omitting side-effects • Provides clear conceptual • There is a trade-off foundation for the study between level of details • Focuses on effects and treats and clarity of the models implementation issues only • Too much sophisticated as one of the explaining theoretical considerations factors can alienate stakeholders • Relates to scholarly theories and turn evaluation into and give bigger picture scholar research • Allows to discuss causal relations

Bibliography • Chen, H. T. (2004) Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness. Thousands Oaks: Sage Publications, Inc. • Donaldson, S. I. (2007) Program Theory-Driven Evaluation Science: Strategies and Applications. New York: Lawrence Erlbaum. • Knowlton, L. W. & Phillips, C. C. (2008) The Logic Model Guidebook: Better Strategies for Great Results. Los Angeles: Sage Publications, Inc. • Leeuw, F. L. (2003) Reconstructing Program Theories: Methods Avaliable and Problems to be Solved. American Journal of Evaluation, 24(1), pp. 5 -20. • Patton, M. Q. (2008) Utilization-focused evaluation. 4 th edition. Los Angeles, London: Sage Publications. • Pawson, R. (2009) "Introduction to Realist Evaluation and Realist Synthesis", lecture on The Academy of Evaluation, EUROREG – University of Warsaw, 7. 02. 2009 • Weiss, C. H. (1997) How can theory-based evaluation make greater headway? Evaluation Review, 21(4), pp. 501 -524. • Weiss, C. H. (2004) On Theory-Based Evaluation: Winning Friends and Influencing People. Evaluation Exchange, IX(4), pp. 2 -3.

Contact details Karol Olejniczak, Ph. D EUROREG – University of Warsaw www. euroreg. uw. edu. pl • E-mail: k. olejniczak@uw. edu. pl • Phone: +48 22 826 16 54 • Mobile: +48 696 41 22 82