Implementing Developmental Evaluation For Peacebuilding And Education Programs
Implementing Developmental Evaluation For Peacebuilding And Education Programs in Conflict-affected Contexts © UNICEF/ETHA-2014 -00422/Ose Bosun Jang & Dr. Ann Doucette 2015 AEA Conference 14 November 2015
From traditional M&E to Developmental Evaluation Bosun Jang
More than 230 million children currently live in fragile settings and areas affected by armed conflicts and violence Countries experiencing internal conflict, 1946 -2014 30% 1/3 64% 80% 2
Côte d'Ivoire © UNICEF/NYHQ 2011 -0260/Asselin 3
Education disparities in Côte d'Ivoire 4
Central African Republic © UNICEF/NYHQ 2007 -0131/Pirozzi 5
Education disparities in Central African Republic 6
Tanzania © UNICEF/NYHQ 2008 -1787/Pirozzi 7
Education disparities in Tanzania 8
Mongolia © UNICEF/NYHQ 2012 -1788/Sokol 9
Education disparities in Mongolia 10
Learning for Peace aims to strengthen resilience, social cohesion, and human security in 14 conflict-affected countries State and policy level Community A / Individual A 1. Burundi 2. Cote d’Ivoire 3. Chad 4. Democratic Republic of the Congo 5. Ethiopia 6. Liberia 7. Myanmar 8. Pakistan 9. Sierra Leone 10. Somalia 11. South Sudan 12. State of Palestine 13. Uganda 14. Yemen Community B / Individual B 11
Learning for Peace aims to transform relationships at three levels, using multiple elements of education as entry points Policy / systems level Safe school environment Teache r training Community level Individual level Yout h Access to conflictsensitive education Early childhood development Conflictsensitive curriculum Gender 12
Selection of DE followed experiences with both strengths and weaknesses of various approaches to M&E Monitoring of outputs Case studies Knowledge, attitude, perception (KAP) surveys Developmental evaluation In-depth research 13
Kicking off DE required sensitization, creative thinking around recruitment, and operationalization in countries Series of sensitization campaigns Recruitment for a developmental evaluator Site selection 14
Implementing Developmental Evaluation Examples from Myanmar and Ethiopia Dr. Ann Doucette 15
Developmental Evaluation • Framework to support the understanding of programs and interventions that – Operate within complex systems and novel environments • PBEA environments are subject to change • PBEA results may be nonlinear – Focus on innovation and the processes and thinking used to support it – Have no clear cut trajectory to achieve desired outcomes – Emphasize strategy and results chains – Highlight learning to optimize outcome – Are willing to reflect on what is learned and to make changes to optimize results Ann M. Doucette, Ph. D 16
Traditional versus Developmental Evaluation Traditional a) Program/intervention testing (effectiveness, efficiency, impact) b) Evaluators are perceived as independent and objective c) Evaluation approach, design, methodology decided in advance of evaluation activities i. Predetermined d) Provides judgment on success of failure of a program/intervention e) Cause and effect modeling f) Focus on fidelity to program/intervention g) Program accountability emphasis is external to oversight and funders h) Evidence - significance Developmental a) Captures what is happening (supportive of change to program) b) Naturalistic inquiry – context specific – focus on learning c) Evaluators are perceived as collaborative – part of the team i. Flexible and adaptive d) Provides feedback on what’s working, for whom, under what conditions e) Systems thinking – patterns, interdependence, emergence f) Focus on context-specific events and how they shape the program g) Program accountability centers on commitment to optimize program (learn, use of feedback) h) Evidence – verified perspectives 17
DE Process Steps 1. 2. 3. 4. 5. 6. Agreement on PBEA programme area learning targets − Processes – impact pathways − Effectiveness − Impact Selected learning topic areas − Challenges − Other areas Identification of information needed to better understand learning target areas Collecting information − Validation of information collected Reflection on what has been learned, and how to use this information as feedback to the PBEA programme to optimize programme processes, outcomes and impact Sharing information, reflections, and information feedback strategies to appropriate stakeholder groups Examining the results of information feedback loops on PBEA optimization Ann M. Doucette, Ph. D 18
Developmental Evaluation Readiness • Is there sufficient receptivity and curiosity about new learning, some of which may be unanticipated – for example: – Is there time to reflect on what is being learned about the PBEA programme? – How do programme teams handle setbacks and failure? – How do teams handle, manage and use feedback? – Are teams willing to take risks? • Are there sufficient resources (time for collaboration and sharing) to support DE? • Is there an understanding of the need for participatory processes, or is there a need to alter or better manage power dynamics? • Are programme teams willing to adapt their structures (e. g. , rules, routines, procedures) to accommodate new ways of operation? • Are there issues that could interfere with the process (e. g. , unclear mandates, and unstable financial support of the program)? Ann M. Doucette, Ph. D 19
Building a DE Framework Defining individual and organizational roles and responsibilities Decisions to be made in terms of What learnings are desired Accountability regarding the collection of needed information at the programme and systems levels Roles and responsibilities regarding the communication and sharing of information, Governance in terms of the use and decisions to be made based on DE feedback information Maintenance and sustainability of DE process Establishing data/information integrity Verification process Safeguards against bias and inappropriate influence and scope creep Sustaining interest and commitment of participating stakeholders Ensuring sustained and sufficient resources – Funding – Reflection time – Commitment to incorporate DE feedback Ann M. Doucette, Ph. D 20
Peacebuilding • Absence / reduction of latent conflict drivers – Inequality/inequity – Resource distribution – Exclusion • Presence and capacity building – – Social bonds Trust Normative reciprocity Civic engagement Ann M. Doucette, Ph. D 21
Myanmar – PBEA • PBEA is not a stand alone programme – Integrated within education • PBEA is characterized by – Co-evolution and interdependence • change in one system initiates change in other systems; rumour can sometimes have more influence than programme staff, etc. ) – Emergence (PBEA agents interact within a peacebuilding network and in networks external to peacebuilding producing multiple sources of influence) – Patterns – predicting outcomes is challenging at best and environments in which PBEA reside are shifting, nonetheless patterns do emerge. These patterns can be verified in terms of output, outcome and impact Ann M. Doucette, Ph. D 22
Myanmar: DE Learning Objectives 1. What are the processes that are used to develop and support peacebuilding capacities? a) What assumptions were made at the beginning? b) What assurance is there that the definition of peacebuilding is shared across stakeholders? c) Given the flexibility of PBEA funding, peacebuilding efforts can be implemented and integrated in new and ongoing UNICEF efforts. Where are these efforts taking place? What has been learned – Selected results. . . • Flood response based on social cohesion not on cultural or religious characteristics of communities served • Peacebuilding Committee – integrated discussion across education, child protection, field services, C 4 C (communication for change • Sensitization training – participant focus on group rights social cohesion Ann M. Doucette, Ph. D 23
Myanmar: DE Learning Objectives 2. How is the UNICEF approach to PBEA in Myanmar different from a “business as usual” perspective? a) What adaptations or additions to existing UNICEF approaches were made to address PBEA objectives and peacebuilding capacity? What has been learned – Selected results. . . • Duplicate trainings for government teachers (Rakhine Buddhist) and volunteer teachers (majority Muslim) – assurance that IDP (internally displaced persons) children benefit from child-friendly, peacebuilding curricula • Flood response – change perception that UNICEF focuses only on IDP support and conflict environments -- change in community perceptions – opportunity to build social cohesion Ann M. Doucette, Ph. D 24
Myanmar: DE Learning Objectives (cont. ) 3. Peacebuilding is influenced by many systems (economic, political, education, health, safety and security, etc. ) a) How does the Myanmar UNICEF CO work in these systems directly or indirectly support PBEA capacities? b) How do existing Myanmar system protocols/practices constrain or enhance PBEA efforts? What has been learned – Selected results. . . • • • Higher enrollment and achievement scores in PBEA Townships Increased net enrollment in primary school Achievement scores have increased in math and language Release of National Curriculum – Commitment to “appreciate, maintain, and disseminate languages and literatures, cultures, arts and traditional customs of all national groups, ” a noted PBEA conflict driver – Intent to promote peacebuilding/social cohesion – Intent to raise awareness and appreciation of diversity – Intent to promote equality and inclusion Ann M. Doucette, Ph. D 25
Myanmar: DE Learning Objectives (cont. ) 4. What are the unique characteristics of initiating and operating PBEA efforts within Myanmar? What has been learned – Selected results. . . • National election – Country’s first election since a nominally civilian government was initiated in 2011 – Aung San Suu Kyi’s party has more the two thirds needed to choose Myanmar’s president, ending decades of military rule. • Finalization of National Ceasefire Agreement – Collective agreement • Armed groups had to previously sign a bilateral ceasefire with the government in order to participate in this collective agreement • Eight armed groups have signed • There are more combatant groups who have not signed (non-signing ethnic groups) – Many have independent ceasefire agreements with the government which is respected Ann M. Doucette, Ph. D 26
Ethiopia: DE Learning Objectives 1. Cultural issues around pastoral education: How adaptable is the current form of alternative basic education (ABE) to the pastoral livelihood? a) Is it flexible enough to adapt to the pastoral migrating patterns? b) If the current form of ABE is too inflexible to pastoral livelihoods, does it perpetuate negative feelings towards pastoralism by indirectly encouraging a more sedentary livelihood? c) How can PBEA support a more adaptable approach to ABE education that is more in line with the pastoral lifestyle? Ann M. Doucette, Ph. D 27
Ethiopia – ABE • • PBEA interventions have the potential to significantly affect the livelihoods and culture of pastoral communities through ABE programmes ABE is the first chance for many pastoral children have to receive any education – For many, gaining basic primary education is a first for their families – Parents engage in multi-ethnic center management committees (CMCs), learn about the importance of education and in some cases, initiate their own adult education programmes in ABE centers – Pastoral migration cards for children allow them to pursue education in nearby ABE centers as they migrate, • It is not clear to what extent these cards are in use, and working effectively • Education in the past was not a priority – Pastoral society highly values children for the work they do to contribute to family livelihoods • • The PBEA intervention provides pastoral communities are now engaging in primary education and tailoring their livelihoods to this educational system through ABEs A balance is needed to ensure that no harm is done to pastoral livelihoods, and such interventions are not reinforcing government resettlement objectives Ann M. Doucette, Ph. D 28
Ethiopia: DE Learning Objectives 2. Mainstreaming lessons learned and management decisions for PBEA a) Lessons from mainstreaming PBEA into the Education Section programme portfolio b) Managing the sensitivities around conflict for effective PBEA engagement 3. Taking pilot projects to scale 2. Finding value-added characteristics of PBEA for UNICEF over traditional approaches 3. Identifying best practices for PBEA replication and scale up 4. Establishing M&E practices for PBEA Ann M. Doucette, Ph. D 29
Summary Reflections • DE provides addresses the complexity of the PBEA initiative – While PBEA has defined objectives • the identification of conflict drivers characterizing the sites is diverse • the PBEA implementation approach taken must be sensitive to site-specific characteristics • Conflict fragile environments are complex – PBEA is delivered through educational initiatives • PBEA is interdependent with the education sector, as well as with political, policy and economic systems • Outcomes are a co-evolution of PBEA and educational results – While patterns emerge across PBEA programmes, different pathways leading to desired results will likely emerge • The expectancy for desired outcomes, for example, increased social cohesion, will take time Ann M. Doucette, Ph. D 30
Questions? Bosun Jang Education Specialist UNICEF-NYHQ bjang@unicef. org Dr. Ann Doucette Director The Evaluators’ Institute Claremont Graduate University ann. doucette 9@gmail. com
- Slides: 32