DEVELOPING A SYSTEMIC EVALUATION METHODOLOGY AN APPROACH TOWARDS

DEVELOPING A SYSTEMIC EVALUATION METHODOLOGY: AN APPROACH TOWARDS SOCIAL PROGRAM EVALUATION Maria Alejandra Torres Cuello & Luis Arturo Pinzon Salcedo. ma. torres 133@uniandes. edu. co, lpinzon@uniandes. edu. co Los Andes University. Bogota, Colombia.

CONTENT Systems Thinking Evaluation Systems concepts in evaluation Fourth Generation evaluation and boundary critique Case Study Context The Hermes Program Methodological design and application

SYSTEMS THINKING • • Systems Dynamics VSM Systems Engineering Systems Analysis 1 st wave 2 nd wave • Soft Systems Methodology • Interactive Planning • TSI • CSH • Systemic Intervention 3 rd wave Boundary Critique

BOUNDARY CRITIQUE “A systematic – reflective and discursive – effort of handling boundary judgements critically, whereby ‘critically’ means both ‘self-critically’ questioning one’s own claims and ‘thinking for oneself’ before adopting the claims of others”. (Ulrich & Reynolds, 2010)

Boundary critique and marginalization Secondary Boundary Wider system Not seen as pertinent Marginalized Elements Primary boundary Elements within the primary boundary Figure 1. Marginalization. Taken from: Systemic Intervention: philosophy, methodology and practice. Modified from Midgley (2000)

WHY IS IT IMPORTANT TO CONSIDER BOUNDARY CRITIQUE IN EVALUATION? Boundary judgments and boundary critique are very useful for incorporating a reflective component in evaluation. To reflect not only implies judging others´ work (constructions) but also on reflecting on own constructions. Critiquing is not directed towards criticizing or pointing out flaws of other people´s constructions. Instead, it is directed towards two different things: Not taking anything for granted, not even our own assumptions. Having the capacity of enriching other people´s views with our own perspective. It is a way of putting everyone's view at a same level detached from a role or a position. It is a way of handling power in relationships.

FOURTH GENERATION EVALUATION Is an evaluation approach developed by Guba & Lincoln consisting of 12 steps. Ø Works around the interests of the stakeholders and the formation of constructions. Ø Follows a cyclical process around hermeneutic/dialectic circles Ø Involves stakeholders in the different stages of the evaluation planning and implementation. Ø Seeks to empower stakeholders by addressing their claims, concerns, and issues. Ø Does not places the evaluator in a position of superior knowledge.

WHY FOURTH GENERATION EVALUATION? Its subjective character denies the subject/object duality. Stakeholders play an important role throughout the evaluation, more than just be seen as a source of information. Quantitative and qualitative tools are equally valid. This form of evaluation seeks to empower the stakeholders. In its methodological steps one can identify systems concepts in an implicit manner.

EVALUATION DESIGN 1. Establishing a contract with the client 2. Organizing to conduct the evaluation. 3. Identifying stakeholders 4. Reflecting upon the stakeholder´s selection. BOUNDARY CRITIQUE 5. Developing, testing and enlarging constructions 7. Reflecting upon the boundaries about “what” should be addressed by the evaluation. 8. Design of adequate tools for data collection. 9. Data Collection. 10. Analysis of results. 11. Discussion and sharing of results. 12. Generating recommendations. 13. Producing reports. 14. Recycling* LOGISTIC 6. Identifying and prioritizing relevant issues, claims, and concerns to be addressed.

EVALUATION DESIGN 1. Establishing a contract with the client • Establishing a relationship with the client. • Defining the purpose of the evaluation. 2. Organizing to conduct the evaluation • Raising awareness of the specific and general contexts surrounding the evaluand the evaluation. 3. Identifying stakeholders • Phase 1: Client-Evaluator • Phase 2: Other stakeholders Understanding roles, boundaries and interrelationships 4. Reflecting upon the stakeholder´s selection • Evaluating the necessity of reshaping the boundary. • Understanding the implications of the decision making process. 5. Developing, testing and enlarging constructions • Within single stakeholder groups and between stakeholders. 6. Identifying and prioritizing relevant issues, claims, and concerns to be addressed • Boundary critique by establishing a narrower boundary.

EVALUATION DESIGN 7. Reflecting upon the boundaries about “what” should be addressed by the evaluation • • • Should the evaluation keep the chosen issues, claims, and concerns? How is going to be the data collection? Does the evaluator has the adequate tools to tackle these? 8. Design of adequate tools for data collection 9. Data Collection 10. Analysis of results 11. Discussion and sharing of results • • Technical component: interpretation of results. Explanatory component: how results and real life situations are linked. 12. Generating recommendations 13. Producing reports 14. Recycling*

KEY COMPONENTS There are two key elements when conducting this type of evaluation: Identifying and selecting who will be part of the evaluation – the “who” component. Identifying the relevant issues, claims, and concerns upon which the evaluation will take place – the “what” component. Both processes take place through a dialogical/hermeneutic process or by means of a negotiation. I will argue that in most of the cases the dialogical/hermeneutic process shifts into a negotiation as those involved take a stand over a particular issue.

IDENTIFYING STAKEHOLDERS – THE “WHO” COMPONENT Takes place in two stages: Client-Evaluator Other selected stakeholders Both stages imply a process of boundary critique. Stage 1 Client and evaluator define separate initial boundaries. Following a dialogical process which may turn into a negotiation the final boundaries of Stage 1 are defined. In this case the boundary refers to those who will be considered as a fundamental part of the evaluation process.

IDENTIFYING STAKEHOLDERS – THE “WHO” COMPONENT – STAGE 1 Partially overlapped Completely overlapped Separated Dialogical process / Negotiation New boundary containing the identified and selected stakeholders – output from Stage 1 and input for stage 2.

IDENTIFYING STAKEHOLDERS – THE “WHO” COMPONENT Stage 2 The resulting boundary from stage 1 are the input boundaries for Stage 2. This boundary impacts the unfolding of two processes in this stage: In identifying which stakeholders will be part of the negotiation. In Setting the ground for the stakeholders to be able to determine the adequacy of the boundary proposed in stage 1. The resulting boundary of stage 2 might be wider, narrower, or the same as the one from stage 1. Both processes (Stage 1 and 2) will determine the way in which the entire methodological development will be conducted.

IDENTIFYING STAKEHOLDERS – THE “WHO” COMPONENT – STAGE 2 Same boundary as the one from stage 1 An expansion from the boundary from stage 1 A contraction of the boundary from stage 1 The processes conducted in stages 1 and 2 are always related as they are directed towards evaluating the same evaluand. A complete reframing of the original boundary should not be taking place.

REFLECTING UPON THE “WHO” COMPONENT Constitutes an independent but necessary process from the selection of the stakeholders. It implies a boundary critique process over the selected boundaries in previous stages. It is usually done by the evaluator or the evaluation team not by the stakeholders. The main purpose with which this is conducted is to determine the viability of the way in which the system of enquiry is being framed. Economic, time, social, etc. It is not a process directed towards good or wrong but towards what is seen as achievable in terms of design.

DEVELOPING CONSTRUCTIONS – THE “WHAT” COMPONENT This process is conducted by those who have been identified as relevant stakeholders for being part of the evaluation. Just as the selection of the “who” component, it is based on two stages: Within stakeholder groups Between stakeholders Both processes are conducted by means of a dialectic/hermeneutic circle which takes place in very similar ways although not identically. Boundary critique takes place in both cases, however the way in which this takes place differs from the traditional conception of boundary critique.

DEVELOPING CONSTRUCTIONS – THE “WHAT” COMPONENT Boundary critique emerges as stakeholders start involving in other stakeholders´ constructions as well as in other individuals from the same group of stakeholders. The fact that different individuals belong to the same group of stakeholders does not mean that a single perspective can be easily achieved. An individual´s perception is influenced by elements such as values, interests, ideas and others that makes it unique. In that sense the main goal is to achieve consensus. However, in order to do so, one needs to come across a process of critique in which different ways of bounding a situation (by means of constructions) are put under the scope by different individuals and perspectives. Achieving consensus does not mean that the way in which a stakeholder bounds a situation “wins” or is imposed to others. Instead, this means that by reflecting, boundaries that where once assumed could be dismissed, reshaped or kept.

PRIORITIZING ISSUES, CLAIMS AND CONCERNS Most of the time (although not always) there are too many issues, claims, and concerns to be included in a single evaluation process. In those cases prioritization needs to take place. By means of prioritizing issues, claims and concerns, a new boundary emerges. There will always me a marginalization process unless every single issue, claim and concern considered will be taken into account for the implementation of the evaluation. Initial boundary of issues, claims and concerns Narrower boundary with relevant issues, claims, and concerns for the evaluation

REFLECTING UPON THE “WHAT” COMPONENT Reflecting over the “what” component in a separate way of selecting relevant issues, claims, and concerns is important as well. How relevant is the consideration of these issues, claims, and concerns? Do we have the adequate means and resources to tackle them? Are there any constraints for collecting information regarding these issues, claims and concerns? What are the implications of selecting these and not others?

BOUNDARY CRITIQUE IN FOURTH GENERATION EVALUATION Every step between the stages 3 and 11 of the reframed Fourth Generation evaluation implies the use of boundary critique whether this is done in an explicit or implicit manner. Two of these steps are conducted as an implicit way of introducing the concepts of boundary judgments, boundary critique and marginalization in this form of evaluation. The usefulness of boundary critique in evaluation depends on the way in which it is incorporated in the methodology at a practical level.

CASE STUDY

CONTEXT

COLOMBIA Population (2016): 48. 747. 708 (DANE, 2016). Poverty (2016): 28% (DANE, 2016) Inequality rate: 0. 517 (DANE, 2016) Displaced population: 6. 900. 000 (UNHCR, 2016) World Atlas (2016). Retrieved from: http: //www. worldatlas. com/webimage/countrys/samerica/co. htm

BOGOTÁ Population: 8. 000 (DANE, 2016) Inequality: 0. 492 (El Tiempo, 2016). 817. 000 people (10. 4%) lives with less than 241. 130 COP (60£) per month (El Tiempo, 2016). 88% of the population lives in levels 1, 2 or 3 of stratification (El Tiempo, 2016).

BULLYING According to a research developed by the Universidad de los Andes and the Instituto Nacional de Medicina Legal y Ciencias Forenses (National Institute of Forensic Medicine and Forensic Sciences), "One in five children is a victim of bullying, one in four children consume drugs, one minor commits suicide in the country every 48 hours"

WHAT IS THE HERMES PROGRAM? “It is a program created to transform school conflicts through the use of Alternative Conflict Resolution Techniques and the prevention of bullying in schools in Bogota and the region surrounding it. ” (Cámara de Comercio de Bogotá, 2017)

WHO IS GOING TO BE INCLUDED IN THE EVALUATION ? WHAT IS GOING TO BE EVALUATED ?

WHO? Teachers Parents Students Former Students Consultants Administrative Staff

WHAT? CCB goal IMPACT (How do students approach conflicts? ) Evaluators: Which are the variables that might affect the real impact and the performance of the program? How do we recognize them? Involving the community!

WORKSHOP Participants: parents, children, teachers, schools' administrative staff, Chamber of Commerce consultants. Three main goals with the activities: Perception Past change Future change (asking them to propose their own ideas).

WORKSHOP STRUCTURE Activity 1: Customized use of Critical Systems Heuristics (CSH) questions Activity 2: Confidence activity Activity 3: Drawing the community Activity 4: Past change stories Activity 5: 6 Thinking hats GROUP ACTIVITIES

CONSTRAINTS FOR DEVELOPING THE WORKSHOP Limited time frame Intensive session Differences in age and levels of education The presence of power related to the position of each of the stakeholders in the program

ACTIVITY 1 - CSH QUESTIONNAIRE Questionnaire characteristics: Customization of the questions as a whole (grammatical structure and vocabulary) Customization for each group of stakeholders Individually solved Some of the questions were oriented towards uncovering perceptions regarding: Motivation Perceived commitment Key variables for the program´s success Influence of social, political and economic variables in the program´s performance Impact

ACTIVITY 2 – “CRAZY TRAINS” Goal: improve confidence among each group of participants Characteristics of each group: Included 5 to 6 individuals with different roles within the programme. Wide spectrum of ages Power related to different positions

ACTIVITY 3 – DRAWING THE COMMUNITY Goal: How did the different stakeholders perceive the community of the program? How were the perceived relationships? How did they include? Shared view of the program Socialization of the drawing and further discussion

ACTIVITY 4 – SUCCESS STORIES Goal: Significant stories that participants of the workshop could identify as successful. Different levels of impact (individual, family, community).

ACTIVITY 5 – 6 THINKING HATS Goal: give the participants the possibility to propose their own ideas for future change. Ideas centered not individual centered. 6 hats with different colors each focusing on a particular way of viewing the idea

WORKSHOP RESULTS Motivation Program recognizing each individual as important The commitment of some of the members of the community was perceived as being lower than the own. Too many burocratic procedures

SURVEY Students survey Number of surveys: 1068 Number of schools: 30 Number of questions: 12 Main topics of the survey: • Demographic information • Perceived commitment of other stakeholders • Conflict frequency and management (victim/abuser/observer) • Most important and less important aspects of the programme for the students • Overall satisfaction

SURVEY Consultants survey: Number of surveys: 40 (85% of the consultants) Number of questions: 20 (multiple choice and open questions) Topics of the survey: ü Demographic questions (age, gender, level and field of studies, time they have been working for the program). ü Communication. ü Commitment and factors that affect the commitment of stakeholders. ü Improvement of relations between stakeholders. ü Success variables ü Consultant career development ü Overall satisfaction

References ACNUR (2016). La agencia de la ONU para los refugiados. Retrieved from: http: //www. acnur. org/recursos/estadisticas/ DANE (2016). Pobreza monetaria y multidimensional en Colombia 2016. Retrieved from: https: //www. dane. gov. co/index. php/estadisticas-por-tema/pobreza-ycondiciones-de-vida/pobreza-y-desigualdad/pobreza-monetaria-ymultidimensional-en-colombia-2016 Cámara de Comercio de Bogota (2016). Retrieved from: http: //www. centroarbitrajeconciliacion. com/en/Services/Studentcoexistence/What-is-the-Hermes-Program Midgley, G. (2000). Systemic Intervention. Philosophy, methodology and practice. New York: Springer. World Atlas (2016). Retrieved from: http: //www. worldatlas. com/webimage/countrys/samerica/co. htm

Thank you!

STUDENTS SURVEY RESULTS Student´s behavior when witnessing conflicting situations between other individuals 1. I do not do anything, it is not of my business. 2. I express my disagreement but I do not get involved in the conflict. 3. I provoke dialogue between them. 4. I confront them and I fight with them. 5. I help the one I like the most. 6. I tell what is happening to the teacher

PERCEIVED LEVELS OF COMMITMENT • Principal and administrative Staff • Teachers

CONSULTANTS’ SURVEY RESULTS • Perceived commitment of the different actors of the program

• Improvement of relationships among different actors THE RELATIONSHIP BETWEEN STUDENTS AND CONSULTANTS IS THE ONE PERCEIVED AS HAVING THE MOST SIGNIFICANT CHANGES FOR THE BETTER, FOLLOWED BY THE RELATIONSHIP AMONG STUDENTS.