Outcome Mapping MasterClass Outcome Harvesting a tool for

Outcome Mapping Master-Class Outcome Harvesting: a tool for identifying and understanding results in complex circumstances Ricardo Wilson-Grau Brussels, Belgium 25 March 2013

Introductions Please mention your name and organisation and briefly describe in a minute or two what is your experience in outcome monitoring and evaluation. ricardo. wilson-grau@inter. nl. net

Purpose and intended results 1. You will learn how to apply the essential principles of the six steps of Outcome Harvesting. 2. At the end of the day, you will be able to design a monitoring process and formative or summative evaluations using the tool. ricardo. wilson-grau@inter. nl. net

Methodology Personal reflection Conceptual presentation Questions & Answers Exercise with case ricardo. wilson-grau@inter. nl. net

✓ 9: 15 9: 30 10: 45 11: 00 12: 00 13: 00 14: 00 15: 30 15: 45 16: 30 16: 45 Agenda Introductions The complex challenges of monitoring and evaluating outcomes Coffee Primary users and principal uses Useable Outcome Harvesting questions Lunch Identifying and formulating outcomes Coffee Substantiation, analysis, interpretation and use Summing up Evaluation ricardo. wilson-grau@inter. nl. net

The complex challenges of monitoring and evaluating outcomes

Outcome Harvesting Inspired by the Outcome Mapping methodology. A tool for practitioners operating in situations of considerable uncertainty to monitor and evaluate the social change results they are achieving. ricardo. wilson-grau@inter. nl. net

Outcome Harvesting Developed since 2003 by me and my colleagues Barbara Klugman, Claudia Fontes, Fe Briones Garcia, Gabriela Sánchez, Goele Scheers, Heather Britt, Jennifer Vincent, Julie Lafreniere, Juliette Majot, Marcie Mersky, Martha Nuñez, Mary Jane Real, Natalia Ortiz and Wolfgang Richert. ricardo. wilson-grau@inter. nl. net

1. International social change networks AL IP OB GL ERSH RTN HE PA OR T N OF O F NTI D E EVE PR ARM ICT NFL O C ricardo. wilson-grau@inter. nl. net

International development funders ricardo. wilson-grau@inter. nl. net

• Over 300 networks and associations, NGOs, communitybased organisations, research institutes, and government agencies on all seven continents. • Identifying and reporting 2, 500+ changes they have influenced. ricardo. wilson-grau@inter. nl. net

What did all of these organisations have in common?

Planning ricardo. wilson-grau@inter. nl. net

Planning Monitoring ricardo. wilson-grau@inter. nl. net

Why? COMPLEXITY!

Personal reflection In one minute each, explain to the person next to you what you understand as “complexity”. Please hold your reflections and allow me to explain how complexity is understood in Outcome Harvesting. Then, we will close the circle with the Questions & Answers. ricardo. wilson-grau@inter. nl. net

Simple Complex Relationships of cause and effect are KNOWN Relationships of cause and effect are UNKNOWN ricardo. wilson-grau@inter. nl. net

Simple intervention A polio vaccination campaign IMPACT OUTCOMES OUTPUTS ACTIVITIES INPUTS Time ricardo. wilson-grau@inter. nl. net Inspired by Jeff Conklin, cognexus. org

Simple intervention A polio vaccination campaign IMPACT OUTCOMES t? n ie c e w e i f f e OUTPUTS ? e iv t c fe Ar ACTIVITIES f e e w e r A ¿ INPUTS Time ricardo. wilson-grau@inter. nl. net Inspired by Jeff Conklin, cognexus. org

When you can plan with relative certainty, conventional evaluation works Ten Steps to Designing, Building and Sustaining a Results. Based Monitoring and Evaluation System Conducting a readiness assessment 1 Planning for improvement — selecting results targets Selecting key indicators to monitor outcomes 2 Agreeing on outcomes to monitor and evaluate 3 4 Baseline data on indicators 5 Using your findings Using evaluation information 6 7 Monitoring for results ricardo. wilson-grau@inter. nl. net 8 Reporting your findings 9 10 Sustaining the M&E system within your organization

But in complex interventions OUTPUT OME OUTCOME OUTPUT ACTIVITY OUTCOME Implementing a new E OM C T U to address. O UT OUTPapproach domestic violence OUTPUT ACTIVITY OUTCOME INPUTS OU TC ACTIVITY INPUTS ACTIVITY OUTPUT Time INPUTS ricardo. wilson-grau@inter. nl. net OM E

Not black and white SIMPLE OUTPUT Dimensions in which the relationships of cause and effect are known E OM OUTCOME OUTPUT ACTIVITY OUTCOME ME CO OUTPUT ACTIVITY OUTCOME INPUTS OU TC ACTIVITY INPUTS ACTIVITY OM E OUTPUT Time INPUTS ricardo. wilson-grau@inter. nl. net 22

Not black and white OUTPUT COMPLEX E OM OUTC Unknown relations OUTPUT of cause and effect dominate. ACTIVITY OUTCOME ME CO OUTPUT Results are ACTIVITY substantially INPUTS unforeseeable. ACTIVITY INPUTS OUTCOME OU TC ACTIVITY OM E OUTPUT Time INPUTS ricardo. wilson-grau@inter. nl. net 23

ricardo. wilson-grau@inter. nl. net Thanks to Michael Quinn Patton

Exercise #1 ➤ Now we will begin to use the simulated case. The idea is for you to imagine that Fund. Now described in this page and a half profile is an NGO in your country. You have the power to decide what to do in order to achieve its multi-annual objectives. ➤ First, however, you want to assess the degree of complexity you face. So please classify on this sheet each objective on a scale from 1 for very certain (or simple) to 7 for highly uncertain (or complex) ➤ It is a matter of judgement. There are really no right or wrong answers. ➤ And only rank the objectives for which you feel comfortable. ricardo. wilson-grau@inter. nl. net

ricardo. wilson-grau@inter. nl. net

Evaluation and complexity Factors of Complexity High COMPLEXIMETER Disagreement Uncertainty About what is the About what will be the development challenge results of your actions or its solution to solve the development challenge Low ricardo. wilson-grau@inter. nl. net 27

PME and complexity in time Evaluation COMPLEXIMETER Monitoring Planning E M TI ricardo. wilson-grau@inter. nl. net 28

COMPLEXIMETER PME and complexity in time Monitoring E M TI Planning ricardo. wilson-grau@inter. nl. net Evaluation

COMPLEXIMETER PME and complexity in time Monitoring E M TI Planning ricardo. wilson-grau@inter. nl. net Evaluation

COMPLEXIMETER PME and complexity in time Monitoring Evaluation E TIM Planning ricardo. wilson-grau@inter. nl. net 31

In sum The greater the complexity the less important is planning and the more vital it is to monitor and evaluate in real time, in order to know what you are achieving, what is working and what you should do differently. ricardo. wilson-grau@inter. nl. net

Outcome Harvesting is NOT useful IMPACT OUTCOMES OUTPUTS ACTIVITIES INPUTS Time ricardo. wilson-grau@inter. nl. net

Outcome Harvesting IS useful OUTPUT OME OUTCOME OUTPUT ACTIVITY OUTCOME ME CO OUTPUT ACTIVITY OUTCOME INPUTS OU TC ACTIVITY INPUTS ACTIVITY OUTPUT Time INPUTS ricardo. wilson-grau@inter. nl. net OM E

Questions & Answers How compatible or incompatible is your understanding of complexity now with what I have explained of complexity for Outcome Harvesting? Any other questions about the complex challenges of monitoring and evaluating outcomes? ricardo. wilson-grau@inter. nl. net

✓ 9: 15 ✓ 9: 30 ✓ 10: 45 11: 00 12: 00 13: 00 14: 00 15: 30 15: 45 16: 30 16: 45 Agenda Introductions The complex challenges of monitoring and evaluating outcomes Coffee Primary users and principal uses Useable Outcome Harvesting questions Lunch Identifying and formulating outcomes Coffee Substantiation, analysis, interpretation and use Summing up Evaluation ricardo. wilson-grau@inter. nl. net

Outcome Harvesting Six steps: 1. Design the harvest 2. Review documentation and draft outcomes 3. Engage with informants 4. Substantiate 5. Analyse, interpret 6. Support use of findings ricardo. wilson-grau@inter. nl. net

The essence I A. Outcome defined as a change in a social actor Individual SOCIAL ACTOR Institution Group or community Organisation ricardo. wilson-grau@inter. nl. net

The essence I A. Outcome defined as a change in a social actor Behaviour Individual SOCIAL ACTOR Institution Relationships Group or community CHANGING Organisation Policies and practices ricardo. wilson-grau@inter. nl. net Actions, activities

The essence II B. Contribution: Effect Detective Cause Epidemiologist ricardo. wilson-grau@inter. nl. net Auto mechanic Arqueologist

For M&E, Outcome Harvesting represents a paradigm change From focusing on what one does ricardo. wilson-grau@inter. nl. net

For M&E, Outcome Harvesting represents a paradigm change To focusing on what one achieves ricardo. wilson-grau@inter. nl. net

Outcome Harvesting Six steps: 1. Design the harvest 2. Review documentation and draft outcomes 3. Engage with informants 4. Substantiate 5. Analyse, interpret 6. Support use of findings ricardo. wilson-grau@inter. nl. net

Personal reflection Always assuming that Fund. Now is an organisation in your country, write down who you consider to be its stakeholders. We will come back to your list in the exercise. ricardo. wilson-grau@inter. nl. net

1. Design the harvest to meet needs It is tempting to collect data on everything a project, programme or organisations does and achieves, but it is not practical. “At IDRC, if you cannot identify and articulate the primary intended users and uses of the evaluation you should not conduct the evaluation. Unused evaluation is a waste of precious human and financial resources. ” ricardo. wilson-grau@inter. nl. net

Primary intended users Principal intended uses ricardo. wilson-grau@inter. nl. net

Primary intended users Those people who need the harvest or its findings in order to make decisions or take actions. The users are actively involved. The harvester and her commissioner agree together who are the primary users, and then what are their principal uses. ricardo. wilson-grau@inter. nl. net

Principal uses There can be use for the process itself of harvesting As well as for the findings of the harvest. ricardo. wilson-grau@inter. nl. net

Typical process uses • Improve communication and foment common understanding • Strengthen the implementation of a project or programme because “what is measured tends to be what is gets done” • Increase participation and ownership • Foster a learning culture ricardo. wilson-grau@inter. nl. net

Common uses of findings • • Inform what was done and is being achieved Take decisions to modify the model or initiative Plan future work Evaluate the performance ricardo. wilson-grau@inter. nl. net

Users vs. Audience Those people who need the harvest or its findings in order to make decisions or take actions. The users are actively involved. Users are different from the broader audience of stakeholders who may see and react to the harvest’s results. The audience is passively interested in the harvest. ricardo. wilson-grau@inter. nl. net

Exercise #2 With the same person with whom you had just had the personal reflection, exemplify whom you both consider would be Fund. Now’s users of an Outcome Harvest and what might be their uses? Who would be the audience? ricardo. wilson-grau@inter. nl. net

Fund. Now Users vs. Audience Primary users: They will be involved in decision-making through all six steps of the harvest Audience: They may be informed periodically of progress but not necessarily. They may be consulted on how to use the findings but they are not responsible for the use. ricardo. wilson-grau@inter. nl. net

Questions & Answers Do you have any questions about primary users and principal uses of an Outcome Harvest and why it is important to distinguish them from the broader audience for the harvest findings and other stakeholders? ricardo. wilson-grau@inter. nl. net

✓ 9: 15 ✓ 9: 30 ✓ 10: 45 ✓ 11: 00 12: 00 13: 00 14: 00 15: 30 15: 45 16: 30 16: 45 Agenda Introductions The complex challenges of monitoring and evaluating outcomes Coffee Primary users and principal uses Useable Outcome Harvesting questions Lunch Identifying and formulating outcomes Coffee Substantiation, analysis, interpretation and use Summing up Evaluation ricardo. wilson-grau@inter. nl. net

Personal reflection If you were to evaluate Fund. Now, in a sentence write one evaluation question that you would consider important for the use(s) of the primary user(s). You will now use your question either to further craft it, or completely re-write it, as we see the purpose of useable questions in Outcome Harvesting. ricardo. wilson-grau@inter. nl. net

Primary intended users Principal intended uses Useable Outcome Harvesting questions ricardo. wilson-grau@inter. nl. net

Relevant Useable harvest questions Evaluation findings Priority Credible Adapted from: A Practical Guide for Engaging Stakeholders in Developing Evaluation Questions, Robert Wood Johnson Foundation, 2009 ricardo. wilson-grau@inter. nl. net

Useable harvest questions What questions must be answered so that the primary users will be able to make decisions or take other action? For a harvest question to be useable, it must provide both : Information: What data has to be collected? Plus Interpretation: So what does the data mean? ricardo. wilson-grau@inter. nl. net

Example of a What question What were Fund. Now’s outcomes in 2011 -12? Fantastic. What does it mean? We have 12 outcomes in two years! ricardo. wilson-grau@inter. nl. net

Example of a So What question What were Fund. Now’s outcomes in 2011 -12? What is the relative importance of Fund. Now’s outcomes in 2011 -12 for increasing the quantity and quality of the work of women’s organisations in the country? A useable question is an evaluative question ricardo. wilson-grau@inter. nl. net

Exercise #3 In groups of twos, agree a different, useable outcome harvest question for Fund. Now, always in one sentence. ricardo. wilson-grau@inter. nl. net

Actionable (useable) evaluation questions 1. Was the programme—and is it still—needed? How well does it address the most important root causes? Is it still the right solution? 2. How well designed and implemented is the programme? 3. How valuable are the outcomes for [insert impactee group]? 4. What works best for whom, under what conditions, and why/how? 5. How worthwhile was it overall? Or, which parts or aspects of the programme generated the most valuable outcomes for the time, money and effort invested? 6. How sustainable is the impact? How sustainable is the programme itself? Source: Jane Davidson, Actionable Evaluation Basics, Real. Evaluation. com, 2012 ricardo. wilson-grau@inter. nl. net

Questions & Answers Do you have any questions about formulating useable “What? ” PLUS “So What? ” Outcome Harvesting questions? ricardo. wilson-grau@inter. nl. net

✓ 9: 15 ✓ 9: 30 ✓ 10: 45 ✓ 11: 00 ✓ 12: 00 ✓ 13: 00 14: 00 15: 30 15: 45 16: 30 16: 45 Agenda Introductions The complex challenges of monitoring and evaluating outcomes Coffee Primary users and principal uses Useable Outcome Harvesting questions Lunch Identifying and formulating outcomes Coffee Substantiation, analysis, interpretation and use Summing up Evaluation ricardo. wilson-grau@inter. nl. net

Outcome Harvesting Six steps: 1. Design the harvest 2. Review documentation and draft outcomes 3. Engage with informants 4. Substantiate 5. Analyse, interpret 6. Support use of findings ricardo. wilson-grau@inter. nl. net

Personal reflection For all of you, what makes monitoring and evaluation data credible? I will make a list here on the flip chart. ricardo. wilson-grau@inter. nl. net

Primary intended users Principal intended uses Useful Outcome Harvesting questions Credible data ricardo. wilson-grau@inter. nl. net

Select the most credible sources People who personally know who changed and what changed. They give initial authenticity, confidence and credibility to the information you harvest. Thus, Outcome Harvesting is highly participatory. ricardo. wilson-grau@inter. nl. net

Triangulation of sources Historical documentation SOURCES Independent third parties ricardo. wilson-grau@inter. nl. net Person(s) who influenced others in time

Exercise #4 Please take five minutes to read this 2012 annual report from Fund. Now. Then, with the person seated next to you: A. Identify one outcome: • Who changed behaviour, relationships, activities, actions, policies or practice? • What changed? • When did it change? • And where? ricardo. wilson-grau@inter. nl. net

Fund. Now 2012 Annual Report Feedback from our allies and co-workers about Fund. Now’s activities in 2012 has been very positive, emphasizing the usefulness of our research publications to inform about the funding challenges and opportunities facing women's organizations in the country. Our research has shown that women's rights organizations have been living in a form of "resist and survive" mode but doing amazing work for the advancement of women's rights. People always mention our online newsletter Fund. Now as a tool that has helped them to think more strategically about raising funds for their organizations. The preparatory research we did for each one of the 13 meetings we convened was greatly appreciated. This year, our innovative research on trends in financing for gender equality and women's organization has been a strongly supportive tool for women’s groups obtaining more and better resources across the country. Based on the collection of "gems" of impact of projects implemented by the grant recipients of central government’s National Fund for MDG 3 (referring to the third Millennium Development Goal which is on gender equality), we compiled a report that has been used by gender activists. For example, donors internally to promote resource allocation for gender equality and women's rights. For example, an ad hoc group in the ministry responsible for the National Fund for MDG 3 used our compilation in their lobbying internally for the replenishment of the Fund, which resulted in the addition of US$4 million to the Fund in early 2012 and a commitment to launch a new phase of the Fund with US$20 million in 2013. In addition, this year we have successfully organized in two provinces six strategic meetings for women's organizations on provincial resource mobilization. (In addition, we organised t nationally for a total of 13. ) These were the first such meetings in those provinces. In one province 5 funders and 18 women's organizations attended and in the other 4 and 12, respectively. The background research we did based on regional surveys in preparation for the meetings was very well received and participants said we were filling a clear gap of information with our findings. Many participants expressed that these meetings are a strategic opportunity to establish new contacts, engage in networking, develop strategies, offer a space for potential donors and grantees to meet and share experiences. Everyone said the meetings enabled them to better understand the context in which both funders and women's organizations operate. The meeting participants also had time to reflect on their work and said these events served as inspiration. All the participants expressed their commitment to work together in the future to guarantee funding for women's organizations in the two provinces. ricardo. wilson-grau@inter. nl. net

MGD 3 Fund outcome A. Description: During 2011, the ministry decided to allocate US$4 million to the National Fund for MDG 3 in early 2012 and committed to launch a new phase of the Fund with US$20 million in 2013, to strengthen the rights and opportunities for women and girls. WHO? ricardo. wilson-grau@inter. nl. net

MGD 3 Fund outcome A. Description: During 2011, the ministry decided to allocate US$4 million to the National Fund for MDG 3 in early 2012 and committed to launch a new phase of the Fund with US$20 million in 2013, to strengthen the rights and opportunities for women and girls. WHAT? ricardo. wilson-grau@inter. nl. net

MGD 3 Fund outcome A. Description: During 2011, the ministry decided to allocate US$4 million to the National Fund for MDG 3 in early 2012 and committed to launch a new phase of the Fund with US$20 million in 2013, to strengthen the rights and opportunities for women and girls. WHAT? ricardo. wilson-grau@inter. nl. net

MGD 3 Fund outcome A. Description: During 2011, the ministry decided to allocate US$4 million to the National Fund for MDG 3 in early 2012 and committed to launch a new phase of the Fund with US$20 million in 2013, to strengthen the rights and opportunities for women and girls. WHEN? ricardo. wilson-grau@inter. nl. net

MGD 3 Fund outcome A. Description: During 2011, the ministry decided to allocate US$4 million to the National Fund for MDG 3 in early 2012 and committed to launch a new phase of the Fund with US$20 million in 2013, to strengthen the rights and opportunities for women and girls. WHERE? ricardo. wilson-grau@inter. nl. net

Exercise #5 B. How did Fund. Now contribute to the ministry deciding to allocate US$4 million to the National Fund for MDG 3 in early 2012 and committed to launch a new phase of the Fund with US$20 million in 2013? What activities and outputs plausibly contributed to the change in the social actor, however partially, indirectly and even unintentionally? ricardo. wilson-grau@inter. nl. net

MGD 3 Fund outcome A. Description: During 2011, the ministry decided to allocate US$4 million to the National Fund for MDG 3 in early 2012 and committed to launch a new phase of the Fund with US$20 million in 2013, to strengthen the rights and opportunities for women and girls. B. Contribution: Fund. Now compiled research that was used by an ad hoc group within the ministry for internal lobbying for the replenishment of the MDG 3 Fund. Plausible? ricardo. wilson-grau@inter. nl. net

Other information? • Significance of the outcome • Collaboration with other social actors • Contribution of other actors and factors • History • Context • Evidence of impact on people’s lives And so forth. ricardo. wilson-grau@inter. nl. net

Harvesting outcomes Less is more! ricardo. wilson-grau@inter. nl. net en? h W o? h W at? h W ? e r e Wh 81

Outcome Harvesting Six steps: 1. Design the harvest 2. Review documentation and draft outcomes 3. Engage with informants 4. Substantiate 5. Analyse, interpret 6. Support use of findings ricardo. wilson-grau@inter. nl. net

Triangulation of sources Historical documentation SOURCES Independent third parties ricardo. wilson-grau@inter. nl. net Person(s) who influenced others in time

Triangulation of methods METHODS ricardo. wilson-grau@inter. nl. net

Description: During 2011, the ministry decided to allocate US$4 million to the National Fund for MDG 3 in early 2012 and committed to launch a new phase of the Fund with US$20 million in 2013, to strengthen the rights and opportunities for women and girls. Comment [RW-G 1]: Can you be more specific about the date — when in 2011, which month and day if possible? Comment [RW-G 2]: Can you be more concrete? Which ministry? Who in the ministry? Comment [RW-G 3]: Is this an appropriate characterisation of the purpose? Can you make it more measurable? ricardo. wilson-grau@inter. nl. net

In sum The description of the outcome and the contribution have to be sufficiently specific and concrete so that they are: • Plausible • Verifiable • Detailed enough to meet the primary users’ principal uses. ricardo. wilson-grau@inter. nl. net

Questions & Answers Do you have any questions about the content of a description of an outcome and the contribution to it? ricardo. wilson-grau@inter. nl. net

✓ 9: 15 ✓ 9: 30 ✓ 10: 45 ✓ 11: 00 ✓ 12: 00 ✓ 13: 00 ✓ 14: 00 ✓ 15: 30 15: 45 16: 30 16: 45 Agenda Introductions The complex challenges of monitoring and evaluating outcomes Coffee Primary users and principal uses Useable Outcome Harvesting questions Lunch Identifying and formulating outcomes Coffee Substantiation, analysis, interpretation and use Summing up Evaluation ricardo. wilson-grau@inter. nl. net

Outcome Harvesting Six steps: 1. Design the harvest 2. Review documentation and draft outcomes 3. Engage with informants 4. Substantiate 5. Analyse, interpret 6. Support use of findings ricardo. wilson-grau@inter. nl. net

4. Substantiate: Verify accuracy + deepen understanding Outcome: To what degree are you in agreement with the description of the ministry’s decision to expand its funding of MDG 3? [ ] Fully agree [ ] Partially agree [ ] Disagree Comments if you like: Which outcomes and how many to substantiate depends on what is required for the credibility of the findings. ricardo. wilson-grau@inter. nl. net

5. Analyse and interpret the outcomes to answer the useable Outcome Harvesting questions ricardo. wilson-grau@inter. nl. net

Triangulation of analysis Methodologist ANALYSTS Person(s) who influenced others in time 1 or 2 content experts ricardo. wilson-grau@inter. nl. net

Excel Database ricardo. wilson-grau@inter. nl. net

Access Database ricardo. wilson-grau@inter. nl. net

Seeking processes, patterns, trends in what happened ricardo. wilson-grau@inter. nl. net

Women’s Land Inheritance Rights in an African Country Agriculture committee discusses draft law Women’s group launches campaign President declares women's land inheritance a right Court decides against widow Women’s group initiates legal aid for widows Policy change Legislature passes women’s right to land inheritance law Court decides landmark case in favour of widow Ten widows take late husbands’ family to court Late husbands’ families murder three widows Widow takes late husband’s family to court Legislator drafts law Finance committee discusses draft law For the first time, press runs story on women’s land inheritance YEARS 1 -3 Land Commission approves regulations for women’s inheritance of deceased husband’s land YEARS 4 -6 ricardo. wilson-grau@inter. nl. net Religious leader declares women's land inheritance a right Practice change Over the past 5 years, in 90% of the cases the Land Commission has given to widows the titles of their late husbands’ land Agricultural union admits widows as members YEARS 7 -10

So what does it all add up to? ricardo. wilson-grau@inter. nl. net

Theory of change MISSION IMPACT ASSUMPTIONS RESULTS STRATEGIES SOCIAL ACTORS ricardo. wilson-grau@inter. nl. net

Systems approach ricardo. wilson-grau@inter. nl. net Peter Woodrow, CDA

6. Support for use of findings ricardo. wilson-grau@inter. nl. net

Questions & Answers Do you have any questions about the three last steps? 4. Substantiate 5. Analyse, interpret 6. Support use of findings ricardo. wilson-grau@inter. nl. net

✓ 9: 15 ✓ 9: 30 ✓ 10: 45 ✓ 11: 00 ✓ 12: 00 ✓ 13: 00 ✓ 14: 00 ✓ 15: 30 ✓ 15: 45 16: 30 16: 45 Agenda Introductions The complex challenges of monitoring and evaluating outcomes Coffee Primary users and principal uses Useable Outcome Harvesting questions Lunch Identifying and formulating outcomes Coffee Substantiation, analysis, interpretation and use Summing up Evaluation ricardo. wilson-grau@inter. nl. net

Methodological challenges • • • Participation Focusing on what one achieved Expressing oneself in writing Coaching Everyone keeping to the deadlines Working as an archaeologist or forensic scientist or police detective Recognising what did not change as an outcome Not all outcomes are positive Harvesting best done by two or more people Supporting use of findings ricardo. wilson-grau@inter. nl. net

Final Questions and Answers? ricardo. wilson-grau@inter. nl. net

✓ 9: 15 ✓ 9: 30 ✓ 10: 45 ✓ 11: 00 ✓ 12: 00 ✓ 13: 00 ✓ 14: 00 ✓ 15: 30 ✓ 15: 45 ✓ 16: 30 16: 45 Agenda Introductions The complex challenges of monitoring and evaluating outcomes Coffee Primary users and principal uses Useable Outcome Harvesting questions Lunch Identifying and formulating outcomes Coffee Substantiation, analysis, interpretation and use Summing up Evaluation ricardo. wilson-grau@inter. nl. net

Evaluation of workshop I would like to ask you to take a few minutes to reflect on the extent to which we accomplished the intended results. Please make suggestions on what I can do differently to improve this workshop. Also, is there anything else that you believe I may be able to do to support you? Ricardo. Wilson-Grau@inter. nl. net ricardo. wilson-grau@inter. nl. net

Many thanks! Ricardo. Wilson-Grau@inter. nl. net