Outcome Harvesting A tool for identifying and understanding

Outcome Harvesting A tool for identifying and understanding results in complex circumstances Ricardo Wilson-Grau

Agenda • • • Outcome Harvesting’s origins When is Outcome Harvesting a useful tool How does the tool work — exercise Six steps Methodological challenges

Origins

e Harvesting Outcome Harvesting Inspired by the Outcome Mapping methodology. A tool for practitioners operating in dynamic, uncertain situations to monitor and evaluate the social change results they are achieving.

Outcome Harvesting Developed since 2003 by me and my colleagues Barbara Klugman, Claudia Fontes, Fe Briones Garcia, Gabriela Sánchez, Goele Scheers, Heather Britt, Jennifer Vincent, Julie Lafreniere, Juliette Majot, Marcie Mersky, Martha Nuñez, Mary Jane Real, Natalia Ortiz and Wolfgang Richert. Ricardo. Wilson-Grau@inter. nl. net

1. International social change networks AL IP OB GL ERSH RTN HE PA OR T ION F ENT ED EV PR ARM T OF FLIC N CO Ricardo. Wilson-Grau@inter. nl. net

2. International development donors Ricardo. Wilson-Grau@inter. nl. net

What did all of these organisations have in common?

Monitoring Planning Ricardo. Wilson-Grau@inter. nl. net

Monitoring Evaluation “I can honestly say that not a day goes by when we don’t use those evaluations in one way or another. ” Ricardo. Wilson-Grau@inter. nl. net

Why? COMPLEXITY!

Simple Relationships of cause and effect are KNOWN 12

M&E of a simple intervention Annual polio vaccination campaign IMPACT OUTCOMES OUTPUTS ACTIVITIES INPUTS Time Inspired by Jeff Conklin, cognexus. org

M&E of a simple intervention Annual polio vaccination campaign IMPACT OUTCOMES ? nt e ci i f f e r A e e w OUTPUTS ACTIVITIES INPUTS Time e w e e? r ¿A ctiv e f f e

Complex Relationships of cause and effect are UNKNOWN

M&E of a complex intervention OUTPUT Strengthening nonviolent OME C T U O responses to communal conflict in the Horn of Africa OUTCOME OUTPUT ACTIVITY OUTCOME INPUTS COM OUTPUT ACTIVITY OUTCOME OU TC ACTIVITY INPUTS ACTIVITY Time INPUTS OUTPUT OM E E

or Not either OUTPUT SIMPLE OUTCOME Dimensions in which the relationships of cause and effect are known OUTCOME OUTPUT ACTIVITY OUTCOME OUTPUT OUTCOME INPUTS COM OUT E OU TC O ME ACTIVITY INPUTS ACTIVITY OUTPUT Time INPUTS Ricardo. Wilson-Grau@inter. nl. net 17

or Not either OUTPUT COMPLEX ME O OUTC Unknown relations of cause and effect OUTPUT dominate. ACTIVITY OUTCOME INPUTS OUTCOME OU TC O ME ACTIVITY OUTPUT Time INPUTS COM OUTPUT Results are. ITY ACTIV substantially TS INPU unforeseeable. ACTIVITY OUTCOME Ricardo. Wilson-Grau@inter. nl. net E

Outcome Harvesting is NOT useful Ricardo. Wilson-Grau@inter. nl. net

Outcome Harvesting IS useful Ricardo. Wilson-Grau@inter. nl. net

When do you harvest outcomes? Developmental Formative Summative Progress of intervention Time

Track record Over 300 networks and associations, NGOs, community -based organisations, research institutes, and government agencies on all seven continents. Identifying and reporting almost 3, 000 changes they have influenced. Ricardo. Wilson-Grau@inter. nl. net

Most recently In August, the UNDP identified Outcome Harvesting as one of eleven “innovative approaches to monitor and evaluate the performance of public policies, programmes and service delivery. ” Ricardo. Wilson-Grau@inter. nl. net

What is an outcome? 1. Social actors changing the way they do things Group or community Individual Relationships Behaviour CHANGES SOCIAL ACTOR Institution Organisation Practices Ricardo. Wilson-Grau@inter. nl. net Policies

Example Change: In 2009, the Palestinian Authority revitalizes an employment fund for qualified people living in Palestine.

When is it your outcome? 2. A change that you influenced Effect Cause

Example of influence Change: In 2009, the Palestinian Authority revitalizes an employment fund for qualified people living in Palestine. Contribution: The Global Call to Action against Poverty (GCAP) coalition in Palestine launched the “Stand Up and Be Counted” campaign, which mobilized 1. 2 million people in 2008. Working with the Ministry of Labor, the coalition also helped secure multilateral funding and technical assistance on the management of the fund. Ricardo. Wilson-Grau@inter. nl. net

Paradigm shift First… Identify outcomes

Paradigm shift Second… Discover the activities and outputs that contributed

Other information? • • Significance of the outcome Collaboration with other social actors Contribution of other actors and factors History Context Evidence of impact on people’s lives And so forth. Ricardo. Wilson-Grau@inter. nl. net

Example Change: In 2009, the Palestinian Authority revitalizes an employment fund for qualified people living in Palestine. Significance: Demonstration of how mass citizen action can be combined with the engagement of political decision makers to lead to transformative changes in government policy and practice. Ricardo. Wilson-Grau@inter. nl. net

Six steps of Outcome Harvesting Six steps: 1. Design the harvest 2. Review documentation and draft outcomes 3. Engage with informants 4. Substantiate 5. Analyse, interpret 6. Support use of findings Ricardo. Wilson-Grau@inter. nl. net

1. Design the harvest, focusing on usefulness Primary intended users Principal intended uses Useful Outcome Harvesting questions Credible data Ricardo. Wilson-Grau@inter. nl. net

Credible data = credible sources People who personally know who changed and what changed. They give initial authenticity, confidence and credibility to the information you harvest. Thus, Outcome Harvesting is highly participatory.

2. Review documentation and draft outcomes 3. Engage with informants to craft the outcome descriptions Less is more! Ricardo. Wilson-Grau@inter. nl. net en? h W o? h W at? h W ere? h W 35

Exercise Please take five minutes to read this 2012 annual report from Fund. Now. Then, with the person seated next to you: A. Identify one outcome: – Who changed behaviour, relationships, activities, actions, policies or practice? – What changed? – When did it change? – And where?

Fund. Now 2012 Annual Report Feedback from our allies and co-workers about Fund. Now’s activities during January-December 2012 has been very positive, emphasizing the usefulness of our research publications to inform about the funding challenges and opportunities facing women's organizations in the country. Our research has shown that women's rights organizations have been living in a form of "resist and survive" mode but doing amazing work for the advancement of women's rights. People always mention our online newsletter Fund. Now as a tool that has helped them to think more strategically about raising funds for their organizations. The preparatory research we did for each one of the 13 meetings we convened was greatly appreciated. This year, our innovative research on trends in financing for gender equality and women's organization has been a strongly supportive tool for women’s groups obtaining more and better resources across the country. Based on the collection of "gems" of impact of projects implemented by the grant recipients of the government’s Fund for Leadership and Opportunities for Women (FLOW), we compiled a report that has been used by an ad hoc group of gender activists within the ministry responsible for FLOW. They used our compilation in their internal lobbying for the replenishment of FLOW, which resulted in the addition of US$1 million to the Fund in late 2012. In addition, this year we have successfully organized 2 strategic meetings for women's organizations on provincial resource mobilization. (In addition, we organised 7 more meetings in the capital for a total of 9. ) There was intense interest in these meetings, the first held in those provinces. In one province, 5 funders and 18 women's organizations attended and in the other 4 and 12, respectively. The background research we presented was based on regional surveys carried out in preparation for the meetings; it was very well received. Participants said we were filling a clear gap of information with our findings. Many participants expressed that these meetings are a strategic opportunity to establish new contacts, engage in networking, develop strategies, and have a space for potential donors and grantees to meet and share experiences. Everyone said the meetings enabled them to better understand the context in which both funders and women's organizations operate. The meeting participants also had time to reflect on their work and said these events served as an inspiration. All the participants expressed their commitment to work together in the future to guarantee funding for women's organizations in the two provinces.

Exercise, continued B. How did Fund. Now contribute to the outcome? What activities and outputs plausibly contributed to the change in the social actor, however partial, indirect and even unintentional the contribution may have been?

Engage with Primary Intended User 1. To clarify and confirm the details of use 2. To establish the Outcome Harvesting questions to be answered 3. And identify the information (and its sources) required to answer the harvest questions 39

4. Substantiate: Verify accuracy + deepen understanding A. Description: In late 2012, the ministry responsible for FLOW took the decision to add US$1 million to the Fund. B. Contribution: Fund. Now compiled research that was used by the ad hoc group within the ministry to advocate internally for replenishment of the Fund. 1. To what degree are you in agreement with the description of the ministry’s action as described? [ ] Fully agree [ ] Partially agree [ ] Disagree Comments if you like: 2. How much do you agree with the description of how Fund. Now influenced the ministry’s decision? [ ] Fully agree [ ] Partially agree [ ] Disagree Comments if you like:

Triangulation of sources Historical documentation SOURCES Independent third parties Ricardo. Wilson-Grau@inter. nl. net Person(s) who influenced others over time

Triangulation of methods METHODS Ricardo. Wilson-Grau@inter. nl. net

5. Analyse and interpret the outcomes to answer the useful evaluation questions Ricardo. Wilson-Grau@inter. nl. net

Triangulation of analysis Methodologist ANALYSTS Person(s) who influenced others over time Content experts Ricardo. Wilson-Grau@inter. nl. net

Excel Database Ricardo. Wilson-Grau@inter. nl. net

Access Database Ricardo. Wilson-Grau@inter. nl. net

Theory of change MISSION IMPACT ASSUMPTIONS RESULTS STRATEGIES SOCIAL ACTORS

Systems approach

6. Support for use of findings Report And Support

In sum Outcome Harvesting is “an evaluation approach that does not measure progress towards predetermined outcomes, but rather collects evidence of what has been achieved, and works backward to determine whether and how the project or intervention contributed to the change. ” - UNDP Ricardo. Wilson-Grau@inter. nl. net

Questions? Further information: • Outcome Mapping Learning Community, www. outcomemapping. ca • Ricardo Wilson-Grau Consultoria em Gestão Empresarial Ltda Rua Marechal Marques Porto 2/402, Tijuca, Rio de Janeiro, CEP 20270 -260, Brasil Telephone: 55 21 2284 6889; Skype: ricardowilsongrau Direct via VOIP, dialing locally from: The Netherlands (or Europe): 31 71 302 0429 USA or Canada: 1 347 404 5379 Ricardo. Wilson-Grau@inter. nl. net http: //ow. ly/d 65 Y 1