Ripple Effect Mapping A Tool for Evaluating the

Ripple Effect Mapping: A Tool for Evaluating the Impacts of Complex Interventions PRESENTED AT THE 19 TH ANNUAL MESI SPRING TRAINING CONFERENCE Scott Chazdon, Ph. D. , Evaluation and Research Specialist Extension Center for Community Vitality Kit Alviz, M. A. , Research Fellow, Extension Center for Family Development © 2011 Regents of the University of Minnesota. All rights reserved.

SESSION OVERVIEW § Background – 20 minutes § Demonstration – 40 minutes Want to try mind mapping? Download from www. xmind. net § Discussion – 45 minutes © 2011 Regents of the University of Minnesota. All rights reserved.

SESSION LEARNING OBJECTIVES § Describe the elements of the Ripple Effect Mapping process. § Identify the skills needed to facilitate a Ripple Effect Mapping session. § Give examples of the types of situations in which Ripple Effect Mapping would make sense. § Produce the first stage of a Ripple Effect mind map (optional). § Weigh the benefits and limitations of Ripple Effect Mapping as an evaluation tool. © 2011 Regents of the University of Minnesota. All rights reserved.

RIPPLE EFFECT MAPPING § Purpose – to better understand intended and unintended results of a program, intervention or collaborative for individuals, groups, sectors or communities. § Can be post-program (more summative) or mid-program (more developmental) © 2011 Regents of the University of Minnesota. All rights reserved.

DIRECT VS. INDIRECT IMPACTS § Collaboratives and high engagement programs often build social capital, but don’t take credit for it. § People do not act in isolation -- strengthened social capital is a necessary pre-condition for other impacts § Other impacts may occur that were not foreseen in program theory © 2011 Regents of the University of Minnesota. All rights reserved.

Mind Mapping – Radiant Thinking Pictorial Method – Note taking – Brainstorming – Organizing – Problem solving – Evaluation For more on mind mapping, see Buzan, T. (2003). The mind map book. London: BBC Books. Image: Mindmap, Graham Burnett, http: //en. wikipedia. org/wiki/File: Mindmap. gif

RIPPLE EFFECT MAPPING: RELATED APPROACHES § § Concept Mapping (Trochim, 1989) Mind Mapping (Eppler, 2006) Outcome Mapping (Outcome Mapping Learning Community, 2011) Participatory Impact Pathway Analysis (Douthwaite et al, 2008) § Most Significant Change (Davies 2005) § Appreciative Inquiry (Preskill & Catsambas, 2006) © 2011 Regents of the University of Minnesota. All rights reserved.

HOW DOES IT WORK? § Identify the intervention § Schedule the event and invite participants § Group mapping session held § Follow-up interviews § Cleaning, Coding, Analysis © 2011 Regents of the University of Minnesota. All rights reserved.

RIPPLE EFFECT MAPPING Method – Identify the intervention § High engagement program or position § Cross-sector initiative § Collaboration – Invite stakeholder group § Participants § Non-participant stakeholders § 12 to 20 participants § Two moderators © 2011 Regents of the University of Minnesota. All rights reserved.

RIPPLE EFFECT MAPPING Method – Appreciative Inquiry interview § Conducted among pairs of participants § Examples of questions: – Tell me a story about how you have used the information from the program? – Is there anything that resulting from the program that you are proud to share? – List an achievement or a success you had based on what you learned. © 2011 Regents of the University of Minnesota. All rights reserved.

RIPPLE EFFECT MAPPING Method – Starting the Map § On wall or using Mind Mapping software with data projector § Floating topics generated from Appreciative Inquiry – Different approaches © 2011 Regents of the University of Minnesota. All rights reserved.

THEMING AND RIPPLING - Building the Map § Theming - Group identifies which items are closely related - Group generates initial theme names - Floating topics moved and organized § Rippling - Group cross-validation - Potential for probing using the Community Capitals Framework © 2011 Regents of the University of Minnesota. All rights reserved.

RIPPLE EFFECT MAPPING Examples of Contexts where REM makes sense § Kit: State systems building to improve quality and quantity of child care § Scott: Community leadership development © 2011 Regents of the University of Minnesota. All rights reserved.

RIPPLE EFFECT MAPPING Demonstration of Mapping Process § Think back to your first program evaluation experience. – Is there anything that you gained from this experience that you are proud to share? – List an achievement or a success you had based on what you learned or who you met. – Did the experience lead to other important developments in your personal or professional life? § Floating topics § Beginning to categorize © 2011 Regents of the University of Minnesota. All rights reserved.

Example: Ripple Effect Map of Community Gardening in Frogtown & Rondo neighborhoods

Example: Ripple Effect Map of Community Gardening in Frogtown & Rondo neighborhoods

EXAMPLE: RIPPLE EFFECT MAP OF HUGO, MN BUSINESS RETENTION AND EXPANSION PROGRAM © 2011 Regents of the University of Minnesota. All rights reserved.

EXAMPLE: RIPPLE EFFECT MAP OF CHILD CARE EXPANSION INITIATIVE © 2011 Regents of the University of Minnesota. All rights reserved.

RIPPLE EFFECT MAPPING Cleaning, Coding, Analysis – Organize map to better identify pathways or combine pathways – Download data to Excel for coding – Code using relevant thematic framework and type of outcome § § § KASA = something learned Behavior change = action taken Impact = change in system – Follow-up interviews if more clarity is needed © 2011 Regents of the University of Minnesota. All rights reserved.

The Community Capitals Framework (Emery and Flora, 2008)

RIPPLE EFFECT MAPPING Coding Demonstration © 2011 Regents of the University of Minnesota. All rights reserved.

Coding Example First order (core Second Third outputs) order ripples order Fourth order Market the City of Hugo Human capital effects (knowledge and behavior Social capital change) effects Civic effects City identity workshop How to attract residents and biz (coninuing work) Have identified key attributes about the City Create, Coordinate, and Encourage Events New position at City for park & rec. planning X X X Health, Food Natural Built capital and Nutrition environment effects Effects Cultural effects X X Financial effects X X ~10 new recreation programs X X Hanifl Fields attracted over 20, 000 kids X X Entrepreneurial Bootcamp X Businesses have used City resources Provide promotion opps. for biz Coupons at football tourney X X X ?

Reporting Example (Hugo BR&E program) Category of Community Program Effects Human capital effects (knowledge and behavior change) Social capital effects Civic effects (aka Political) Count of reported effects (out of Definitions of Categories Changes in knowledge, attitudes, or skills among community members. Strengthened or expanded connections among people, groups and organizations. Increased ability of communities to access and mobilize public resources. Financial effects Increased private and public wealth that is invested in the wellbeing of communities. Built capital Improvement of a total of 41 effects reported) Percent of reported effects 25 61. 0% 12 29. 3% 20 48. 8% 11 26. 8% Examples 1. Increased awareness of how many and what kind of businesses are in the community 2. City staffers got to know businesses 1. Hugo Business Association is gathering more often with a purpose. 2. City and business groups are interacting more. 1. Created and filled a park & recreation planning position 2. City is still using a biz resource guide that was created in the BR&E program to assist businesses. 1. Xcel Energy creating a training center with 10 jobs and lots of visiting trainees. 2. Provide promotion opps. for biz 3. City actively recruiting events to be held in Hugo Removed blighted buildings using

Reporting Example (Child Care Expansion) Appendix – Ripple Effect Tables Ripples for Theme 1: Updating regulations Individuals volunteer their time and energy to Health and Safety Regulations Work Group No funding This is their passion; many have been working in the trenches for over 30 years Passion to make child care settings as safe as possible Prevention and keeping parents aware Grounded in the reality of day to day Children that fell through the cracks- reason for being here White papers, living document that can be used daily Completed and submitted Safe Sleep Paper to Dept. of Social Services Community Care Licensing Parent group member presented the paper Within one month, the Licensing Division responded and said they would start moving on the changes Babies will be safer when they sleep

BENEFITS § Simple and cheap tool § Captures impacts of complex or evolving work § Captures intended and unintended impacts § Participatory and appreciative approach that engages stakeholders § Group validation of results © 2011 Regents of the University of Minnesota. All rights reserved.

LIMITATIONS § Risk of bias in participant selection and data collection § Participants may not have complete information about a program or program outcomes § Potential for inconsistency in implementation © 2011 Regents of the University of Minnesota. All rights reserved.

SUGGESTIONS § Use same facilitator, recorder and “mapper” § Develop a facilitator guide with ideas for probes. § Make decision prior to mapping whether to use a pre-existing framework as probes during group interviews § May need to recognize that one organization isn’t trying to take all credit for all change § It is important to probe for negatives © 2011 Regents of the University of Minnesota. All rights reserved.

LESSONS LEARNED THUS FAR § Find the right balance between breadth and depth § Schedule the event along with another activity § Put much effort into recruitment and explaining the process § Choose a good setting – not too informal § Use external facilitators, not program staff © 2011 Regents of the University of Minnesota. All rights reserved.

Q&A AND DISCUSSION © 2011 Regents of the University of Minnesota. All rights reserved.

REFERENCES § Baker, B. , Calvert, M. , Emery, M. , Enfield, R. , & Williams, B. (2011). Mapping the impact of youth on community development: What are we learning? [Power. Point slides]. Retrieved from http: //ncrcrd. msu. edu/uploads/files/133/Mapping%20 Impact%20 of%20 Youth%20 on%20 Com%20 Dev%2 012 -3 -10. pdf § Buzan, T. (2003) The Mind Map Book. London: BBC Books. § Douthwaite, B. , Alvarez, S. , Thiele, G. , & Mac. Kay, R. (2008). Participatory impact pathways analysis: A practical method for project planning and evaluation. ILAC Brief 17. § Emery, M. , & Flora, C. B. (2006). Spiraling-up: Mapping community transformation with community capitals framework. Community Development: Journal of the Community Development Society 37(1), 1935. § Eppler, M. J. (2006). A Comparison Between Concept Maps, Mind Maps, Conceptual Diagrams, and Visual Metaphors as Complementary Tools for Knowledge Construction and Sharing. Information Visualization 5: 202 -210. § Hansen Kollock, D. A. , Flage, L, Chazdon, S. , Paine, N. , and Higgins, L. (2012). Ripple Effect Mapping: A “Radiant” Way to Capture Program Impacts. http: //www. joe. org/joe/2012 october/tt 6. php § Kollock, D. A. (2011). Ripple effects mapping for evaluation. Washington State University curriculum. Pullman, WA. § Outcome Mapping Learning Community website. (2011). http: //www. outcomemapping. ca § Preskill, H. & Catsambas, T. T. (2006). Reframing evaluation through appreciative inquiry. Thousand Oaks: Sage Publications. © 2011 Regents of the University of Minnesota. All rights reserved.

CONTACT INFORMATION Scott Chazdon, Ph. D. Evaluation and Research Specialist Center for Community Vitality 612 -624 -0982 schazdon@umn. edu Kit Alviz, M. A. Research Fellow Center for Family Development 612 -625 -8271 alviz 002@umn. edu © 2011 Regents of the University of Minnesota. All rights reserved.

Thank you! © 2011 Regents of the University of Minnesota. All rights reserved. The University of Minnesota is an equal opportunity educator and employer. This Power. Point is available in alternative formats upon request. Direct requests to 612 -625 -8233.