Evaluation Learning From the Inside Out Freya Bradford
Evaluation (Learning) From the Inside Out Freya Bradford, Learning Officer Rotary Charities & North. Sky Nonprofit Network June 24, 2015
Theories Important to Learning 1. Behavior Change Requires Attitude Change 2. All Change Starts with Individuals 3. Evaluation Should Aim for Contribution, not Attribution
https: //www. mentimeter. com/s/d 6 daac 6 c 4 c 1 0 a 3 f 27654 a 88 d 2 db 042 d 0/f 872 f 0991 d 15 http: //bit. ly/1 di. FXHn
https: //www. mentimeter. com/s/d 6 daac 6 c 4 c 1 0 a 3 f 27654 a 88 d 2 db 042 d 0/d 1 d 7 cc 2 b 94 cc
Theories Important to Learning 1. Behavior change requires attitude change Informational/ Cognitive (i. e. Beliefs) Affective (i. e. Emotions) Attitude Behavior
What? So What? Now What? What are some of your/your organizations beliefs and emotions about evaluation that may be contributing to your evaluation behaviors? Why do these beliefs and emotions matter? How might they be contributing to your evaluation behaviors? What are some immediate things you might do to decrease/change negative beliefs or emotions or increase/spread positive beliefs or emotions? *Adapted from Liberating Structures, www. liberatingstructures. com
www. liberatingstructures. com
Theories Important to Learning 2. All Change Starts with Individuals Planet Regions Communities Systems Organizations Individual
Theories Important to Learning 3. Evaluation Should Aim for Contribution, not Attribution Grant bought this oar Boat traveled this far. We cannot attribute the distance traveled to the grant. The grant contributed to the distance traveled.
Theories Important to Learning 3. Contribution not Attribution
Evaluation for Learning Plan Use/ Adapt Do
Evaluation Cycle Use Implementation Planning Define Evaluation Purpose Create Theory of Change Define Evaluation Questions Select Indicators Develop Methods & Tools Collect & Analyze the Data Share, Learn & Adapt
Time Commitment Use Planning Implementation *Picture adapted from William & Flora Hewlett Foundation
Evaluation Cycle Use Implementation Planning Define Evaluation Purpose Create Theory of Change Define Evaluation Questions Select Indicators Develop Methods & Tools Collect & Analyze the Data Share, Learn & Adapt
Define Evaluation Purpose • Discuss stakeholders’ ideas about why you should evaluate your grantmaking • Discuss the principles that should guide planning
• Silently, write down some things you do currently to evaluate your work • Get into pairs • Unleash your inner toddler • Each person in a pair is interviewed by his or her partner for 5 minutes. Starting with “What do you do when you evaluate your work? ” the interviewer gently seeks a deeper answer by repeating the query: “Why is that important to you? ” Switch roles after 5 minutes. *Adapted from Liberating Structures, www. liberatingstructures. com
Evaluation Cycle Use Implementation Planning Define Evaluation Purpose Create Theory of Change Define Evaluation Questions Select Indicators Develop Methods & Tools Collect & Analyze the Data Share, Learn & Adapt
Create Theory of Change NEEDS RESOURCES/INPUTS STRATEGIES/ACTIVITIES OUTCOMES What you will The community needs/conditions The human, that you are financial, addressing intellectual (etc. ) resources that have to contribute to the strategies do/deliver to address the need The individual or organizational identified changes you expect to see for those you are working with IMPACT The community level changes you contributed to if your outcomes are achieved
Create a Theory of Change
Rotary Charities Grantmaking Theory of Change NEEDS NPOs are vital to prosperity NPOs are challenged to meet increasing demand with declining resources Social problems are complex – solutions require NPOs to be agile & innovative RESOURCES/INPUTS STRATEGIES/ACTIVITIES 4 FTE 150 Volunteer Pre-application OUTCOMES Rotarians meetings w all 20+ North. Sky Consultants Grand Vision guiding principles Org Capacity Assessment tool Evaluation curriculum Investment earnings Partners grantees Grantees will: Free application/eval Be better able to training & TA to all meet their mission applicants Leverage more Multi-level review of resources applications Be better Feedback to all connected applicants TA to all grantees Have greater capacity IMPACT Grantees will have greater impact on community level needs
Evaluation Cycle Use Implementation Planning Define Evaluation Purpose Create Theory of Change Define Evaluation Questions Select Indicators Develop Methods & Tools Collect & Analyze the Data Share, Learn & Adapt
Define Evaluation Questions • “If I had an hour to solve a problem and my life depended on the answer, I would spend the first 55 minutes figuring out the proper questions to ask. ” • Measure what matters to your organization • Think about uses for the data – Can you identify a decision that could be made as a result of the data? – Are you ready to make that change? • Reach agreement on questions & stay with them
Rotary Charities Evaluation Questions 1) To what extent, and in what ways, does Rotary Charities grantmaking increase grantees’ abilities to meet their mission? 2) What Rotary Charities/North. Sky activities & resources (beyond providing grant funds) are impacting the ability of grantees to be successful? 3) To what extent are grantees achieving outcomes that are likely to impact substantiated community needs?
Evaluation Cycle Use Implementation Planning Define Evaluation Purpose Create Theory of Change Define Evaluation Questions Select Indicators Develop Methods & Tools Collect & Analyze the Data Share, Learn & Adapt
Select Indicators • Indicators: The pieces of data to be collected that together will answer each evaluation question • Multiple indicators will likely be needed to provide a complete answer to an evaluation question. Common Process Indicators • FTE/volunteer hours contributed • # of partners • Amount of products/services delivered • #/type of clients served • #/type of materials produced/disseminated • Timeliness of service provision • Quality of services (satisfaction data) Common Outcome Indicators • #/% indicating increased knowledge/skill • #/% indicating attitude or behavior change • % changes in conditions (longer-term)
Select Rotary Charities Indicators Outcome Indicators Process Indicators • Number/total dollar amount of grants made in reporting period • By sector • Annual budgets of grantees, # of employees • Use of North. Sky/RC supportive services • Extent to which grantees reported that their grant impacted their ability to meet their mission • % of grantees reporting leverage • Amount of leverage • % of grantees reporting new connections that were a result of their grant • By sector connected with • % of grantees reporting capacityrelated outcomes • % of grantees reporting an increased ability to evaluate programs
Evaluation Cycle Use Implementation Planning Define Evaluation Purpose Create Theory of Change Define Evaluation Questions Select Indicators Develop Methods & Tools Collect & Analyze the Data Share, Learn & Adapt
Methods & Tools • Activity tracking • Document review • Surveys • Tests • Focus groups • Interviews • Case studies
Rotary Charities Methods Evaluation Methods 1) Post-Grant Period Evaluation 2) Grantee Focus Groups 3) Grantee Final Reports – Document Review 4) Logic Model Training Evaluation 5) RC Board & Staff Survey & Discussion
Evaluation Cycle Use Implementation Planning Define Evaluation Purpose Create Theory of Change Define Evaluation Questions Select Indicators Develop Methods & Tools Collect & Analyze the Data Share, Learn & Adapt
Collect & Analyze Data • Participatory – Plan outlines responsibilities • Develop infrastructure to house/manage the data • Evaluator monitors & facilitates collection • Follow timeline (fiscal year) • Use evaluation team to discuss preliminary findings
Evaluation Cycle Use Implementation Planning Define Evaluation Purpose Create Theory of Change Define Evaluation Questions Select Indicators Develop Methods & Tools Collect & Analyze the Data Share, Learn & Adapt
Share, Learn & Adapt • Share report(s) internally – adapt reports • Share final report(s) externally – With grantees – Partners • Less is more & Pictures are good • Adapt – Grantmaking practices/programs – Evaluation Plan & Tools • Repeat cycle
Share, Learn & Adapt • Potential uses for evaluation data include: – Develop recommendations for improvement – Identify additional technical assistance or training needs – Highlight the most effective strategies by comparing outcomes for different grant types, strategies, between years, etc. – Use in strategic planning – Adjust resource allocations
Resources North. Sky Resource Center This presentation and other free resources http: //www. northskynonprofitnetwork. org/resourcecenter Including: Liberating Structures, www. liberatingstructures. com Evaluation Principles and Practices, The William and Flora Hewlett Foundation, December 2012 From Insight to Action: New Directions in Foundation Evaluation, FSG Social Impact Advisors , April 2007 Foundation Effectiveness, Definitions and Challenges, The Urban Institute, 2004 Indicators of Effectiveness: A Practical Worksheet for Foundation Staff and Board, The Center for Effective Philanthropy, 2002
Grantees By Sector Arts & Culture 19% 27% 5% 5% 10% 15% 19% Societal Benefit (Gov't) Environment Human Services Education Health Care All Others Employees $2, 171, 437 in 59 Grants to 56 Organizations Annual Budgets Grantees Reporting FY 2012 - 2013 Mean: $2, 782, 615 Median: $643, 756 Range: $2, 000 to $80, 000+ Mean: 10 Median: 16 Range: 0 to 66 (12% have 0 staff)
Helping Grantees Achieve Their Missions 92% of grantees reported that their Rotary Charities Grant impacted their ability to meet their mission to a Great What Rotary Charities is Doing • Pre-Grant meetings with RC staff and North. Sky (when appropriate) • Grant application and logic model training (3 grant types) • Grant investigation process with Rotarian volunteers (4 grant types) • Evaluation requirement and assistance (3 grant types) • Open door for all grantees • Capacity building services through North. Sky Extent To Support These Conditions
Attracting and Leveraging New Resources 61% reported a total of $7, 633, 362 in leveraged resources that were a result of their Rotary Charities grant.
Encouraging Partnerships 72% reported at least one new connection as a result of their grant. “The history and folklore in our world is that Rotary’s grant changed our organization in itself. It forced collaboration where many organizations were out doing their own thing…not in any grant language, but in the relationship they have with us and the community. We are now coordinating our effort and are more effective as a result. ” – Focus Group Participant
Increasing the Capacity of Organizations
Increasing the Sustainability of Programs AVERAGE EXTENT GRANT IMPACTED SUSTAINABILITY Multiple Year Grantees were 2. 7 x more likely to report increased sustainability as an outcome than single year grantees.
Increasing Learning and Adaptability “The logic model is one of the hallmarks of Rotary. In the beginning it was scary, but now we see the value in tracking programs and it taught us a lot on how to approach the work. We have a lot of artsy educators who want to do a million programs and it helps to ground them in what is realistic. ” Focus Group Participant
Supportive Systems Grantees 2 X The Number of Reported Outcomes
IMPACTING POSITIVE CHANGE IN THE REGION What Grantees are Doing • 23 Improved their Programs and Services • 20 Provided More Service • 10 Expanded Service to a New Geographic Area • 12 Created Jobs • 16 Retained Jobs What are They Achieving Raising awareness Increasing knowledge Building skills Changing behaviors Influencing public policies • Increasing access to and protecting our natural and cultural assets • • •
IMPACTING POSITIVE CHANGE IN THE REGION More Prosperous & Resilient Community
- Slides: 48