Tips and Tools for Measuring Program Impacts From

Tips and Tools for Measuring Program Impacts: From Delivering Programs to Reporting Impacts Office of Educational Innovation and Evaluation 2323 Anderson Avenue, Suite 220 Kansas State University Manhattan, Kansas 66502 Phone: 785 -532 -5930 Fax: 785 -532 -7185

Part I: Preparing to Evaluate Your Program

Tips ØFind and review evaluation instruments ØUtility and feasibility ØPurposes of evaluation: Accountability, Improvement, Knowledge Development, Oversight and Compliance § “Who wants to know what about your program? ” ØMultiple, complementary sources of data

Tools Rubric: Reviewing Evaluation Instruments UTILITY Value to PFT Consider: How well will the evaluation instrument gather data that aligns with the PFT action plan? Alignment with Program Curriculum Consider: How well aligned is the evaluation instrument with the curriculum that was taught? Low Medium High Notes

Part II: Collecting and Managing Data to Measure Impacts

Tips Ø Plan early – determine when evaluation will take place in the programming cycle Ø Review and manage data as you receive it Ø Analysis § quantitative and qualitative § by hand, using Excel § “Who wants to know what about your program? ”

Tools Data Analysis Plan Worksheet Question What did I hope to What analysis How could I present the or Item learn from this technique would assist data to convey what I Number question? in that learning? learned? E. g. , #2 If participants felt that they increased their knowledge of best management practices for a healthy yard. Count and percentage of respondents who reported “Agree” or “Strongly Agree” Bar or pie graph that compares “Agree” and “Strongly Agree” responses to “Disagree, ” “Strongly Disagree” and “Unsure” responses

Part III: Presenting Impact Data

Tips ØKSRE Impact Reports § “Who wants to know what about your program? ” ØIllustrate your data ØBe clear and concise, avoid jargon ØUse the active voice

Tips Choose Graphics to Effectively Convey Data Example #1 – Table NOT formatted effectively Table 1. Chewing Gum Rates in the United States by Geographic Region and Level of Education* Region Education (in Years of Schooling) ≤ 8 9 -11 12 13 -15 ≥ 16 Northeast 25. 32 25. 29 18. 26 18. 29 16. 34 North Central 32. 09 29. 04 18. 78 24. 32 19. 02 South 38. 81 31. 02 19. 33 15. 66 16. 79 West 25. 37 21. 09 20. 29 23. 97 17. 52 Example #2 – Table IS formatted effectively Table 2. Chewing Gum Rates in the United States by Geographic Region and Level of Education* Education Level ≤ 8 9 -11 12 13 -15 ≥ 16 Average Region Northeast West South 25 25 18 18 16 20 25 21 20 24 18 22 39 31 19 16 17 24 North Central 32 29 19 24 19 25 Average 30 27 19 21 17 23 *Education level based on number of years of schooling. Adapted from a report of the U. S. Department of Chewing Gum.

Tools What Does the Data Mean? What are the key findings? Write a paragraph to summarize the results. E. g. , After participating in the program, many participants reported feelings of increased financial abilities. In particular, a majority of participants reported increased skills to track spending (64%), to create a spending plan (71%), and to plan for unexpected expenses (77%). One respondent commented, “The skills I learned from this program will help me stretch my money to the next payday. ” What do you find interesting about these results? What would your audience find interesting?

QUESTIONS?

Contact Information Sarah Bradford (785) 532 -5677 sbradfor@ksu. edu Amy Hilgendorf (785) 532 -5538 aehilgen@ksu. edu Office of Educational Innovation and Evaluation Kansas State University 2323 Anderson Avenue, Suite 220 Manhattan, KS 66502 Phone: 785 -532 -5930 www. ksu. edu/oeie