A guide to outcomes based reporting Presented by

A guide to outcomes based reporting Presented by: United Way of Tompkins County (www. uwtc. org) & Johnson Board Fellows - Frank Aburto and David Andros November 23, 2020

The current economic environment, increased community needs, and a crowded nonprofit sector are pressuring organizations to become more outcomes based Competitive pressures § Increasing community needs unmet by government services § More focus on the impact nonprofits deliver to their clients § Decreasing federal, state, and local funding § Resources (staff, funding, etc. ) limit nonprofits’ ability to conduct longterm program tracking § Increased number of nonprofit organizations 2 Challenges

The program outcome model depicts the relationship between inputs, activities, outputs, and outcomes Inputs Resources dedicated to or consumed by the program (Examples: money, staff time, facilities, equipment) Activities ▪ What the program does with the inputs to fulfill its mission Definition ▪ Example ▪ Inputs for a parent ▪ Parent education class include: class activities − The hours of staff include: time spent − Classes on proper designing and children’s nutrition, delivering the well-being, and program development for − Constraints on the parents program (Examples: − A class on dealing laws, regulations, with children requirements for approaching receipt of funding teenage years Outputs ▪ The direct products of program activities (Examples: number of classes taught, number of participants) ▪ Benefits or changes for recipients during and/or after participating in program activities (Examples: behavior, skills, knowledge, attitudes, values, condition ) ▪ Parent education outputs include: − Number of parents taught − Number of class hours provided ▪ Parent education outcomes are: − Parents now have more knowledge of how to properly care for and support the development of new-born babies − Parents utilize all available resources in Ithaca for health care Source: United Way of America, Measuring Program Outcomes: A Practical Approach (United Way of America, 1996), 1 -9, 18. 3 Outcomes

In addition to determining outcomes, organizations must establish outcome targets and use indicators to track their progress Outcomes ▪ Benefits or changes for recipients during and/or after participating in program activities ▪ Examples: behavior, skills, knowledge, attitudes, values, condition Outcome indicators ▪ Observable and measurable “milestones” toward an outcome target ▪ Example: number and percent of teens that quit smoking directly after the program Source: United Way of America, Measuring Program Outcomes: A Practical Approach (United Way of America, 1996), 61. 4 Outcome targets ▪ The number and percent of participants that you want to achieve the outcome ▪ Example: an outcome goal of 500 teens (10% of teens in Ithaca) who quit smoking over the next year

The three levels of outcomes, a series, each contribute to a benefit or change that leads to the ultimate program outcome Long-term outcomes Intermediate outcomes Initial outcomes ▪ Most closely related to and influenced by a program’s outputs ▪ Link a program’s initial outcomes to the longerterm outcomes ▪ Changes in participants’ knowledge, attitude, or skills ▪ Changes in behavior that result from participants’ new knowledge, attitude, or skills Source: United Way of America, Measuring Program Outcomes: A Practical Approach (United Way of America, 1996), 32. 5 ▪ Represent meaningful change for participants, often in their condition or status ▪ The most removed benefits a program can reasonably expect to influence

Outcome based reporting is a continuous process that allows organizations to demonstrate achievement of goals to donors while improving quality of service 1 Define service goals and specify an outcome indicator ¡ Specify an outcome indicator: − Identify what will indicate achievement of the goal (Example: improve reading comprehension) − Determining the statistic(s) used to calculate the level of achievement (Example: child improved his test score from a 1 to a 5) 2 Collect outcome data ¡ Things to consider: − Is there at least one indicator for each outcome? − Does the indicator measure some important aspect of the outcome that no other indicator measures? − Is the wording of each indicator sufficiently specific? • Capture less concrete outcomes through creative methods (Example: a survey given at the start and end of a program) 3 Measure performance ¡ Data provides context in two ways: − Number of program participants achieving the goal − Percentage of program participants achieving the goal 4 Report outcomes and adjust program if necessary ¡ Share the measured performance with funders ¡ Review internally, align performance with service goal(s) Repeating steps 1 through 4 will increase the accuracy of outcomes, while improving quality of service Source: Based on “Eight Steps to Success” in United Way of America, Measuring Program Outcomes: A Practical Approach (United Way of America, 1996). 6

Step 1: Define service goals and specify an outcome indicator Defining service goals ▪ Choose a program with a reasonably clear set of clients and clear methods for providing service ▪ In cases with less concrete outcomes, learn as you apply your outcomes evaluation plan Specifying outcome indicators ▪ Identify outcome indicators for: − Initial: 0 -6 months − Intermediate: 3 -9 months − Long-term: 6 -12 months ▪ Consider words like: “enhanced…, ”“increased…, ” or “altered…” ▪ Think about how the indicators for each time period are connected ▪ If this is your first time determining indicators do not seek initial perfection, goals can be general in initial stages and become more specific as the program develops Example: An organization that focuses on reducing hunger would define their goal as – providing a feeding program that will give children access to quality food in an adequate quantity. An outcome indicator would be “altered” pattern of behavior such as increased amount of fruits and vegetables consumed per day or better eating habits at a young age. 7

Step 2: Collect outcome data Purpose Observation ▪ May not receive careful feedback ▪ Wording of questions can create bias ▪ Does not always capture the full story ▪ To fully understand a client’s impression and experience ▪ Gathers a fuller range of information ▪ Can help to develop client relationships ▪ Time intensive ▪ Can be difficult to analyze and compare ▪ To gather accurate information about how a program operates ▪ Adaptive ▪ Accurate ▪ Collected in real time ▪ Time intensive ▪ Can influence the way the program operates ▪ Difficult to interpret results Source: United Way of America, Measuring Program Outcomes: A Practical Approach (United Way of America, 1996), 81 -88. 8 Challenges ▪ Inexpensive for any size group ▪ Easily comparable ▪ Sample questions already exist ▪ To quickly and easily collect data in a non. Questionnaires threatening or surveys manner Participant interviews Advantages

Step 3: Measure performance Dealing with numbers ▪ Tabulate the information (Example: Sum the ratings) ▪ Consider computing a mean, median, high, and low for each question Analyzing comments ▪ Read all data ▪ Organize comments into similar categories (Examples: concerns, suggestions, strengths) ▪ Attempt to identify patterns, associations, and causal relationships Example: The following individuals completed the basic computer skills course. At the beginning and at the end of the course they were asked to provide, on a scale of 1 to 5, their comfort level with computers. 9 Name Before After Lisa 2 4 Frank 1 5 John 3 5 Jeff 1 3 Susan 2 5 Average improvement of 2. 6 Median improvement of 2 With an overall 44% increase in computer skills

Step 4: Report outcomes and adjust program if necessary Reporting evaluations ▪ Align the level and scope of information in report with intended recipients (Example: funders) ▪ Ensure employees have a chance to review and discuss the report before distribution Example of Evaluation Report Contents § Executive Summary – one-page, concise overview of findings and recommendations § Program background – describe the service, community need, overall goal § List – outcomes, outcome targets, outcome indicators § Define the data collection methodology: − Types of data/information collected − How data/information were analyzed − Limitations of the evaluation − Interpretations and Conclusions § Recommendations - regarding the decisions that must be made about the service § Appendices: − Data: numbers, observations, testimonials − Logic model Source: Carter Mc. Namara, Field Guide to Nonprofit Program Design, Marketing and Evaluation (Authenticity Consulting, LLC. ). 10

A useful framework - the “Is there a positive impact” question is a simple way to evaluate a program’s outcomes How much did we do? How well did we do it? # customers served (by customer characteristics) % common measures (e. g. , customer satisfaction) # activities (by type of activity) % activity-specific measures (e. g. , % of clients completing the program) Is there a positive impact? # skills/knowledge % skills/knowledge # attitude/opinion % attitude/opinion # behavior % behavior # circumstance % circumstance Source: Robert M. Penna, The nonprofit outcomes toolbox (New Jersey: John Wiley & Sons, Inc. , 2011), 63. 11

Questions 12

Key definitions ▪ Inputs: Resources a program uses to achieve program objectives (Examples: staff/personnel, teaching aids, classroom time, donated food or funds) ▪ Activities: Implementation of the inputs to move towards the goals, what a program does with its inputs, (Examples: classes, sessions, feedings or distributing clothing) ▪ Outputs: Products of a program’s activities - raw data collected at the end of the activities (Examples: number of people trained/helped, meals provided or classes taught) ▪ Outcomes: The key piece of information to report, this is the discernable difference the program created for its clients (Examples: skills/knowledge gained (as measured by an exit survey), attitudes/behaviors changed, and improved living conditions) ▪ Outcome Targets: Numerical objectives for a program’s level of achievement on its outcomes (Intended number and percentage of clients targeted for a particular outcome in the next reporting period) ▪ Outcome Indicators: The specific items of information that track a program’s success on outcomes (Examples: gaining incremental skills/knowledge or small changes in behavior during the program or shortly after) ▪ Measure: Quantifying the outcome by using a baseline (before picture) to determine the performance you are achieving from your service (e. g. , the % in skills/knowledge from the beginning to the end of the program) Source: United Way of America, Measuring Program Outcomes: A Practical Approach (United Way of America, 1996), xv. 13

Program logic model worksheet Source: United Way of America, Measuring Program Outcomes: A Practical Approach (United Way of America, 1996), 44. 14

Outcome measurement framework Source: United Way of America, Measuring Program Outcomes: A Practical Approach (United Way of America, 1996), 60. 15