Why are you doing what youre doing OUTCOMESBASED

Why are you doing what you’re doing? OUTCOMES-BASED EVALUATION FOR PLANNING AND IMPACT MIKE CRANDALL & SAMANTHA BECKER UNIVERSITY OF WASHINGTON INFORMATION SCHOOL 1

Goals for Today’s Session ◦ Introduce the basics of outcomes-based evaluation ◦ Introduce the key elements of an OBE plan ◦ Walk through the OBE process together ◦ Prepare you for your own OBE 2

Why Evaluation? WHAT IS OUTCOMES-BASED EVALUATION? HOW IS IT USEFUL TO ME? 3

What is outcomes-based evaluation? Needs of community How Goals of providing services Focuses Indicators of achievement Methods for data collection Defines Why Success Room for improvement Documents 4

Practice brings added benefits Outcomes-based evaluation is part of a culture of performance management, and when done well: ◦ Fosters a culture of organizational learning ◦ Fosters intentionality, clarity, and specificity ◦ Helps clarify goals of services ◦ Prevents mission creep ◦ Gets staff involved ◦ Frames successful advocacy ◦ Is useful for decision making ◦ Is integrated into organization (not just one person’s job) 5

The process AN OVERVIEW OF THE 9 -STEPS TO AN EVALUATION PLAN 6

9 steps to an outcomes evaluation Assemble team Conduct needs assessment Map change Develop logic model Develop evaluation questions Select indicators Identify data sources Create analysis plan Employ reflective practice 7

Getting started LAYING THE GROUNDWORK FOR YOUR EVALUATION PLAN 8

Step 1: Assemble your working team üProgram staff/managers üLibrary friends üPatrons/end-users üOther orgs/agencies/departments People who can help you: ◦ Understand the needs of your community ◦ Collect and analyze data ◦ Understand expectations in the political environment ◦ Define success ◦ Roll up their sleeves and do the work 9

Step 2: Needs assessment Resources/Input Activity Output Outcomes Impact 10

Documented need • Gather sources of data about your community and the population you serve: • • Census data Labor/business statistics Schools Municipal surveys Strategic plans News articles/op-eds Mayor’s speech ◦ Assemble a community profile like the case studies 11

Field work for needs assessment ◦ Find out about other community needs ◦ Outreach to populations that don’t come into the library ◦ Discovery of outcomes from library use and how change occurs (what works) ◦ Expectations about what the library should do/role in the community ◦ Can be done through ◦ ◦ ◦ Interviews Focus groups Community forums Observations Surveys 12

Step 3: Map theory of change Resources/Inputs Activities Outputs Outcomes Impacts Theory of change • Express how you see your activities as creating change in your target population to bridge what you do with what you want to accomplish for the community. 13

Theory of change: Having, Knowing, Doing Books Reference Need Public technology Ability to do Potentia l to become Programs to help your patrons get here? What do you do here… And why/how do you think it helps?

Theory of change examples Impact intention: Help people gain self-sufficiency Digitization impact intention: Help community have increased access to digital library resources Activities: ◦ ◦ Public computer sessions Digital literacy training Job search classes Resume preparation Theory of change ◦ Access to computers enables ◦ Learning digital literacy skills leads to ◦ Having skills necessary to fill job vacancies and look for work so that ◦ Patrons are able to get hired in order to ◦ Earn an income so they can ◦ Be self-sufficient ◦ Digitization training for library staff ◦ Community repository ◦ Access to databases Theory of change ◦ Digitization training enables ◦ Libraries to gain knowledge and skills to create local digital collections leads to ◦ Libraries take a coordinating role in building community collections so that ◦ Libraries build partnerships, increasing their visibility, resource base, and sustainability so that ◦ Residents access digital holdings and realize enhanced research outcomes 15

Activity 1 WRITE A THEORY OF CHANGE ABOUT THE PROGRAM YOU WANT TO EVALUATE

Teen STEM program TOC Teens in our community go to one of the best high schools in the county, have afterschool activities, and generally come from households with higher incomes and educational achievement, As a result of these other resources available for teens in Pebble Pond, library use among teens has gone down. The library wants to engage teens and provide targeted services for them so that they begin attending teen programming, So that librarians can begin to build relationships with teens, so that the library can create a teen advisory committee So that the library can hear more easily from teens, so they can make better programmatic decisions, so that teens like coming to library programs, so that the number of teens reached by the library increases, so that Libraries can begin introducing STEM topics into programming and working with schools to align STEM activities with school curriculum, so that Teens have enrichment activities and supportive resources that will help them engage in STEM topics, so that they gain confidence and experience with STEM occupations, so they feel more inclined to pursue other STEM experiences, so that they study STEM fields in college 17

Developing metrics THREE STEPS TO HELP FIGURE OUT WHAT TO MEASURE 18

Step 4: Develop logic model Inputs Activities Outputs Outcomes Impacts • Program resources used to produce outputs and outcomes • Dollars • Program staff • Equipment • Activities and services delivered; expected to lead to outcomes • Number trainings provided to grantees • The quantity of activities completed • Hours of training attended • Number of people served by portal • Number of item-level views • Short-term - changes in knowledge, awareness, skills • Intermediate - changes in actions, behaviors, policies • The big picture results of what the program did • Relates to your mission • Aspects of the clients or community that you are trying to change __________________________________________ Your Planned Work Your Intended Results 19

Teen STEM program logic model Inputs Activities Outputs Outcomes Impacts • Teen librarians • Teen library space/room • Programming budget • Form and maintain a teen advisory committee • Conduct a teen needs assessment • Library staff coordinate with schools • Library develops and delivers teen STEM programs • The number of meetings of the teen • Early • Increase in the number of teens • Teens have greater support for advisory committee • Actionable conclusions from the needs assessment • Number of new partnerships with schools • Number of new teen STEM programs • Hours of teen participation in new STEM programs visiting the library • Increase in the number of teens engaged in library leadership activities • Increase in the numbers of teachers who regard the library as an educational partner • Intermediate • Teens have increased confidence in STEM participation • Teachers and the library regularly coordinate curriculum with library programming to support STEM education pursuing STEM-related education and career paths • More teens from Pebble Pond end up in STEM-related educational programs • More teens from Pebble Pond enter STEM-related fields • Businesses recognize Pebble Pond for having STEM-ready young workforce and relocate. • The average earnings for residents of Pebble Pond increases 20

Activity 2 DEVELOPING YOUR EVALUATION QUESTIONS 21

Step 5: Develop evaluation questions Identify what you want to learn about your program and its results ◦ Who will use what you learn and for what purposes? Logic model provides a framework when brainstorming Questions will guide your choice of indicators, data collection, and reporting 22

Library staff coordinate with schools Library develops and delivers teen STEM programs Do we have enough resources to support a STEM program? Why has teen use declined? Does STEM programming entice teens to come to the library? Actionable conclusions from the needs assessment Number of new partnerships with schools Number of new teen STEM programs Hours of teen participation in new STEM programs Number of teen visits to the library Early Increase in the number of teens visiting the library Increase in the number of teens engaged in library leadership activities Increase in the numbers of teachers who regard the library as an educational partner Intermediate Teens have increased confidence in STEM participation Teachers and the library regularly coordinate curriculum with library programming to support STEM education Does engaging with schools/teachers result in better programming? Impacts Conduct a teen needs assessment The number of meetings of the teen advisory committee Outcomes Form and maintain a teen advisory committee Outputs Teen librarians Teen library space/room Programming budget Activities Inputs Teen STEM evaluation questions Teens have greater support for pursuing STEM-related education and career paths More teens from Pebble Pond end up in STEM-related educational programs More teens from Pebble Pond enter STEM-related fields Businesses recognize Pebble Pond for having STEM-ready young workforce and relocate. The average earnings for residents of Pebble Pond increases What aspects of the program support STEM confidence? 23

Step 6: Select your indicators Indicators need to meet certain utilitarian standards. Beyond the actual content of the indicator, they should also be: Specific, unique and unambiguous Observable, practical, cost effective to collect, and measurable Understandable and comprehensible Relevant (measures important dimensions, are appropriate and related to the program, that are of significance, predictive and timely) Time bound Valid, providing reliable, accurate, unbiased, consistent, and verifiable data (Hatry, 2006) 24

Frame outcomes as indicators Turn your expected outcomes into measurable indicators, e. g. we want library staff to increase their technology skills ◦ Number of library staff able to complete digitization tasks who were unable to do so previously ◦ Number of library staff reporting increased technology confidence, knowledge/skills ◦ Number of library staff who report applying new technology knowledge/skills to other tasks/projects For Teen STEM program ◦ Increase in the number of teens visiting the library ◦ The monthly change in the number of teens visiting the library 25

Look for existing indicators ◦ Many organizations and researchers have invested time and money in developing good indicators for a variety of social/public programs, as well as data collection procedures you can borrow. ◦ Using existing indicators has the added bonus of providing you with benchmarks to compare against, since you will be using the same metrics http: //www. communityindicators. net/ 26

Making the plan THE FINAL STEPS FOR OPERATIONALIZING YOUR EVALUATION PLAN 27

Step 7: Identify data sources QUANTITATIVE DATA/COLLECTION QUALITATIVE DATA/COLLECTION Get at the “what” Get at the “how” and “why” ◦ Descriptive data about your population ◦ Programmatic data ◦ Survey data Often used in summative evaluation & formal reports ◦ Interviews ◦ Focus groups ◦ Survey data (open-ended questions) ◦ Observations ◦ Interactive/instructor collected Often used in formative evaluation & for storytelling 28

Best practices for collecting original data Understand your ethical obligations ◦ Respect your subjects ◦ Don’t ask more than you need to know ◦ Know your human subjects requirements Have a plan & a purpose ◦ Identify what kind of data you need ◦ Identify data sources, method, timing Use existing indicators and data collection procedures from other orgs. Look for opportunities to gather outcome data that you can observe (rather than asking in a survey): ◦ Students sending instructor email ◦ Completed projects ◦ Numbers of new partners Collect for quality, not quantity ◦ Test your instruments ◦ Have interview guides ◦ Be consistent and timely 29

Step 8: Create your analysis plan Use data to answer your evaluation questions ◦ What happened: What did we do? What resulted? How and why? ◦ What worked, for whom, in what ways, under what conditions, with what implications? ◦ Are we doing the right things? (Testing program’s theory of change) ◦ Are we doing them right? (Assessing implementation quality) 30

Sample from analysis plan Evaluation questions: • How many people use the library computers to look for jobs? • What kinds of people use library computers for looking for jobs? • Are people using computers for job seeking different than people who use library computers for other purposes? Self-reported characteristics of public access technology users Weighting? Univariate or Bivariate? Independent var(s) and scale? Dependent var(s) and scale? Is dependent normal? Proposed test (if normal) Proposed test (if not normal) Notes Yes Bivariate Z_income Continuous Z_race_cats Categorical Z_education Ordinal N/A One-sample t-test Point or interval estimate of population mean, Margins of error Chi-square goodness-of-fit test (grouped scores) Point or interval estimate of population mean, Margins of error Check CIs and SEs, as well as skew/kurtosis. If non-normal, use quartiles. 31

Step 9: Employ reflective practice Telling your story Accountability Wise program decisions 32

Questions & Discussion 33

Resources to help you Taking Stock: A Practical Guide to Evaluating Your Own Programs (Horizon Research) http: //www. horizon-research. com/reports/1997/taking_stock. php Hatry, H. P. (2008). Performance measurement: Getting results. Washington, DC: The Urban Inst. Press. The Urban Institute: Outcome Indicators Project http: //www. urban. org/center/cnp/projects/outcomeindicators. cfm University of Wisconsin-Extension Program Development and Evaluation Resources http: //www. uwex. edu/ces/pdande/evaluation/index. html Crandall, M. & Fisher, K. E. (2009). Digital inclusion: Measuring the impact of information and community technology. Medford, NJ: Information Today, Inc. 34

Journal reflection What are some takeaways you have from this session? How might you use these concepts, tools, and/or resources in your evaluation project? What challenges might you experience when implementing these new concepts, tools, and/or resources? What are some strategies you can use to address these challenges? List 1 -3 action items related to what you learned in this session. When will you complete each of these items?