Evaluation North Orange County Public Safety Task Force

Evaluation North Orange County Public Safety Task Force CBO Capacity Building Workshop Series

WORKSHOP OBJECTIVES • Reconsider your mindset about evaluation • How to approach your work with an evaluative mindset • Take some of the mystery out of evaluation • Increase knowledge of what makes for a stronger evaluation • Increase confidence for those taking steps toward planning their first evaluation/ strengthening their evaluation capacity

Principles • We engage in evaluation every day. • Evaluation is not an activity but a way of doing business. • There is not a one and/best way to do evaluation • However, there are steps which, if taken, set your agency up for a more successful evaluation.

Principles • Evaluation is fundamentally about ascribing value to some program, intervention, agency, etc. • When planning the evaluation, aim to measure the least amount possible. • Evaluation is not about looking for 100% certainty that a program, agency, or intervention resulted in the outcomes – don’t let this belief stop you!

EVALUATION IS SOMETHING WE ALL ENGAGE IN • When was the last time you purchased an expensive product? • When was the last time you conducted a job search? • When was the last time you had to choose between two or more curricula for a new program?

Evaluative thinking Cognitive process motivated by inquisitiveness and a belief in the value of evidence 1. Identifying assumptions 2. Posing thoughtful questions 3. Pursuing deeper understanding through reflection and perspective taking 4. Making informed decisions in preparation for action

Mainstreaming evaluation means making it central to an agency’s work, rather than an add-on, end-of-project mandate. EVALUATIVE THINKING EMBEDDED AND VALUED AS A WAY OF DOING BUSINESS EVALUATION AS A COMPLIANCE ACTIVITY Thinking about what kinds of information are most needed for learning and improvement. Focusing on evaluation contract requirements and procedures. Reflecting together on evaluation findings, learning lessons, and applying them from future decisions. Checking off that evaluation reports have been submitted and filed.

What can evaluation do for your agency? • Measure change your programs create. • Keep agencies on track and offer ways to make strategic decisions when shifts • Tells you where you are doing well and in plans are needed. where improvements are needed. • Helps to determine if the work we are • Helps develop language that tell their doing is the most meaningful work story. and/or the right work for the right people at the right time. • Holds agency accountability to participants, community members and • Helps to develop an evidence base for funders. agency’s work

THERE IS NO ONE BEST WAY TO CONDUCT EVALUATION • Purpose • Evaluative questions • Methodology • Inclusiveness • Level of confidence required

considerations that make for a more successful evaluation • When you plan the evaluation • Who should be included in evaluation planning / implementation? • What is the context of your evaluation?

WHO TO INCLUDE IN THE EVALUATION PLANNING/IMPLEMENTATION? • Who is impacted by the • To whom will the evaluation program and evaluation? need to speak? To whom do • Who makes decisions about you need or want to tell your or can impact how the story of your work? program or the evaluation is • What areas of expertise do implemented? you need? Who can you • Whose voices are most in draw on for that expertise? need of amplification during the process of evaluation planning?

Context of evaluation • What external factors impact the way your evaluation will be carried out and the way the data will be or won’t be used? • What resources are available to support evaluation efforts? • What characteristics of your target audience might have an impact on your evaluation. • What funder restrains or guidelines do you need to follow or meet?

How to do evaluation (simplified) • Ask important question about your work • Collecting information that helps you answer that question • Make sense of that information (form evaluative conclusion) • Act on that information

Elements of a good evaluation • Inclusion of rich program description • Clearly stated purpose • Meaningful evaluation questions • Explicit description of criteria to be used to judge the value of program • A mix of evidence • Evaluative conclusion

PROGRAM DESCRIPTION Logic Models - a way of depicting the program by specifying inputs, activities, outputs, and outcomes. • Must be sequential. • Must be logical. Theory of Change - explains how to produce desired outcomes. • Must be explanatory- it must explain why and how the activities produce the outcomes.

Where evaluative thinking comes in handy NOT LOGICAL AND REASONABLE Participating in an after-school-drop-in Attending an after-school-drop-in center will help keep kids out of trouble will increase school achievement. after school. A safe house for domestic abuse survivors will lead to jobs. A safe house for survivors of domestic abuse will provide support and stability to enable participants to figure out next steps and get referrals for longer term help.

DIFFERENT PURPOSES of evaluation • To determine overall quality or value of something • To find areas of improvement • Both of the above

Types of evaluation questions • Questions about absolute value (Summative) • Questions about the relative value of something (Formative)

Purpose and questions PURPOSE Overall quality (summative) Finding areas for improvement (formative) QUESTIONS ABOUT ABSOLUTE VALUE QUESTIONS ABOUT RELATIVE VALUE Is this intervention worth what it costs? Which of these three professional development interventions we tried was the most cost-effective? How well does the content of this training program match the real needs of our minority trainees? How do the initial improvements in manufacturing efficiency compare with those achieved elsewhere in the industry?

Selecting criteria (outcomes) • Outcomes are clear statements of targeted change. • Community / social norms • School/community climate • Individual attitudes, beliefs, or behavior • Relationship dynamics • Organizational operations and practices

Selecting indicators • An indicator is a measurement of the outcome • Number of children in foster care who are safely reunited with family of origin • Number of unemployed who become employed • Number of former truants who regularly attend school • Score on an instrument that measures self-esteem

A note on selecting indicators • Choose indicators that are meaningful, easy to collect, and closest to your questions. • • • Resist following your curiosity when planning to collect data. How will you know if …. Consider what data is credible for your audience • “No numbers without stories; no stories without numbers” (Patton, 2014)

WHEN TO COLLECT DATA? • Pre/Post • Retrospective Pre/Post • Post only (with comparison) • Ongoing and integrated

How to collect data • Qualitative, qualitative, or both • Observational data • Focus groups/ interviews • Questionnaires • Existing data/documents • Creative materials

Analyzing data • Qualitative data • Certain methods are relatively easy to learn and implement. • With the use of rubrics, qualitative data can be quantified. • Quantitative data • Most often used are simple descriptive statistics. • Frequencies • Percentages • Means, modes, and medians • Inferential statistics • t test • ANOVA • Chi square

The causality issue • Outside of implementing a randomized control trial there are ways to infer causation: • • Ask observers Check if content of evaluand matches the outcome Look for other telltale patterns that suggest one cause or another Check if timing of outcomes makes sense Check if does is related logically to the indicators Make comparison with a control or comparison group Control statistically for extraneous variables Identify and check underlying causal mechanisms

Pulling it all together

Q&A

Thank you!