A brief overview What is program evaluation How

A brief overview What is program evaluation? How is an evaluation conducted? When should it be used? When can it be used? Used with Permission of: John R. Slate

Presentation Outline n n n Definitions Purposes Types Key concepts of evaluative research Research designs Requirements for program evaluation

A Definition of “Program” “An organized set of resources and activities directed toward a common purpose or goal”

Two Definitions of “Program Evaluation” n “. . . an assessment, through objective measurement and systematic analysis, of the manner and extent to which Federal programs achieve intended objectives. ” (Source - Government Performance and Results Act (GPRA)) n “The application of scientific research methods to assess program concepts, implementation and effectiveness” (Source - General Accounting Office, Designing Evaluations, Report GAO/PEMD-10. 1. 4)

Where Does Evaluation Fit Into a Program’s Planning, Development and Implementation Process? Make decision to create program and set strategic direction 5

Where Does Evaluation Fit Into a Program’s Planning, Development and Implementation Process? Make decision to create program and set strategic direction Determine what the program will do and how it will do it , set targets and program objectives 6

Where Does Evaluation Fit Into a Program’s Planning, Development and Implementation Process? Make decision to create program and set strategic direction Determine what the program will do and how it will do it , set targets and program objectives Create program infrastructure & management, administrative & information systems, develop performance measures 7

Where Does Evaluation Fit Into a Program’s Planning, Development and Implementation Process? Make decision to create program and set strategic direction Determine what the program will do and how it will do it , set targets and program objectives Determine required levels of human resources and material support Create program infrastructure & management, administrative & information systems, develop performance measures 8

Where Does Evaluation Fit Into a Program’s Planning, Development and Implementation Process? Make decision to create program and set strategic direction Implement action plan (Program operation & management, performance measurement, corrective action, etc. . . , . ) Determine required levels of human resources and material support Determine what the program will do and how it will do it , set targets and program objectives Create program infrastructure & management, administrative & information systems, develop performance measures 9

Where Does Evaluation Fit Into a Program’s Planning, Development and Implementation Process? Program effectiveness, impact, efficiency evaluations to determine the continue need for the program, alter program design, resource requirements, etc. . Make decision to create program and set strategic direction Implement action plan (Program operation & management, performance measurement, corrective action, etc. . . , . ) Determine required levels of human resources and material support Determine what the program will do and how it will do it , set targets and program objectives Create program infrastructure & management, administrative & information systems, develop performance measures 10

Types of Program Evaluations Formative - Judging the worth of a program while activities are forming or in process • focus is on the process more than the outcome • can help make inprocess improvements • often involves a small scale field test n 11

Types of Program Evaluations Formative - Judging the worth of a program while activities are forming or in process • focus is on the process more than the outcome • can help make inprocess improvements • often involves a small scale field test n Summative - Judging the effectiveness of a fully operating or completed program • focus in on outcome and overall program worth • can help decisions to expand, terminate, modify • usually encompasses the entire program n 12

Questions Asked by Program Evaluations* n Descriptive - Statistics on inputs, outputs, and outcomes; how does the program work? n Normative - What is the expected performance (goal) of the program in relation to actual achievement? n Impact - If a goal is not met, why not? Program evaluations must establish a cause/effect relationship between an unmet goal and program activities or other, external factors *from Designing Evaluations, GAO/PEMD-10. 1. 4

Decisions Program Evaluation Can Help Make n Continue or discontinue a program n Improve policies and procedures n Add or drop specific program elements n Institute similar programs elsewhere n Allocate resources n Accept or reject approaches & theories

Key concepts of Evaluative Research According to standards in the research and academic communities, program evaluations should strive for scientific proof and method, i. e. . , they should be: n n n empirical - based on valid, reliable data replicable - study can be repeated in exactly the same way in another time, place or setting falsifiable - hypothetical cause/effect relationships can be demonstrated or not

Evaluation Methodology (highly simplified) n Define program n Identify outcome goals and objectives n Model hypothetical cause/effect relationships between program activities and outcomes n Develop goal/objective measurement criteria and desired achievement levels n Locate, collect and analyze data on program participants (and maybe control group) n Compare actual results with target levels

Evaluation Strategies and Designs* *Based on Designing Evaluations, GAO/PEMD-10. 1. 4 17

Evaluation Strategies and Designs* *Based on Designing Evaluations, GAO/PEMD-10. 1. 4 18

Evaluation Strategies and Designs* *Based on Designing Evaluations, GAO/PEMD-10. 1. 4 19

Evaluation Strategies and Designs* *Based on Designing Evaluations, GAO/PEMD-10. 1. 4 20

Evaluability Assessment n n n A study to determine if, when, and how a program can be evaluated Prevents premature impact and outcome evaluations Usually involves: • clarification of the intent of the program • formulation of “testable” cause/effect statements • determines what can be measured and how • assesses validity, reliability, and relevance of data on which the evaluation would based

Circumstances When Program Evaluation is Not a Good Idea n When there is no question about the program n When there is no clear program structure or focus n When program activities cannot be distinguished from other activities n When people cannot agree on what the program is trying to achieve n When valid measurement methods and data do not exist and cannot be created n When the study would serve no useful purpose

Overview Program Evaluation Prerequisites n n n n An operating “program”, i. e. . , a distinct set of activities and resources with a common purpose and focus Agreement on program goals and objectives Agreement on program metrics for its goals and objectives Agreement on what constitutes program “success” Existence, or ability to develop, valid data Absence of legal, administrative, or cultural barriers to the study Agreement on intended use of the study