35 Years of Evaluation Learning in 5 Minutes

35 Years of Evaluation Learning in 5 Minutes David J. Bernstein, Ph. D. Owner, DJB Evaluation Consulting 2018 American Evaluation Association Conference Cleveland, OH November 1, 2018

35 Years in 5 Minutes? That’s…. 2

Accountability © United Features Syndicate, 3/13/99 3

Accountability External Processes The Public: Voters, Taxpayers, Citizens, Clients Internal & External Processes Strategic Plans Elections Performance Measurement Public Budget Documents Program Evaluations/Audits Internal Processes Elected & Appointed Officials Internal Management Reports Budget & Decision Documents Personnel Operations Policy Analysis Program Managers Government Staff 4

PERFORMANCE MEASUREMENT SYSTEMS Montgomery County, MD Department of Finance FEWER NUMBER OF P E R F O R M A N C E M E A S U R E S GREATER External Reporting of Finance Performance Measures Executive Use of Finance Performance Measures for Decision Making Departmental Use of Finance Performance Measures Finance Department Divisional Use of Program Indicators to Manage Programs EXTERNAL ACCOUNTABILITY: Report Outcomes & Efficiency USE OF P E R F O R M A N C E M E A S U R E S INTERNAL MANAGEMENT: Program/Resource Management Employee Appraisal 5

Benchmarking and Performance Measurement 1. 2. 3. 4. 5. Legal Standard Historical Standard Strategic Standard Peer-based Standard Best Practices Standard 6

PM Versus Evaluation “You can measure without evaluating, but you cannot evaluate without measuring. ” © David J. Bernstein, Ph. D. 7

Performance Measurement vs. Evaluation 1. Temporal Order Performance Measurement 2. Co-Variation 3. Non-spuriousness (Elimination of extraneous variables, aka internal validity) EVALUATION 8

Intended/Unintended Users and Uses Unintended Uses Intended Uses Unintended Users Trouble Success Intended Users Need More Technical Assistance and Training WINNER: Utilization -focused (Patton) 9

Patton and Epstein on “Use” Patton Epstein Primary Uses of Evaluation Uses of Performance Measurement 1. Judge Merit and Worth 1. Improve Public Accountability/ Communication 2. Improve Programs 2. Improve Service Performance 3. Knowledge Generation 3. Improve Decision Making Patton, 2012 Epstein, 1988 10

Evaluation Policy + = Federal Evaluation Policy 11

Paperwork Reduction Act 1990 s, National Performance Review 12

Logic Model-Kellogg Foundation, 2004 13

Theory of Change - Not a Logic Model 14

Logic Chart - NOT a Logic Model, but It’s Close 15

Logic Model 16

Are Random Control Trials the Gold Standard? http: //www. wikipaintings. org/en/gerrit-dou/weighing-gold-1664 17

Evaluation, Implementation, Recommendations Source: http: //dilbert. com/strips/comic/2011 -01 -14/ 18

Mentors/Friends Joe Wholey Michael Quinn Patton Rakesh Mohan Kathy Newcomer (and me) 19

For all your evaluation needs, please contact… David J. Bernstein, Ph. D. DJB Evaluation Consulting djbeval@gmail. com 20