EVAL 6000 Foundations of Evaluation Dr Chris L

EVAL 6000: Foundations of Evaluation Dr. Chris L. S. Coryn Nick Saxton Fall 2014

Agenda • Quasi-evaluation studies • Activity (if time allows)

Quasi-evaluation studies

Quasi-evaluation studies • Address specific questions (often employing a wide range of methods) • Advocate use a particular method • Whether the questions or methods are appropriate for assessing merit and worth is a secondary consideration • Both are narrow in scope and often deliver less than a full assessment of merit and worth

Approach 7: Objectives-based studies • Advance organizers – Statement of program objectives • Purposes – To determine to what extent a program achieved objectives • Sources of questions – Objectives as defined by staff, funder, or evaluator • Questions – To what extent were each of the stated objectives met?

Objectives-based evaluation results from a national research center

• Methods – Any relevant method for determining to which operationally defined objectives were met • Pioneers – Ralph Tyler • Use considerations – Must have clear, supportable objectives • Strengths – Ease of application • Weaknesses – Narrowness and inability to identify positive and negative side effects

Approach 7: The success case method • Advance organizers – Comparison between successful and unsuccessful instances • Purposes – To determine how well and in what respects a program is ‘working’ • Sources of questions – Generally from program providers • Questions – What are the noteworthy successes? – How were successes produced? – What factors contributed to success/failure?

• Methods 1. 2. 3. 4. Focus and plan the study Create an impact model Survey all participants Interview a sample of success and nonsuccess cases 5. Communicate findings, conclusions, and recommendations • Pioneers – Robert Brinkerhoff • Use considerations – Intended to assist service providers in increasing ‘successes’ and decreasing ‘nonsuccesses’ • Strengths – Ease of application – Use for improvement • Weaknesses – Narrowness of scope

Approach 9: Outcome evaluation as value-added assessment • Advance organizers – System-wide measures of ‘growth’ or ‘gains’ • Purposes – ‘Value added’ by a program and its constituent parts • Sources of questions – Oversight bodies • Questions – What parts of a program contribute most to ‘growth’ or ‘gains’?

• Methods – Gain score analysis, hierarchical linear modeling, etc. • Pioneers – Raudenbush, Sanders, Horn, Timms, etc. • Use considerations – Can be used to make and/or support policy decisions • Strengths – Longitudinal rather than cross-sectional • Weaknesses – Potential misuse by policy makers in placing ‘blame’

Approach 10: Experimental and quasi-experimental studies • Advance organizers – Cause-and-effect hypotheses, competing treatments, etc. • Purposes – To determine causal relationships between independent and dependent variables • Sources of questions – Researchers, developers, policy makers, etc. • Questions – To what extent is one treatment superior to another?

• Methods – Random or other method of assignment to conditions • Pioneers – Campbell, Cook, Shadish • Use considerations – Addresses only one particular type of question • Strengths – Strong causal conclusions (if assumptions are met) • Weaknesses – Requires substantial expertise, time, money, etc.

Approach 11: Cost studies • Advance organizers – Costs associated with program inputs, outputs, and outcomes • Purposes – The costs and outcomes of one more alternatives • Sources of questions – Policy makers, planners, taxpayers, etc. • Questions – What are the costs of obtaining desired outcomes?

• Methods – Analysis of monetary and nonmonetary units • Pioneers – Levin, Mc. Ewin, Yates, etc. • Use considerations – Expertise required • Strengths – ‘Bottom line’ conclusions of interest to most decision makers • Weaknesses – Often difficult to validly execute

Approach 12: Connoisseurship and criticism • Advance organizers – Specialized expertise, sensitivities, tacit knowledge, etc. • Purposes – To describe, appraise, and illuminate • Sources of questions – Determined by the ‘connoisseurs’ or ‘critics’ • Questions – What are a program’s salient characteristics, strengths, and weaknesses?

• Methods – Perceptual sensitivity, prior experience, refined insights, etc. • Pioneers – Elliot Eisner • Use considerations – An audience willing to accept the approach • Strengths – Exploitation of refined expertise • Weaknesses – Objectivity and reliability

Approach 13: Theory-based evaluation • Advance organizers – A carefully specified ‘theory’ of how a program is intended to operate • Purposes – To determine the extent to which a program is ‘theoretically sound’ • Sources of questions – Determined by the guiding program theory • Questions – To what extent does the program theory ‘work’ or not?

• Methods – Any method appropriate for testing the program theory • Pioneers – Chen, Donaldson, Weiss, Rogers, Rossi, etc. • Use considerations – Difficulty in applying the approach • Strengths – Useful for determining potential ‘measurement’ variables • Weaknesses – Few programs are grounded by validated ‘theories’

Approach 14: Meta-analysis • Advance organizers – Sufficient studies of the same or similar programs • Purposes – To assemble and (statistically) integrate findings from multiple studies of the same or similar programs • Sources of questions – Policy makers, ‘research repositories’, etc. • Questions – What is the average effect of a particular type of program?

• Methods – Statistical methods for integrating study results (varies widely) • Pioneers – Glass • Use considerations – Major source of contemporary ‘best practices’ across a variety of domains • Strengths – Evidence of effectiveness over units, treatments, observations, and settings • Weaknesses – Exclusive emphasis on program outcomes

Activity

Activity • We will split the class into two sections (1 and 2) – In each section, appoint one student to chair your appointed group – Each member of section 1 should select one of the approaches discussed today and discuss why it is useful – Members of section 2 should listen and take notes – Members of section 2 should then outline weaknesses of the selected approaches – Finally, the chair of each group should discuss the potential strengths, weaknesses, and utility of the selected approaches

Encyclopedia Entries • • • Bias Causation Checklists Chelimsky, Eleanor Conflict of Interest Countenance Model of Evaluation Critical Theory Evaluation Effectiveness Efficiency Empiricism Independence • Evaluability Assessment • Evaluation Use • Fournier, Deborah • Positivism • Relativism • Responsive evaluation • Stake, Robert • Thick Description • Utilization of Evaluation • Weiss, Carol • Wholey, Joseph