The heart of evaluation influence a compelling evaluative

The ‘heart’ of evaluation influence: a compelling evaluative argument AES Conference 2011 Heather Nunns

Workshop overview • Why evaluative reasoning and argument are important • Defining evaluative reasoning • The origins of evaluative reasoning • ‘Unpacking’ evaluative reasoning • Connecting evaluative reasoning and argument

My contention No matter. . . the sophistication of our evaluation design. . . our attention to methodological rigor. . . our strategies and tools to encourage evaluation use, an evaluation will lack impact without sound evaluative reasoning and argument

Evaluative reasoning defined “The systematic means for arriving at evaluative conclusions, the principles that support inferences drawn by evaluators” Fournier, 1995, p. 1 “. . . professionalised evaluation has spent much of its time and effort on developing methodological sophistication and less so on logical sophistication. Understanding the reasoning process used to establish evaluative conclusions. . . has to be the field’s greatest unmet challenge” Fournier, 1995, p. 1.

Values Value (noun): – mental pictures of idealised state of affairs or models of idealised behaviour considered by a person to be of worth Value (verb): – the act of making a reflective judgment of worth (as in ‘valuing’) – to value something is to take up a certain positive attitude towards it Magendanz, 2003

Values are at the heart of evaluation Evaluation = judgment of the merit (quality), worth (value), significance (importance) of an evaluand House 1999, Scriven 1991, Stufflebeam 2001 Evaluators are in the business of assessing value of some sort: “How responsive is the programme delivery. . ? ” “Is X or Y the more effective approach. . . ? ” “Which parts of the programme are worth replicating. . . ? ”

Western philosophy: the traditional fact – value distinction VALUES Subjective FACTS Objective Able to be verified Able to be studied empirically Personal preference Unable to be verified Unable to be the subject of systematic empirical enquiry

The origins of evaluative reasoning • Axiology = a branch of western philosophy focused on the study of values (early 20 th century) • Asserted that it is possible to draw ‘logical’, objective conclusions about values Richard Hare (1919 -2002), Paul W. Taylor (? ), Nicholas Rescher (1928 - )

How to assess value Hare 1967, Rescher 1969, Taylor 1961 1. Identify the object (X) and the value to be applied to the object 2. Identify the class of comparison to which X belongs (Z) 3. Identify norms for Z 4. Develop a set of ‘operational’ statements describing levels of performance for each of the norms of Z 5. Determine the characteristic(s) that X has 6. Compare X’s characteristics with the operational statements to come to an evaluative conclusion 7. Justify the norms and standards used

Michael Scriven Over 100 publications about evaluation in 44 years Evaluation Thesaurus, 1991 Reasoning, 1976

General logic of evaluation Scriven 1967, 1980, 1991, 1994 Making an evaluative judgment involves: 1. establishing criteria of merit for the evaluand 2. constructing standards for the criteria 3. measuring performance of the evaluand against the criteria 4. synthesising and integrating data into a judgment of merit or worth

Comparing evaluation approaches Hare (1967), Rescher (1969), Taylor (1961) • • Identify the object (X) and the value to be applied to the object Identify the “class of comparison” to which X belongs (Z) Identify norms for Z Develop a set of operational statements describing levels of performance for each of the norms of Z Determine the characteristic(s) of X (the “good making characteristics”) Compare X’s characteristics with the operational statements above to come to an evaluative conclusion Justify the norms used • Scriven’s logic of evaluation (1967, 1980, 1991, 1994) Establish criteria of merit for the evaluand • Construct standards for the criteria • Measure performance of the evaluand against the criteria Synthesise and integrate data into a judgment of merit or worth •

EVALUATION (GENERAL) LOGIC Scriven Explanation EVALUAND

Working logic, Fournier 1995 How the general evaluation logic is applied in practice depends on the: • phenomenon: what is being evaluated • problem: what is the ‘problem’ being investigated • question: what question(s) are being asked • claim: what kind of claims(s) are being made

Examples of working logics, Product evaluation Programme evaluation Fournier 1995 Policy evaluation 1. Phenomenon A product A programme A policy intervention 2. ‘Problem’ Performance effectiveness How well the Effectiveness of the implementation of the policy intervention programme is working 3. Question(s) Is X a good one of its kind? Is X better/less better than others of its kind? Is X value for money? Is the implementation working as intended? What improvements to process are required? 4. Evaluative claim Performance and value Process/ implementation performance What are the outcome(s) of the policy intervention? How valuable are these outcomes to participants? Causal and value

EVALUATION (GENERAL) LOGIC Scriven WORKING LOGIC Fournier EVALUAND Context Values pluralism (Greene, Kushner, Lincoln & Guba, Schwandt)

EVALUATION (GENERAL) LOGIC Scriven WORKING LOGIC Fournier EVALUAND + VALUES criteria & standards identified Context Values pluralism (Greene, Kushner, Lincoln & Guba, Schwandt)

Thinking further about ‘valuing’ “Ethical discussion aims at making us more critically aware of what we are doing. It brings us back to thinking about what it is to be a good evaluator, and to ask in whose interests should we be acting and for what purpose? These are ethical questions, and they should take precedence over technical questions about how to do evaluation. ” Tom Schwandt, Evaluation Practice Reconsidered, 2002, p. 154

Valuing approaches: Schwandt, 1997 • Analytical value-free approaches: Evaluators should not make value judgments. They can describe the value positions of stakeholders and participants but not make a judgment about which is best e. g. Shadish, Cook & Leviton, 1991 • Emancipatory value-committed approaches: Evaluation practice should challenge values that serve existing power inequalities and be linked to political action e. g. Mertens, 2010 • Value-critical approaches: Evaluators use their expertise to add to and encourage practitioners’ reflective, conversational critiques of the value commitments embedded in their practice e. g. Patton’s developmental approach

EVALUATION (GENERAL) LOGIC Scriven WORKING LOGIC Fournier EVALUAND + VALUES criteria & standards identified Valuing approaches (House & Howe, Schwandt, Shadish, Cook & Leviton, Stake) Context Values pluralism (Greene, Kushner, Lincoln & Guba, Schwandt)

EVALUATION (GENERAL) LOGIC Scriven WORKING LOGIC Fournier EVALUAND + VALUES criteria & standards identified + Evidence Valuing approaches (House & Howe, Schwandt, Shadish, Cook & Leviton, Stake) Context Values pluralism (Greene, Kushner, Lincoln & Guba, Schwandt)

Surprises, puzzles, comments, questions. . .

EVALUATION (GENERAL) LOGIC Scriven WORKING LOGIC Fournier PROBATIVE INFERENCE Scriven EVALUAND + VALUES criteria & standards identified + Evidence EVALUATIVE JUDGMENT Valuing approaches (House & Howe; Schwandt; Shadish, Cook & Leviton; Stake) Context Values pluralism (Greene, Kushner, Lincoln & Guba, Schwandt)

EVALUATION (GENERAL) LOGIC Scriven WORKING LOGIC Fournier PROBATIVE INFERENCE Scriven EVALUAND + VALUES criteria & standards identified + Evidence EVALUATIVE JUDGMENT Evaluative argument House Valuing approaches (House & Howe, Schwandt, Shadish, Cook & Leviton, Stake) Context Values pluralism (Greene, Kushner, Lincoln & Guba, Schwandt)

Values in evaluation and social research, 1999 Evaluating with validity, 2010 Professional Evaluation: Social impact and political consequences, 1993 Ernie House

House, 1977 Evaluation as argumentation: “. . . evaluation persuades rather than convinces, argues rather than demonstrates, is credible rather than certain, is variably accepted rather than compelling” p. 6

Evaluative argument – why is it important? “My concern is that in the press to master methods of generating data, we ignore the idea of developing a warranted argument – a clear chain of reasoning that connects the grounds, reasons or evidence to an evaluative conclusion” Schwandt, 2008. “Worrying about warrant is a core evaluator responsibility. It is because our inferences are consequential that we must have confidence that they are warranted” Greene, 2011.

An evaluative argument • Toulmin et al (1979) has identified a logic of reasoning underpinning all types of inquiry that aims to build an argument. This logic consists of six main features which include claims, evidence, warrants, backings. • A warrant is the “because” part of an argument. It legitimates the inference from the claim and evidence to the conclusion by appealing to an appropriate authority. • Backings are added authority as to why the warrant should be accepted as legitimising the inference.

EVALUATION (GENERAL) LOGIC Scriven WORKING LOGIC Fournier PROBATIVE INFERENCE Scriven EVALUAND + VALUES criteria & standards identified Valuing approaches (House & Howe, Schwandt, Shadish, Cook & Leviton, Stake) + Evidence EVALUATIVE JUDGMENT Evaluative argument House Warrants & backings Toulmin Context Values pluralism (Greene, Kushner, Lincoln & Guba, Schwandt)

and finally. . . the evaluative judgment Our end point: an evaluative judgment that is legitimate and justified (Fournier, 1995) But ! the synthesis debate (House, Scriven, Stake)

EVALUATION (GENERAL) LOGIC Scriven WORKING LOGIC Fournier PROBATIVE INFERENCE Scriven EVALUAND + VALUES criteria & standards identified + Evidence EVALUATIVE JUDGMENT Evaluative argument House Warrants & backings Toulmin Valuing approaches (House & Howe, Schwandt, Shadish, Cook & Leviton, Stake) Context Values pluralism (Greene, Kushner, Lincoln & Guba, Schwandt)

Facts and values: an updated perspective Old perspective: VALUES FACTS Updated perspective: “Brute” facts (House & Howe, 1999) e. g. Stone is denser than cheese “Bare” values e. g. Pinot Gris tastes better than chardonnay Statements that blend facts and values together “Evaluative statements consist of fact and value claims intertwined, melded together” House and Howe, 1999

Comments, feedback: heather@analyticmatters. co. nz

Recommended reading Evaluation logic Fournier, D. (1995). Establishing Evaluative Conclusions: A Distinction between General and Working Logic. New Directions for Evaluation 68, 15 -32 House E. R. (1977). The logic of evaluative argument. Centre for the study of evaluation. University of California Values and valuing Davidson, E. J. (2005). Evaluation methodology basics. Thousand Oaks: Sage House E. R. & Howe, K. R. (1999). Values in evaluation and social research. Thousand Oaks: Sage Schwandt, T. A. (2002). Evaluation Practice Reconsidered. New York: Peter Lang Publishing Argumentation Toulmin, S. , Rieke, R. & Janik, A. (1979). An introduction to reasoning. New York: Macmillan Publishing Evaluative argument Fournier, D. , & Smith, N. L. (1993). Clarifying the Merits of Argument in Evaluation Practice. Evaluation and Program Planning, 16, 315 -323 Schwandt, T. A. (2008). Educating for Intelligent Belief in Evaluation. American Journal of Evaluation 29, 139 Evidence Donaldson, S. T. , Christie, C. A. & Mark, M. M. (Eds. ), (2009). What counts as credible evidence in applied research and evaluation practice. Thousand Oaks: Sage