Review Alternative Assessments I l Describe the two

Review: Alternative Assessments I l Describe the two epistemologies in ch. 3 (o/s) l Compare the two principles for assigning value (util/int-pl) l Identify pros/cons of the two evaluation approaches we discussed last week

Alternative Approaches to Evaluation II Dr. Suzan Ayers Western Michigan University (courtesy of Dr. Mary Schutten)

Consumer-Oriented Approach l Typically a summative evaluation approach l This approach advocates consumer education and independent reviews of products l Scriven’s contributions based on groundswell of federally funded educational programs in 1960 s l Differentiation between formative/summative eval.

Consumer-Oriented Checklist (Scriven, 1974, p. 102) l l l Need Market Performance l l l l l True field trials [tests in a “real” setting] True consumer tests [tests with real users] Critical comparisons [comparative data] Long term [effects over the long term] Side effects [unintended outcomes] Process [product use fits descriptions] Causation [experimental study] Statistical significance [supports product effectiveness] Educational significance

Cost effectiveness l Extended support [in service training] Producer’s efforts to meet these standards improve product effectiveness l Key Evaluation Checklist developed to evaluate program evaluations l Educational Products Information Exchange (EPIE): Independent product-reviewer service l Curriculum Materials Analysis System (CMAS) checklist: Describe product, analyze rationale, consider: antecedent conditions, content, instructional theory & teaching strategies, form overall judgments l

Uses of Consumer-Oriented Evaluation Approach l Typically used by gov’t. agencies and consumer advocates (i. e. , EPIE) l What does one need to know about a product before deciding whether to adopt or install it? l Process information l Content information l Transportability information l Effectiveness information

Consumer-Oriented Pros/Cons l Strengths: valuable info given to those who don’t have time to study, advance consumers’ knowledge of appropriate criteria for selection of programs/products l Weaknesses: can increase product cost, stringent testing may “crimp” creativity, local initiative lessened b/c of dependency on outside consumer services

Consumer-Oriented Qs l What educational products do you use? l How are purchasing decisions made? l What criteria seem to most important in the selection process? l What other criteria for selection does this approach suggest to you?

Expertise-Oriented Approach l Depends primarily upon professional expertise to judge an institution, program, product, or activity l This is the first view that relies heavily on subjective expertise as the key evaluation tool l Examples: doctoral exams, board reviews, accreditation, reappointment/tenure reviews etc…

Expertise-Oriented Types l Formal Review Systems (accreditation) l Existing structure, standards exist, set review schedule, experts, status usually affected by results l Informal Review systems (grad S committee) l Existing structure, no standards, infrequent schedule, experts, status usually affected l Ad hoc panel review (journal reviews) l Multiple l opinions, status sometimes affected Ad hoc individual review (consultant) l Status sometimes affected

Expertise-Oriented Pros/Cons l Strengths: those well-versed make decisions, standards are set, encourage improvement through self-study l Weaknesses: whose standards? (personal bias), expertise credentials, can this approach be used with issues of classroom life, texts, and other evaluation objects or only with the bigger institutional questions?

Expertise-Oriented Qs l What outsiders review your program or organization? l How expert are they in your program’s context, process, and outcomes? l What are characteristics of the most/least helpful reviewers? (list brainstorms on board)

Participant-Oriented Approach l Heretofore, the human element was missing from program evaluation l This approach involves all relevant interests in the evaluation l This approach encourages support for representation of marginalized, oppressed and/or powerless parties

Participant-Oriented Characteristics l Depend in inductive reasoning [observe, discover, l Use multiple data sources [subjective, objective, l Do not follow a standard plan [process evolves as l Record multiple rather than single realities [e. g. , understand] quant, qual] participants gain experience in the activity] focus groups]

Participant-Oriented Examples l Stake’s Countenance Framework l Description l and judgment Responsive Evaluation l Addressing stakeholders’ concerns/issues l Case studies describe participants’ behaviors l Naturalistic Evaluation l Extensive observations, interviews, documents and unobtrusive measures serve as both data and reporting techniques l Credibility vs. internal validity (x-checking, triangulation) l Applicability vs. external validity (thick descriptions) l Auditability vs. reliability (consistency of results) l Confirmability vs. objectivity (neutrality of evaluation)

l Participatory Evaluation l Collaboration between evaluators & key organizational personnel for practical problem solving l Utilization-Focused Evaluation l Base l all decisions on how everything will affect use Empowerment Evaluation l Advocates for societies’ disenfranchised, voiceless minorities l Advantages: training, facilitation, advocacy, illumination, liberation l Unclear how this approach is a unique participantoriented approach l Argued in evaluation that it is not even ‘evaluation’

Participant-Oriented Pros/Cons l Strengths: emphasizes human element, gain new insights and theories, flexibility, attention to contextual variables, encourages multiple data collection methods, provides rich, persuasive information, establishes dialogue with and empowers quiet, powerless stakeholders l Weaknesses: too complex for practitioners (more for theorists), political element, subjective, “loose” evaluations, labor intensive which limits number of cases studied, cost, potential for evaluators to lose objectivity

Participant-Oriented Qs l What current program are you involved in that could benefit from this type of evaluation? l Who are the stakeholders?

Alternative Approaches Summary Five cautions about collective evaluation conceptions presented so far 1) Writings in evaluation are not models/theories l Evaluation is a transdiscipline (not yet a distinct discipline) l “Theoretical” underpinnings in evaluation lack important characteristics of most theories l Information shared is: sets of categories, lists of things to think about, descriptions, etc.

2) “Discipleship” to a single ‘model’ is dangerous l Use of different approaches as heuristic tools, each appropriate for the situation, recommended 3) Calls to consolidate evaluation approaches into a single model are unwise l These efforts based in attempts to simplify evaluation l Approaches are based on widely divergent philosophical assumptions l Development of a single omnibus model would prematurely close a divergent phase in the field l Just because we can does not mean we should; would evaluation be enriched by synthesizing the multitude of approaches into a few guidelines?

4) The choice of an evaluation approach is not empirically based l Single most important impediment to development of more adequate theory and models in evaluation 5) Negative metaphors underlying some approaches can cause negative side effects l Metaphors shared in ch. 3 are predicated on negative assumptions in two categories: l Tacitly assume something is wrong in system being evaluated (short-sighted indictment) l Based on assumptions that people will lie, evade Qs or withhold information as a matter of course

Alternative Approaches’ Contributions Approaches shared in ch. 4 -8 influence evaluation practices in important ways l Help evaluators think diversely l Present & provoke new ideas/techniques l Serve as mental checklists of things to consider, remember, or worry about l Alternative approaches’ heuristic value is very high, but their prescriptive value is less so l Avoid mixing evaluation’s philosophically incompatible ‘oil/water’ approaches; eclectic use of alternative approaches can be advantageous to high-quality evaluation practices Table 9. 1

Exercise l Clearly identify your evaluand l Is it a program, policy, product, service, other? l Who does it (or should it) serve? l Who is in charge of it? l Find a partner and explain what you have written l Does it make sense? l Does it match what you wrote? l Does it avoid specifying criteria? l Is it simple enough? l Did you avoid commenting on the merits of the evaluand?