ParticipantOriented Evaluation Prepared by Daniel Wagner Jahmih Aglahmie

Participant-Oriented Evaluation Prepared by: Daniel Wagner Jahmih Aglahmie Kathleen Samulski Joshua Rychlicki

Participant-Oriented Evaluation l l l Evaluation approach focusing on the participants Participants have interest in the results Developed in response to the needs and interests of those associated with the evaluation

Characteristics of a Participant. Oriented Approach l l The evaluation is dependent on inductive reasoning The evaluation utilizes data from multiple sources The evaluation does not follow a standard plan The evaluation records multiple point-of -views or perspectives

An Evolved Approach Other evaluation approaches have been adapted into participant-oriented approaches.

Three Theorists Involved in Approach l l l Robert Stake Egon Guba & Yvonne Lincoln Michael Q. Patton

Robert Stake l l First evaluation theorist to introduce Participant Oriented Evaluation into the field of education. Introduced theory of Responsive Evaluation which focuses on “re-directing data gathering and interpretative efforts around emerging issues of importance to program practitioners in the evaluation setting” (Abma, 2005) Created the Countenance Framework in 1967. The model refers to the two faces of evaluation: description & judgment. Description includes the evaluator’s observation and list of benchmarks for the activities being evaluated. Judgment is the evaluator’s overall rating of merit.

Guba & Lincoln l l l Built on Stake’s Responsive Evaluation theory and came up with the method of fourth generation evaluation. This method was used when listing the generations of evaluation. Generations include: - 1 st generation: evaluators measure participants - 2 nd generation: evaluators describe participant - 3 rd generation: evaluators judge participants - 4 th generation: evaluators respond to participants. The evaluation includes 12 steps. 1. Contacting 7. Prioritizing Unresolved Items 2. Organizing 8. Collecting Information 3. Identifying Stakeholders 9. Preparing for Negotiation 4. Developing within-group Joint Constructions 10. Carrying Out Negotiation 5. Enlarging joint stakeholder constrictions 11. Reporting 6. Sorting out Resolved Claims, Concerns, and Issues 12. Recycling

Michael Patton l l l Identified more than 100 evaluations Known for his words: “Research is aimed at truth. Evaluation is aimed at action” He focuses on ‘identifying and working with key participants in organizations who can develop a vision for the value of gathering information and using it to improve the functions of the organization in ways responsive to each situation. ’ (Williams, 2000)

Key Considerations l l l l (Fitzpatrick. Sanders, and Worthen, 2004), report that most proponents of participant-oriented evaluation view the participants in the object of evaluation as key to the evaluation. As opposed to evaluating in a top-down manner, participant-oriented evaluation is conducted with a bottom-up approach Qualitative methods are used much more than quantitative methods. Jacobs (1985) describes that qualitative methods, “seek to describe not the frequency but the behaviors, beliefs, and feelings of the respondents in a setting” (p. 5). Jacobs continues to describe qualitative data as “direct quotations, observations, personal impressions, analyses of records and documents, summaries of unobtrusive measures, case studies, and the like” (pp. 5 -6). Jacobs maintains the naturalistic design process in participantoriented evaluation is emergent and the setting is changeable and natural. As Fitzpatrick, et al. report, “the evaluator is in the role of the learner, and the stakeholders serve as teachers” (p. 149).

Advantages of This Approach l Gain new insights and usable theories – l Flexibility – – – l Addresses user training as a feature of the assessment Consideration to related variables Numerous data collection techniques Empowers stakeholders – l Educational, social or corporate programs Reaches even those who are at times left out of the evaluation process Add a political constituent – Cultivates activism (Fitzpatrick, Sanders, and Worthen, 2004)

Disadvantages of This Approach l Subjective – l l Relies heavily on human observation and personal perspective Expensive Labor-intensive Time-consuming and extensive Evaluator loses objectivity – may become caught up in advocacy trends (Fitzpatrick, Sanders, and Worthen, 2004)

Implementation of Participant. Oriented Evaluation When to use l When cost is not an issue l When time allows l When responding to stakeholders l When evaluator can take on the role of the learner When not to use l When detailed quantitative data is essential l When there is “controlled, laboratory or contrived setting” (Jacobs, 1985).

Scenarios for Use of Participant. Oriented Evaluation l Social reform – l Education – – – l Evaluating new and existing training programs Government – l Evaluating new and existing instructional programs on all levels Evaluating instructional materials Evaluating professional development Corporations – l Evaluating programs developed to improve the environment Evaluating programs Whenever qualitative formative evaluation is conducted

References Abma, T. A. (2005). Responsive Evaluation and Program Planning: Online version. Retrieved September 28, 2007 from World Wide Web http: //64. 233. 167. 104/search? q=cache: Dy. CD 4 R_To. E 0 J: mailer. fsu. e du/~sullivan/SEA_Newsletter/Responsive_Evaluation. pdf+what+is+r esponsive+evaluation&hl=en&ct=clnk&cd=6&gl=us Fitzpatrick, J. , Sanders, J. , & Worthen, B. (2004). Program Evaluation: Alternative Approaches and Practical Guidelines, (3 rd Ed. ), Boston: Allyn & Bacon. Jacobs, R. L. (1985). Naturalistic Inquiry and Qualitative Methods: Implications for Training and Development. Columbus, OH: Ohio State University. (ERIC Document Reproduction Service No. ED 305518). Williams, D. D. (2000). Evaluation of learning objects and instruction using learning objects. In D. A. Wiley (Ed. ), The Instructional Use of Learning Objects: Online Version. Retrieved September 27, 2007, from the World Wide Web: http: //reusability. org/read/chapters/williams. doc