Understanding Evaluation Definition of Evaluation Systematic determination of

Understanding Evaluation

Definition of Evaluation Systematic determination of the quality or value of something (Scriven, 1991) ¥ What can we evaluate? ¥ ¥ ¥ ¥ Projects, programs, or organizations Personnel or performance Policies or strategies Products or services Proposals, contract bids, or job application Almost everything in our daily life because before you make decision, you do the evaluation first. Comparison is a kind of evaluation. Scriven, M. (1991). Evaluation thesaurus (4 th ed. ). Newbury Park, CA: Sage

Terminology (Davison, Glossary) ¥ Evaluand That which is being evaluated (e. g. Program, policy, project, product, service, organization, almost everything) ¥ In personnel evaluation the term is evaluee ¥ S 519

Issues of Evaluation ¥ Evaluation is for ¥ ¥ ¥ Find areas for improvement Generate an assessment of overall quality Answer the question of „Merit“ or „Worth“ (Scriven, 1991) ¥ ¥ Merit is the „intrinsic“ value of something = „quality“ Worth is the value of something to an individual, an organization, an institution – contextualized merit -- = „value“ S 519

Choosing the right group ¥ Accountability ¥ evaluation It is important to conduct an independent evaluation ¥ i. e. Nobody on the evaluation team should have a significant vested interest in whether the results are good or bad S 519

The steps involved (D-p 4) ¥ Step 1: understanding the basic about evaluation (ch 1) ¥ Step 2: defining the main purposes of the evaluation and the „big picture“ questions that need answers (ch 2) ¥ Step 3: Identifying the evaluative criteria (ch 3) ¥ Step 4: Organizing the list of criteria and choosing sources of evidence (collecting data) (ch 4) S 519

The steps involved (D-p 4) ¥ Step 5: analyzing data dealing with the causation/correlation issue (which cause what, why), to avoid „subjectivity“ (ch 5+6) ¥ importance weighting: weight the results (ch 7) ¥ ¥ Meric determination: how well your evaluand has done on the criteria (good? Unacceptable? ) (ch 8) Synthesis methodology: systematic methods for condensing evaluation findings (ch 9) Staticistical analysis: Salkind (2007) S 519

The steps involved (D-p 4) ¥ Step 6: ¥ Putting it all together: fitting the pieces into the KEC framework (ch 10) ¥ Step 7: ¥ result feedback Meta-evaluation: how to figure out whether your evlauation is any good (ch 11) S 519

The Key Evaluation Checklist (Davidson, 2005, p. 6 -7) I. Executive Summary II. Preface 1. Background 2. Descriptions & Context & Definitions 6. Process Evaluation 7. Outcome Evaluation III. Methodology 3. Consumers 4. Resources 8 & 9. Comparative Cost-Effectiveness 5. Values 10. Exportability 11. Overall Significance 12. 13. Responsibilities Recommendations & Explanations S 519 14. Reporting & Follow-up 15. Meta-evaluation

Step 1: Understand the basic of evaluation ¥ ¥ Identify the evaluand Background and context of evaluand ¥ ¥ Descriptions and definitions ¥ ¥ Why did this program or product come into existence in the first place? Describe the evaluand in enough detail so that virtually anyone can understand what it is and what it does How: collect background information, pay a firsthand visit or literature review S 519

Are you ready for your first evaluation project? ¥ Some tips before you start Make sure that your evaluand is not difficult to access (geolocation, inanimate objects) ¥ Make your evaluand a clearly defined group (avoid abstract and complex system) ¥ Avoid political ramification (assess your boss pet project, university administration) ¥ To avoid your invovlement in the evaluand (to assess a class which you are teaching, etc. ) ¥ S 519

Previous Projects ¥ ¥ Metadata discussion group, Brown Bag discussion group, Twitter, SLIS website, Information Visulization Lab website, Media and Reserve Services in Wells Library, IU CAT, So. E website, Chemistry library website How about something else related to real-world problems ¥ ¥ ¥ For social good (http: //dssg. io/projects/) Industry and Government talks at KDD 214 (http: //www. kdd. org/kdd 2014/) Maybe this will inspire you with a great project to work on

Standards for Evaluation (Patton, 1997, p. 17) ¥ Utility ¥ ¥ Feasibility ¥ ¥ To ensure that an evaluation will be realistic, prudent, diplomatic, and frugal Propriety ¥ ¥ To ensure that an evaluation will serve the practical information needs of intended users To ensure that an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation, as well as those affected by its results (IRB: http: //research. iu. edu/rschcomp/revlocation. html) Accuracy ¥ To ensure that an evaluation will reveal and convey technically adequate information about the features S 519 http: //www. wmich. edu/evalctr/jc/

Guiding Principles for Evaluators (American Evaluation Association, 1995) ¥ Systematic inquiry ¥ Competence ¥ Integrity/honesty ¥ Respect for people ¥ Responsibilities for general and public welfare S 519

Step 1: Output report ¥ Output: one or two page overview of the evaluand findings What is your evaluand ¥ Background and context of your evaluand ¥ Description of your evaluand ¥ ¥ Try to be as detail as possible S 519

Step 2: Defining the Purpose of the Evaluation (D-Ch 2) ¥ Who asked for this evaluation and why? ¥ What are the main evaluation questions? ¥ Who are the main audience? S 519

Evaluation purposes ¥ A. what is (are) the main purpose(s) of the evaluation? ¥ To determine the overall quality or value of something (summative evaluation, absolute merit) ¥ ¥ i. e. Decision making, funding allocation decision, benchmarking products, using a tool, etc. To find areas for improvement (formative evaluation, relative merit) ¥ ¥ To help a new „thing“ to start To improve the existing „thing“ S 519

Big picture questions ¥ Big ¥ picture questions: B. What is (are) the big picture question(s) for which we need answers? ¥ Absolute merit ¥ ¥ Do we want to invest this project? relative merit ¥ ¥ How does this project compare with the other options? Ranking S 519

Step 2: Output report ¥ Your step 2 output report should answer the following questions: ¥ Define the evaluation purpose ¥ ¥ Do you need to demonstrate to someone (yourself) the overall quality of something? Or Do you need to find a file for improvement? Or do you do both? Once you answer above questions, figure out what are your big picture questions: ¥ ¥ Is your evaluation related to the absolute merit of your evaluand? Or the relative merit of your evaluand S 519