- Slides: 22
S 519: Evaluation of Information Systems Analyzing data: Merit Ch 8
Merit determination l Keep in mind of what the evaluation is. l l Evaluation is the systematic determination of the quality or value of something (Scriven, 1991). Today we will discuss how to determine the importance of evaluand components or dimensions
Merit determination l It is the process of setting „standards“ (definitions of what performance should constitute „satisfactory“, „good“, etc. ) and applying those standards to descriptive data to draw explicitly evaluative conclusions about performance on a particular dimension or component.
Two steps l l Step 1: defining what constitutes poor, adequate, good, very good and excellent performances on a particular dimension (or component) Step 2. using this definition to convert empirical evidence into evaluative conclusions (e. g. something explicit about quality or value)
Determing merit Decscriptive facts about performance Quality or value determination guide Evaluative conclusions
Usingle quantitaive measure l In a simple case, performance is measured on a single quantitative dimension l The quality or value determination guide is just a set of cutoffs l l l E. g. , >90%=A/excellent, 80%-89%=B/good, 70%79%=C/adequate E. g. , satisfactory/unsatisfactory Difficulty: where to put the cutoff score and how to compare with different systems?
Exercise l School grading system l l USA: A(>90%), B(80 -90%), C(70 -79%), D(60 -69%), F(<60%) New Zealand: A(>80%), B(65 -79%), C(50 -64%), D(3549%), F(<35%). Does that mean in New Zealand, it is easier to get A? Can this way of grading ensure objectivity and consistency of grading across courses? If yes, why? If no, why not?
Using qualitative or multiple measures l l l Using a single measure is not generally good practice. When using multiple measures, it is tricky on how to merge them together to come out with conclusion. See table 8. 1
Experience l l Do not try to go for high precision It is perfectly appropriate to give an answer that still has a certain amount of fuzziness or uncertainty associated with it. Please do not oversell the precision of your work Providing a well-supported broad-brush answer to an important question is not a bad idea.
Rubric l l Rubric is a tool that provides an evaluative description of what performance or quality „looks like“. It has two levels: l l Grading rubric is used to determine absolute quality or value (e. g. , Table 8. 2) Ranking rubric is used to determine relative quality or value
Rubric for absolute value l Rubric for “grading” is based on: l l l Discussion with domain expert Discussion with upstream stakeholders Existing rules (scope of duties) or literatures Evaluand expectations (needs assessment) Evaluation context (job market, current situation)
Sample grading rubric 1 l l l Table 8. 3 Using Table 8. 3 to grade Table 8. 1 – what is grade for this master program? Why? Form a pair and discuss your point
Sample grading rubric 2 l Table 8. 4 provides you a better grading rubric l l l Scope Performance indicators Scores (1 -5)
Exercise l Develop a grading rubric for your evaluation project l Take Table 8. 4 as example l Please include scope, performance indicators and ranking description
Using rubric for determining „relative“ merit l Relative metric is important for experiment that uses a control or comparison group. l l Student scores are interpreted by comparison with other similar schools It simply tells us how the person or program did relative to peers or competitors.
Using rubric for determining „relevative“ merit Score falls in Grade assigned Top 10% A Next 20% B Next 50% C Next 15% D Next 5% F “Grading on the curve”: instructors rank students into different percentage E. g. : GRE, SAT, GMAT, IQ
Significance l Statistical significance: l l Any observed difference (or statistical relationship) is unlikely to be due to chance Practical significance: l Real impact on people‘s life l l E. g. , the difference has a noticeable and nontrivial effect on functioning or performance When determining the merit of a particular outcome, we should take both significance into consideration
Relative merit l Using comparison to determine relative merit l Benchmark process, outcome, and cost criteria against what has been achieved elsewhere (e. g. by other evaluands of a similar setting).
Benchmarking l l It is a systematic study of one or more other organizations‘ systems, processes, and outcomes to identify ideas for improving organizational effectiveness. It refers to a process of gathering comparison data about what organizations in similar or related industries are achieving (e. g. About process, outcomes, and costs). l l Quantiative data Qualitative data (observation of processes)
Some hints l l Keep an open mind Any help that can help you to make good sense of the data Balance of time and effort Balance of time and level of details
Exercise l Grading Table 8. 8 according to the rubric in Table 8. 7 l l What is your grading Why is it? How can you improve table 8. 7? Form a group to discuss
Exercise l Take the grantsmanworkshop, draw the absolute and relative rubrics to grade this training program l l Absolute rubric (see Table 8. 4) Relative rubric (see Table 8. 7)