S 519 Evaluation of Information Systems Analyzing data

  • Slides: 22
Download presentation
S 519: Evaluation of Information Systems Analyzing data: Merit Ch 8

S 519: Evaluation of Information Systems Analyzing data: Merit Ch 8

Merit determination l Keep in mind of what the evaluation is. l l Evaluation

Merit determination l Keep in mind of what the evaluation is. l l Evaluation is the systematic determination of the quality or value of something (Scriven, 1991). Today we will discuss how to determine the importance of evaluand components or dimensions

Merit determination l It is the process of setting „standards“ (definitions of what performance

Merit determination l It is the process of setting „standards“ (definitions of what performance should constitute „satisfactory“, „good“, etc. ) and applying those standards to descriptive data to draw explicitly evaluative conclusions about performance on a particular dimension or component.

Two steps l l Step 1: defining what constitutes poor, adequate, good, very good

Two steps l l Step 1: defining what constitutes poor, adequate, good, very good and excellent performances on a particular dimension (or component) Step 2. using this definition to convert empirical evidence into evaluative conclusions (e. g. something explicit about quality or value)

Determing merit Decscriptive facts about performance Quality or value determination guide Evaluative conclusions

Determing merit Decscriptive facts about performance Quality or value determination guide Evaluative conclusions

Usingle quantitaive measure l In a simple case, performance is measured on a single

Usingle quantitaive measure l In a simple case, performance is measured on a single quantitative dimension l The quality or value determination guide is just a set of cutoffs l l l E. g. , >90%=A/excellent, 80%-89%=B/good, 70%79%=C/adequate E. g. , satisfactory/unsatisfactory Difficulty: where to put the cutoff score and how to compare with different systems?

Exercise l School grading system l l USA: A(>90%), B(80 -90%), C(70 -79%), D(60

Exercise l School grading system l l USA: A(>90%), B(80 -90%), C(70 -79%), D(60 -69%), F(<60%) New Zealand: A(>80%), B(65 -79%), C(50 -64%), D(3549%), F(<35%). Does that mean in New Zealand, it is easier to get A? Can this way of grading ensure objectivity and consistency of grading across courses? If yes, why? If no, why not?

Using qualitative or multiple measures l l l Using a single measure is not

Using qualitative or multiple measures l l l Using a single measure is not generally good practice. When using multiple measures, it is tricky on how to merge them together to come out with conclusion. See table 8. 1

Experience l l Do not try to go for high precision It is perfectly

Experience l l Do not try to go for high precision It is perfectly appropriate to give an answer that still has a certain amount of fuzziness or uncertainty associated with it. Please do not oversell the precision of your work Providing a well-supported broad-brush answer to an important question is not a bad idea.

Rubric l l Rubric is a tool that provides an evaluative description of what

Rubric l l Rubric is a tool that provides an evaluative description of what performance or quality „looks like“. It has two levels: l l Grading rubric is used to determine absolute quality or value (e. g. , Table 8. 2) Ranking rubric is used to determine relative quality or value

Rubric for absolute value l Rubric for “grading” is based on: l l l

Rubric for absolute value l Rubric for “grading” is based on: l l l Discussion with domain expert Discussion with upstream stakeholders Existing rules (scope of duties) or literatures Evaluand expectations (needs assessment) Evaluation context (job market, current situation)

Sample grading rubric 1 l l l Table 8. 3 Using Table 8. 3

Sample grading rubric 1 l l l Table 8. 3 Using Table 8. 3 to grade Table 8. 1 – what is grade for this master program? Why? Form a pair and discuss your point

Sample grading rubric 2 l Table 8. 4 provides you a better grading rubric

Sample grading rubric 2 l Table 8. 4 provides you a better grading rubric l l l Scope Performance indicators Scores (1 -5)

Exercise l Develop a grading rubric for your evaluation project l Take Table 8.

Exercise l Develop a grading rubric for your evaluation project l Take Table 8. 4 as example l Please include scope, performance indicators and ranking description

Using rubric for determining „relative“ merit l Relative metric is important for experiment that

Using rubric for determining „relative“ merit l Relative metric is important for experiment that uses a control or comparison group. l l Student scores are interpreted by comparison with other similar schools It simply tells us how the person or program did relative to peers or competitors.

Using rubric for determining „relevative“ merit Score falls in Grade assigned Top 10% A

Using rubric for determining „relevative“ merit Score falls in Grade assigned Top 10% A Next 20% B Next 50% C Next 15% D Next 5% F “Grading on the curve”: instructors rank students into different percentage E. g. : GRE, SAT, GMAT, IQ

Significance l Statistical significance: l l Any observed difference (or statistical relationship) is unlikely

Significance l Statistical significance: l l Any observed difference (or statistical relationship) is unlikely to be due to chance Practical significance: l Real impact on people‘s life l l E. g. , the difference has a noticeable and nontrivial effect on functioning or performance When determining the merit of a particular outcome, we should take both significance into consideration

Relative merit l Using comparison to determine relative merit l Benchmark process, outcome, and

Relative merit l Using comparison to determine relative merit l Benchmark process, outcome, and cost criteria against what has been achieved elsewhere (e. g. by other evaluands of a similar setting).

Benchmarking l l It is a systematic study of one or more other organizations‘

Benchmarking l l It is a systematic study of one or more other organizations‘ systems, processes, and outcomes to identify ideas for improving organizational effectiveness. It refers to a process of gathering comparison data about what organizations in similar or related industries are achieving (e. g. About process, outcomes, and costs). l l Quantiative data Qualitative data (observation of processes)

Some hints l l Keep an open mind Any help that can help you

Some hints l l Keep an open mind Any help that can help you to make good sense of the data Balance of time and effort Balance of time and level of details

Exercise l Grading Table 8. 8 according to the rubric in Table 8. 7

Exercise l Grading Table 8. 8 according to the rubric in Table 8. 7 l l What is your grading Why is it? How can you improve table 8. 7? Form a group to discuss

Exercise l Take the grantsmanworkshop, draw the absolute and relative rubrics to grade this

Exercise l Take the grantsmanworkshop, draw the absolute and relative rubrics to grade this training program l l Absolute rubric (see Table 8. 4) Relative rubric (see Table 8. 7)