Dissertation Progress Report By R Bassett April 26

Dissertation Progress Report By: R. Bassett April 26, 2003

Experimental Overview Ü Machine Based Reliability Testing 1 Experiment with 5 tests Ü Online Human Grading 6 Experiments

Machine Based Reliability Test Ü The goal of this experiment was to test the reliability of the machine-based automated coin grader. Ü The coin images from this system were to become the input for the Online Human Grading Experiments. Ü Twenty coins were sent out to third-party grading services to establish ‘Expert Grades’. Ü The Expert Grades were tested in the machine-based system to how reliable the system was. (The target was 95% reliability)

Machine Based Reliability Test - Results -

Online Human Grading Ü 50 Expert graders participated in the online grading experiments during the first 10 days of April via a 22 screen ASP application. Ü The graders graded 20 coin images and were part of six simultaneous experiments.

Online Human Grading Ü Experiment #01: Experts Acceptance Experiment Ü Experiment #02: Advance Grade Knowledge Experiment Ü Experiment #03: The Importance of Subjective Qualities Ü Experiment #04: Grader Consensus Ü Experiment #05: Duplicate Grades Ü Experiment #06: Perception of Internet based Grading

Opening Screen

Coin 1 Screen

Closing Screen

Experiment #01: Experts Acceptance Experiment Research Questions: 1. Do graders accept the expert grade provided by the machine as being accurate? 2. Or do the graders feel that it is correct, too liberal or too conservative? Summary Results: In 14 out of 20 grading tests (70% of the time) expert graders assigned grades higher than those of the machine-based system while in 6 out of 20 tests (30% of the time) expert graders assigned lower average grades than the machine-based system.

Exp 01

Experiment #02: Advance Grade Knowledge Experiment Research Questions: 1. Does knowing a grading opinion in advance influence the grade given by graders? 2. Does providing the correct grade yield a more consistent result than providing misleading grades? Summary Results: Expert grading consultants: • Came closet to the machine-based grade (55% of the time) when they were provided with a reliable machine-based grade in advance. • Came next closest to the machine-based grade when (35% of the time) when they were provided with a misleading incorrect grade in advance. • Were the furthest away from the machine-based grade with only 10% accuracy when they were provided with no grading opinion in advance • Were 90% effective when they were provided with some grading opinion even if the grading opinion was incorrect.

Exp 02

Experiment #03: The Importance of Subjective Qualities Research Questions: 1. If graders feel that subjective qualities must be incorporated into grading which subjective qualities are most significant? Summary Results:

Experiment #04: Grader Consensus Research Questions: 1. To what extent do human graders come to consensus when grading the same coins as other human graders? Summary Results: • The experts do disagree in their grading opinions. • The opinions of experts are diverse as they take different factors into consideration when they grade.

Exp 04

Experiment #05: Duplicate Grades Research Questions: 1. How did the grading experts do with the duplicates? Summary Results: • The results were interesting in that there was a more significant variation in the grades assigned on the first series of coins #19 and #05 than those of the second series of coins #20 and #07. The average difference in the first series was 4. 663 grade points or 6%. While the average difference in the second series was a scant. 941 grade points or 1. 3%.

Exp 05

Experiment #06: Perception of Internet based Grading Research Questions: 1. Was the Internet an effective vehicle for grading? Summary Results: The majority of the graders indicated that is more difficult grading digitized coin images over the Internet to the physical coins but almost an equal majority indicated that the images were good enough to properly grade the coins. These results can be interpreted as the experts preferring to grade the physical coins but acknowledging that the Internet and the digital images are a suitable substitute in the absence of the actual coins.

The first question asked the experts how grading images on the Internet is compared to grading physical coins 78. 37% said that it is more difficult than grading coin images over the Internet than the physical coins 20. 03% said that is the same or better than grading the physical coins 1. 6% said that the practice of grading over the Internet should be banned The second question asked the experts about the quality of the images presented 64. 87% said that the images was good enough to properly grade the coins 35. 13% said that the images were of poor quality and made it hard to grade properly