Tried and Tested Evaluating Data Entry Consistency in

Tried and Tested? Evaluating Data Entry Consistency in the Component Model Prototype Brit Hopmann, July 1, 2008

Tried and Tested? Presentation • Background & aim consistency check • Images & data • Analysis & results (in 3 steps) • Problems & possible solutions • Conclusion & discussion data entry Koninklijke Bibliotheek – National Library of the Netherlands

Tried and Tested? Background & aim consistency check Background • Problems with data entry ambiguity • Consequences: decreased recall and precision Aim • Check and improve data entry consistency • Solutions for existing problems? • Changes? data entry Koninklijke Bibliotheek – National Library of the Netherlands

Tried and Tested? Images Koninklijke Bibliotheek – National Library of the Netherlands

Tried and Tested? Analysis Data • 5 images • 25 features (+ 2 discarded) • 6 respondents Analysis in 3 steps • Categorizing the answers (per image / per feature) • Analysis per image and per feature • Problems and possible solutions Koninklijke Bibliotheek – National Library of the Netherlands

Tried and Tested? Analysis – step 1 example: answers for image 4 Koninklijke Bibliotheek – National Library of the Netherlands

Tried and Tested? Analysis – step 1 Categorizing the answers (per image / per feature) 3 categories category consistency times percentage no problem 57 45, 6% indecisive 16 12, 8% problem 52 41, 6% There were many problematic descriptions. Koninklijke Bibliotheek – National Library of the Netherlands

Tried and Tested? Analysis – step 2 Summary per watermark (5 images) None of the images was described unanimously. image no prob ≈ problem per image 1 15 2 8 fine image 2 13 3 9 fine image 3 10 3 12 problematic image 4 11 3 11 indecisive (≈) image 5 8 5 12 problematic 57 16 total Koninklijke Bibliotheek – National Library of the Netherlands 52

Tried and Tested? Analysis – step 2 Summary per feature (25 features + 2 discarded) Described consistently 5 -7 features* 3 of 5 images not consistently described 10 -8 features* * Depending on quality of definitions Koninklijke Bibliotheek – National Library of the Netherlands

Tried and Tested? Analysis – step 3 How problematic are specific features? 1. Is or might the feature be problematic with other watermarks? 2. What caused the problem and can it be solved easily? -> image quality, definition, subjective description 3. What is the distinctive value of a particular feature? 4. Is there a possible solution? 5. Which problems will remain problematic after having applied a specific solution? Koninklijke Bibliotheek – National Library of the Netherlands

Tried and Tested? Analysis – step 3 What could contribute to a solution? • • Grouping • Allow assigning multiple features • Drawback: increased decision effort Extra tool • For instance graphical indicator of angle • Drawback: exceptions such as rotated scans Clarify definitions • For instance description of additional elements • Relatively straightforward to apply Otherwise Koninklijke Bibliotheek – National Library of the Netherlands

Tried and Tested? Crucial Question Which features will still pose a problem with a particular solution applied? Koninklijke Bibliotheek – National Library of the Netherlands

Tried and Tested? Conclusion Problems that will eventually remain • Eyes: alignment, distance, position • Horns: endings, orientation, inside Koninklijke Bibliotheek – National Library of the Netherlands

Tried and Tested? Conclusion Remaining problems • Ears: endings, roundness, width • Eyes: shape • Head: shape • In-between horns: shape • Nose: concomitance with other elements, shape Koninklijke Bibliotheek – National Library of the Netherlands

Tried and Tested? Discussion Features that evoke subjective descriptions (shape!) cause problems Thinking about possible changes in the model • Clarify certain issues: clear definitions, instruction • Expand model: extra tool, grouping • Simplify model: adjust or take out features Reconsider role of component model within Bernstein data Koninklijke Bibliotheek – National Library of the Netherlands

Tried and Tested? Discussion Questions and Comments? Koninklijke Bibliotheek – National Library of the Netherlands

Tried and Tested? Koninklijke Bibliotheek – National Library of the Netherlands