Using Artificial Queries to Evaluate Image Retrieval Nicholas

Using Artificial Queries to Evaluate Image Retrieval Nicholas R. Howe Department of Computer Science Cornell University June 12, 2000 Workshop on Content-Based Access of Image and Video Libraries 1

How Do We Compare Image Retrieval Algorithms? • Different research groups use images from different sources. • Image sets are of different sizes. • Tasks are different. – Each researcher identifies set of queries and targets through subjective criteria. – Can’t share keys because image sets are not standard. Answer: Badly! June 12, 2000 Workshop on Content-Based Access of Image and Video Libraries 2

How It’s Usually Done • Each researcher tests a proposed algorithm against a few baselines. – e. g. , Color Histograms. • No data to compare latest techniques. – Test sets are different. – Implementation of baselines may differ also. June 12, 2000 Workshop on Content-Based Access of Image and Video Libraries 3

Some Difficulties • Given a query, which target is most relevant? ? • Context will determine answer. June 12, 2000 Workshop on Content-Based Access of Image and Video Libraries 4

What Should a Good Test Do? • Provide comparable results even with different image sets. • Offer insight into the behavior of different retrieval algorithms. • Run quickly. • Allow for easy implementation. June 12, 2000 Workshop on Content-Based Access of Image and Video Libraries 5

Proposal: Altered-Image Queries Retrieved ranks: Image Library 1 Query Look for rank of original: 2 f 3 Image from Library June 12, 2000 Altered Image Workshop on Content-Based Access of Image and Video Libraries etc. 6

The Crop Test • Crop image to k% of its original area. Original Crop-50 • Simulates close-up shot of same subject. June 12, 2000 Workshop on Content-Based Access of Image and Video Libraries 7

The Jumble Test • Shuffle tiles in image divided on an h k grid. Original Jumble-4 4 • Simulates image with similar elements in a different arrangement. June 12, 2000 Workshop on Content-Based Access of Image and Video Libraries 8

The Low-Con Test • Decrease contrast to k% of its original range. Original Low-Con-80 • Simulates altered lighting conditions and/or camera differences. June 12, 2000 Workshop on Content-Based Access of Image and Video Libraries 9

Typical results • Most retrievals are at low rank. • A few retrievals are at much higher rank. Median: 26 Mean 205 • Median and mean summarize the results of multiple repetitions. June 12, 2000 Workshop on Content-Based Access of Image and Video Libraries 10

Difficulty of Altered-Image Queries • Both mean and median increase with difficulty. • Note order-of-magnitude changes. June 12, 2000 Workshop on Content-Based Access of Image and Video Libraries 11

How Stable are Altered-Image Queries? • Ran Crop-50 on three entirely different sets of 6000 images. Set 1 Set 2 Set 3 Mean Dev. Median Rank 5 5 7 5. 7 1. 2 Mean Rank 29. 6 33. 9 45. 5 36. 3 8. 2 • Some consistency even with different test sets. • Look for order-of-magnitude change. June 12, 2000 Workshop on Content-Based Access of Image and Video Libraries 12

Does the Number of Images Matter? • Found linear dependence on number of images. June 12, 2000 Workshop on Content-Based Access of Image and Video Libraries 13

How Many Queries Must Be Run? • Small % of total image set gives decent figure. June 12, 2000 Workshop on Content-Based Access of Image and Video Libraries 14

Comparing Algorithms Using Altered-Image Queries • Three algorithms compared using altered image queries. Histograms Correlograms STAIRS (Tuned) Crop 18 126. 6 Jumble 1 1 1 12. 4 17. 0 2. 0 1. 2 Low-Con 86. 5 350. 3 5 1 83. 6 22. 6 • Especially good or bad performance can be identified. June 12, 2000 Workshop on Content-Based Access of Image and Video Libraries 15

Final Thoughts • Altered-Image Queries are. . . – Well defined. – Easy to implement. – Consistent over different image sets. • A useful addition to our evaluation toolkit. • Also offer diagnostic potential. June 12, 2000 Workshop on Content-Based Access of Image and Video Libraries 16