An Unsupervised Learning Approach to ContentBased Image Retrieval
An Unsupervised Learning Approach to Content-Based Image Retrieval Yixin Chen & James Z. Wang The Pennsylvania State University IEEE Int'l Symposium on Signal Processing and its Applications 1
Outline l Introduction l Cluster-based retrieval of images l Experiments l Conclusions and future work IEEE Int'l Symposium on Signal Processing and its Applications 2
Image Retrieval l The driving forces Internet l Storage devices l Computing power l l Two approaches Text-based approach l Content-based approach l IEEE Int'l Symposium on Signal Processing and its Applications 3
Text-Based Approach l Input Elephants keywords descriptions Text-Based Image Retrieval System Image Database IEEE Int'l Symposium on Signal Processing and its Applications 4
Text-Based Approach l Index images using keywords (Google, Lycos, etc. ) Easy to implement l Fast retrieval l Web image search (surrounding text) l Manual annotation is not always available l A picture is worth a thousand words l Surrounding text may not describe the image l IEEE Int'l Symposium on Signal Processing and its Applications 5
Content-Based Approach l Index images using low-level features CBIR System Image Database Content-based image retrieval (CBIR): search pictures as pictures IEEE Int'l Symposium on Signal Processing and its Applications 6
A Data-Flow Diagram Histogram, color layout, sub-images, regions, etc. Feature Extraction Image Database Euclidean distance, intersection, shape comparison, region matching, etc. Compute Similarity Measure Linear ordering, Projection to 2 -D, etc. Visualization IEEE Int'l Symposium on Signal Processing and its Applications 7
Open Problem Nature of digital images: arrays of numbers l Descriptions of images: high-level concepts l l l Sunset, mountain, dogs, …… Semantic gap l l Discrepancy between low-level features and highlevel concepts High feature similarity may not always correspond to semantic similarity IEEE Int'l Symposium on Signal Processing and its Applications 8
Outline l Introduction l Cluster-based retrieval of images l Experiments l Conclusions and future work IEEE Int'l Symposium on Signal Processing and its Applications 9
Motivation l. A query image and its top 29 matches returned by a CBIR system Horses (11 out of 29), flowers (7 out of 29), golf player (4 out of 29) IEEE Int'l Symposium on Signal Processing and its Applications 10
CLUE: CLUsters-based r. Etrieval of images by unsupervised learning l Hypothesis In the “vicinity” of a query image, images tend to be semantically clustered l CLUE attempts to capture high-level semantic concepts by learning the way that images of the same semantics are similar IEEE Int'l Symposium on Signal Processing and its Applications 11
System Overview A general diagram of a CBIR system using CLUE Image Feature Extraction Database Select Neighboring Images Image Clustering Display And Feedback Compute Similarity Measure IEEE Int'l Symposium on Signal Processing and its Applications 12
Neighboring Images Selection l Nearest neighbors method l l l Pick k nearest neighbors of the query as seeds Find r nearest neighbors for each seed Take all distinct images as neighboring images IEEE Int'l Symposium on Signal Processing and its Applications k=3, r=4 13
Weighted Graph Representation l Graph representation l l l Vertices denote images Edges are formed between vertices Nonnegative weight of an edge indicates the similarity between two vertices IEEE Int'l Symposium on Signal Processing and its Applications 14
Clustering l Graph partitioning and cut l Normalized cut (Ncut) [Shi et al. , IEEE Trans. PAMI 22(8)] l Recursive Ncut IEEE Int'l Symposium on Signal Processing and its Applications 15
Outline l Introduction l Cluster-based retrieval of images l Experiments l Conclusions and future work IEEE Int'l Symposium on Signal Processing and its Applications 16
An Experimental System l Similarity l measure UFM [Chen et al. IEEE PAMI 24(9)] l Database COREL l 60, 000 l IEEE Int'l Symposium on Signal Processing and its Applications 17
User Interface IEEE Int'l Symposium on Signal Processing and its Applications 18
Query Examples l Query Examples from 60, 000 -image COREL Database Bird, car, food, historical buildings, and soccer game UFM CLUE Bird, 6 out of 11 Bird, 3 out of 11 IEEE Int'l Symposium on Signal Processing and its Applications 19
Query Examples CLUE UFM Car, 8 out of 11 Car, 4 out of 11 Food, 8 out of 11 Food, 4 out of 11 IEEE Int'l Symposium on Signal Processing and its Applications 20
Query Examples CLUE UFM Historical buildings, 10 out of 11 Historical buildings, 8 out of 11 Soccer game, 10 out of 11 Soccer game, 4 out of 11 IEEE Int'l Symposium on Signal Processing and its Applications 21
Clustering WWW Images l Google Image Search Keywords: tiger, Beijing l Top 200 returns l 4 largest clusters l Top 18 images within each cluster l IEEE Int'l Symposium on Signal Processing and its Applications 22
Clustering WWW Images Tiger Cluster 1 (75 images) Tiger Cluster 2 (64 images) Tiger Cluster 3 (32 images) Tiger Cluster 4 (24 images) IEEE Int'l Symposium on Signal Processing and its Applications 23
Clustering WWW Images Beijing Cluster 1 (61 images) Beijing Cluster 2 (59 images) Beijing Cluster 3 (43 images) Beijing Cluster 4 (31 images) IEEE Int'l Symposium on Signal Processing and its Applications 24
Retrieval Accuracy IEEE Int'l Symposium on Signal Processing and its Applications 25
Outline l Introduction l Cluster-based retrieval of images l Experiments l Conclusions and future work IEEE Int'l Symposium on Signal Processing and its Applications 26
Conclusions l Retrieving image clusters by unsupervised learning l Tested using 60, 000 images from COREL and images from WWW IEEE Int'l Symposium on Signal Processing and its Applications 27
Future Work l Recursive Ncut l Representative image l Other graph theoretic clustering techniques l Nonlinear dimensionality reduction IEEE Int'l Symposium on Signal Processing and its Applications 28
Thank You! IEEE Int'l Symposium on Signal Processing and its Applications 29
- Slides: 29