ContentBased Image Retrieval Natalia Vassilieva Alexander Dolnik Ilya

Content-Based Image Retrieval Natalia Vassilieva Alexander Dolnik Il’ya Markov natalia@ntc-it. ru alexander. dolnik@gmail. com ilya. markov@gmail. com Saint-Petersburg State University February, 26 2007

Our team § § § § Natalia Vassilieva Alexander Dolnik Ilya Markov Maria Teplyh Maria Davydova Dmitry Shubakov Alexander Yaremchuk February, 26 2007

General problems § Semantic gap between system and human mode of image analysis § Specific of human visual perception § How to catch semantics of an image § Signature calculation and response time § Combining different features and metrics February, 26 2007

Image retrieval system How to minimize “semantic gap”? semantic gap low-level features General goal: an image retrieval system ü ü ü that that is able to process natural language query is able to search among annotated and non-annotated images takes into account human visual perception processes various features (color, texture, shapes) uses relevance feedback for query refinement, adaptive search February, 26 2007

indexation CBIR : Traditional approach auto-annotation image signature calculation database multidimensional indexing (vp-tree) color space partition according to human perception retrieval fusion of results: independent search by different features query signature calculation comparison result Relevance feedback: query refinement annotations refinement February, 26 2007

Research directions § Color space partition according to human visual perception § Correspondence between low-level features and semantics: auto-annotation § Fusion of retrieval result sets § Adaptive search: color and texture fusion § Using relevance feedback February, 26 2007

Human visual perception: colors Experiments with color partition: HSV space (H=9; S=2; V=3) – 72 % (H=11; S=2; V=3) – 66% (H=13; S=2; V=3) – 63% (H=15; S=2; V=3) – 60% Compare partitions of different spaces (RGB, HSV, Lab) February, 26 2007

Research directions § Color space partition according to human visual perception § Correspondence between low-level features and semantics: auto-annotation § Fusion of retrieval result sets § Adaptive search: color and texture fusion § Using relevance feedback February, 26 2007

Auto-annotation § Training set selection § Color feature extraction for every image from the set § Similarity calculation for every pair of images from the set § Training set clustering § Basis color features calculation: one per every cluster § Definition of basis lexical features § Correspondence between basis color features and basis lexical features Natalia Vassilieva, Boris Novikov. Establishing a correspondence between low-level features and semantics of fixed images. In Proceedings of the Seventh National Russian Research Conference RCDL'2005, Yaroslavl, October 04 - 06, 2005 February, 26 2007

Examples city, night, road, river snow, winter, sky, mountain February, 26 2007

Retrieve by textual query § Image database is divided into clusters § Search for appropriate cluster by textual query using cluster’s annotations § Browse the images from the appropriate cluster § Use relevance feedback to refine the query § Use relevance feedback to reorganize the clusters and assign new annotations N. Vassilieva and B. Novikov. A Similarity Retrieval Algorithm for Natural Images. Proc. of the Baltic DB&IS'2004, Riga, Latvia, Scientific Papers University of Latvia, June 2004 February, 26 2007

Feature extraction: color § Color: histograms § Color: statistical approach First moments for color distribution (every channel) and covariations February, 26 2007

Feature extraction: texture § Texture: use independent component filters that results from ICA Image I 1 dist(I 1, I 2) = N Σ KL (H i=1 H 1 i , H 2 i) H. Borgne, A. Guerin-Dugue, A. Antoniadis … Image I 2 “Representation of images for classification with independent features” N filtres February, 26 2007

Research directions § Color space partition according to human visual perception § Correspondence between low-level features and semantics: auto-annotation § Fusion of retrieval result sets § Adaptive search: color and texture fusion § Using relevance feedback February, 26 2007

Fusion of retrieval result sets Fusion of weighted lists with ranked elements: ω1 (x 11, r 11), (x 12, r 12), … , (x 1 n, r 1 n) ω2 (x 21, r 21), (x 22, r 22), … , (x 2 k, r 2 n) … ωm § § § (xm 1, rm 1), (xm 2, rm 2), … , (xml, rml) How to merge fairly? How to merge efficiently? How to merge effectively? ?

Ranked lists fusion: application area § Supplement fusion – union textual results (textual viewpoints ) § Collage fusion – combine texture (texture viewpoint) & color results (color viewpoint) – different color methods (different color viewpoints)

Ranked lists fusion: application area § Search by textual query in partly annotated image database Textual query Text. Result 1, textrank 1 TR 2, tr 2, . . . content-based by annotations tr 1 … tr 2 … … Result

Three main native fusion properties commutative property associative property value of result object's rank independent of another object's ranks Examples: COMBSUM, COMBMIN, COMBMAX merge functions

Additional native fusion properties normalization & delimitation property conic property § attraction of current object for mix result depend on value of function g(rank, weight) ≥ 0 ; § snare condition:

Conic properties, function g § g monotonically decreases with fixed weight parameter § g monotonically decreases with fixed rank parameter § g must satisfy boundaries conditions: § § g( 0, w ) > 0 if w != 0 g( r, 0 ) = 0

Ranked lists fusion: Formulas Fusion formula where

Ranked lists fusion: Algorithm § All lists are sorted by object id § Using step by step lists merging (object id priory) § If object_id 1 not equal object_id 2 => some object is absent in one of the lists Current object_id 1 List 1 Result list List 2 Current object_id 2

Ranked lists fusion: Experiments Necessary conditions: § Viewpoint should provide some “valuable” information. Retrieval system's performance at least should be better than a random system. § Information is not fully duplicated. There should be partial disagreement among viewpoints.

Ranked lists fusion: Experiments Parameters: § Roverlap && Noverlap conditions § Intercomparison of methods – Classical methods: COMBSUM, COMBMIN, COMBMAX – Probability methods: prob. Fuse – Random method: random values that satisfied to merge properties.

Research directions § Color space partition according to human visual perception § Correspondence between low-level features and semantics: auto-annotation § Fusion of retrieval result sets § Adaptive search: color and texture fusion § Using relevance feedback February, 26 2007

Adaptive merge: color and texture Dist(I, Q) = α*C(I, Q) + (1 - α)*Т(I, Q), C(I, Q) – color distance between I and Q; T(I, Q) – texture distance between I and Q; 0≤α≤ 1 Hypothesis: Optimal α depends on features of query Q. It is possible to distinguish common features for images that have the same “best” α. February, 26 2007

Adaptive merge: experiments February, 26 2007

Estimation tool § Web-application § Provides interfaces for developers of searchmethods § Uses common measures to estimate search methods: § Precision § Pseudo-recall § Collects users opinions – > builds test database February, 26 2007

Datasets § Own photo collection (~2000 images) § Subset from own photo collection (150 images) § Flickr collection (~15000, ~1. 5 mln images) § Corel photoset (1100 images) February, 26 2007

Research directions § Color space partition according to human visual perception § Correspondence between low-level features and semantics: auto-annotation § Fusion of retrieval result sets § Adaptive search: color and texture fusion § Using relevance feedback February, 26 2007