Outline n ContentBased Image Retrieval n n n

Outline n Content-Based Image Retrieval n n n 1 Query-by-Example Query-by-Feature Vector

CBIR and CBR n n n Content-based Image Retrieval (CBIR) como exemplo de Content-based Retrieval (CBR) concentra em low-level features. Principais idéias de CBIR: n n n 2 Representar uma imagem como um conjunto de feature descriptors. Definir medidas de similaridade dos descritores Quando um usuário especificar uma query, o sistema retorna imagens, que são ordenadas por similaridade.

CBIR Architecture 3

Image Retrieval Metadatabase Feature Extraction 4 Query Image Feature Extraction Image Database Images Select Compare Feature Vectors Query Result

CBIR de Butterflies n n Permitir non-expert users encontrar algumas espécies de butterflies usando informações de aparência de butterflies Aparência: n 5 Color, Texture, Shape

Problemas n n 6 Como podemos descrever uma butterfly? Como podemos comunicar nossa descrição para uma máquina?

Problemas n n 7 Usuários diferentes têm percepções diferentes. Usuários podem não se lembrar claramente a aparêcia de uma butterfly. Usuários normalmente não têm expertise para descrever butterflies. Usuários normalmente não têm paciência para fazer o browse num grande conjunto resultado.

Soluções n Usar um processo de consulta interativo e direcionado ao usuário: QBF/QBE query process n n 8 Query By Features e Query By Example Fuzzy feature description para cada butterfly Uma “What You See Is What You Get” query interface Um conjunto representativo de coleção butterflies

QBF/QBE query process (1) n QBF query: n n A QBF query is to choose some features of butterflies and expect that the system returns all butterflies with those features. Features of butterflies: n n QBE query: n 9 Dominant color, texture pattern, shape. A QBE query is to point an image and expect that the system returns all butterflies similar to that.

QBF/QBE query process (2) n Properties of QBF: n n Rough search When to use: n n Properties of QBE: n n Fine search When to use: n 10 The first query and when users want to enlarge the view in the search space Usually the last query and when users want to see the neighbors of the query one in the search space.

QBF/QBE query process (3) n Result page: (Each result page should contain two parts) n Result Images: These are the butterfly images satisfy the query conditions. n Users can invoke QBE queries from these images. n n Related Features: These are the features related to the previous query conditions. n Users can invoke QBF queries from these features. n 11

Feature Description (1) n Feature Description for a butterfly: n n Feature descriptor: n 12 Like metadata which describe the appearance of this butterfly. This makes QBF queries possible. Feature Description consists of some feature descriptors. A ( “feature value” , “match level” ) pair.

Feature Description (2) 13

Feature Description (3) Color 14

Feature Description (4) Texture 15

Feature Description (5) Shape 16

Feature Description (6) n QBF query: n Single feature query: Result images: images with its corresponding degree of match > 0. n Ranked by: degree of match. n We call this ranked sequence “Feature sequence. ” n n Multiple features query: n 17 Merge the corresponding feature sequences.

Result Presentation n For QBF query: n n n For QBE query: n n 18 Property: rough search Presentation: representative butterflies only Property: fine search Presentation: n For very similar images: present them all n For less similar images: representative ones

Feature Vector Indexing n Goal: n n Problems of Indexing in CBIR: n n n Dimension of feature space is very high. Index structure should support Euclidean and non-Euclidean similarity measures. Solution: n n 19 To make search efficiently. Dimension reduction: KLT, DCT, DWT. Similarity indexing: R*-tree, SS-tree, SRtree.

Semi-Automatic Feature Extraction n Segmentation: n n n Feature extraction: n n n 20 Background segmentation Butterfly object segmentation Color: color histogram Texture: manual annotation Shape: manual annotation

Classic CBIR with Color Feature n Most of the CBR systems rely on the notion of color, this may differ: n n n 21 Dominant color Scalable color based on color histograms (local for one region, global for the whole image) Color Structure Descriptor (incoporates the spatial structure)

What color is the apple ? We are so visual !!!! I’d say it is Bright Red I really couldn’t tell you (I am color blind) I think it is “Crimson” It is Red! 22

Color Histogram: Representation n A list of Color-Percentage pairs: n 23 Describe the colors and its percentages in an image.

Color Quantization n 24 Indexed Colors A jpg Image with 256 -color components in each RGB channel 256 x 256 colors in total → n groups, e. g, in 256 groups, that makes a reduction 256 x 256, I. e. , that each group takes 256 colors to count.

Similarity Measures - Overview n Minkowski Similarity n Distance L 1 : r = 1 n Distance L 2 : r = 2 n Quadratic Similarity n Intersection Similarity (Swain et Ballard 1991) 25

Example (cont. ) n Minkowski Similarity n Is a L-1 metric where Ik and Jk is the number of pixels in bin k for image I and J n Distance between above three images D(H 1, H 2) = 1 + 1 + 1 = 8 n D(H 1, H 3) = 6 + 2 + 2 = 24 n D(H 2, H 3) = 5 + 3 + 1 + 1 + 1 = 23 n 26

Example (cont. ) n Minkowski Similarity n n Is a L-2 metric Distance between above three images D(H 1, H 2) = (1 + 1 + 1)1/2 = 2. 8 n D(H 1, H 3) = (36 + 4 + 4) 1/2 = 9. 8 n D(H 2, H 3) = (25 + 9 + 1 + 1 + 1) 1/2 = 8. 5 n 27

QBIC distance n Weighted Euclidean distance (QBIC) n Is a L-2 metric(? ) distance between histogram H 1 and H 2: D = (H 1 - H 2)T A (H 1 - H 2) where A is a symmetric color similarity matrix A (i, j) = 1 – d (ci, cj) / dmax where ci and cj are the i-th and j-th color bins, d (ci , cj) is the color distance in the color space, and dmax is the maximum distance between any two colors in the color space 28

Limitation n 29 Ignore similarity between colors n Example n Two color bins n Bin-1 color range: 1 – 10 n Bin-2 color range: 11 – 20

• Three color pixels –Pixel 1 is Color 10 Bin-1 –Pixel 2 is Color 11 Bin-2 –Pixel 3 is Color 20 Bin-2 –Pixel 2 is similar to Pixel 3 than Pixel 1 unreasonable ! 30

Limitation (cont. ) n Ignore spatial relationships among pixels Different image with same histogram 31

Noise-Free Queries (NFQ’s) Rectangular query § NFQ is more precise. § User can specify semantic constraints: n n Spatial constraints (relative distances) Scaling constraints (relative sizes) Noise-free query Similar 32 Less relevant

Challenges n n Noise-free query 33 How do we extract features if we do not know the matching areas beforehand ? How do we index the images ?

One Solution – Local Color Histogram (LCH) n n 34 Each subimage has a color histogram. Any combination of the histograms can be selected for comparison with the corresponding color histograms of the query image.

Limitations of LCH n Dilemma: n n n Using small partitions is too expensive Limitation: n 35 Using large partitions is not precise difficult to handle scaling

Resultados esperados de uma boa CBIR com segmentação Query 2 Query 36 4 3 2 3 5 216 12 396 18

DEMOS n Hermitage Museum Web Site (QBIC) http: //hermitagemuseum. org/fcgibin/db 2 www/qbic. Color. mac/qbic? sel. Lang=En glish n 37 http: //www. aa-lab. cs. uu. nl/cbirsurvey/