RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND

RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES Mingming Lu, Qiyu Zhang, Wei-Hung Cheng, Cheng-Chang Lu Department of Computer Science Kent State University

Data Mining and Knowledge Management • • • Processing multimedia objects Defining and extracting features Feature dimension reduction Multimedia data retrieval Knowledge representation and management September, 2009 Kent State University 2

Current Tasks • Off-line data training – Segment images – batch mode – Find region of interest (ROI) – Interface with feature extraction and analysis – Feature domain processing September, 2009 Kent State University 3

Current Tasks (cont. ) • Users Interfaces – Reading user-input images – Segmentation – Find ROI – Feature extraction of ROI – Compare with trained data in repository – Return data (images) satisfying certain criteria September, 2009 Kent State University 4

Data Training Image Domain Segmentation Finding ROI Sending Images for Processing September, 2009 Feature Domain Interface Image & Feature Data Repository Kent State University Feature Extract Dimension Reduction Store Feature Data back 5

Image Domain Procesisng • Segmentation – Color VQ, Texture based image segmentation • Find ROI – ROI occupies large area – ROI locates near the image center – ROI contains homogenous texture September, 2009 Kent State University 6

Color-Texture Segmentation Applications • • Identify Regions of Interest (ROI) in a scene Image classification Image annotation Object based image and video coding September, 2009 Kent State University 7

Color-Texture Segmentation Current Limitations • Many existing techniques work well on homogeneous color regions, while natural scenes are rich in color and texture. • Many texture segmentation algorithms require the estimation of texture model parameters, which is a difficult problem and often requires a good homogeneous region for robust estimation. September, 2009 Kent State University 8

Color-Texture Segmentation Advantage of Color VQ and Texture based segmentation • Does not attempt to estimate a specific model for a texture region. • Tests for the homogeneity of a given colortexture pattern, which is computationally more feasible than estimation of model parameters. September, 2009 Kent State University 9

Color-Texture Segmentation Two-Step Process • Color Quantization – Performed in the color space without consideration of spatial distribution of colors. – Label each pixel with a quantized color to form a class-map. • Spatial Segmentation – Performed on the class-map September, 2009 Kent State University 10

Color-Texture Segmentation Color Quantization • Use Peer Group Filtering • As a result, coarse quantization can be obtained while preserving the color information in the original images. • Usually 10 -20 colors are needed in the images of natural scenes. September, 2009 Kent State University 11

Color-Texture Segmentation Criteria for Good Segmentation September, 2009 Kent State University 12

Color-Texture Segmentation -A Criterion for Good Segmentation • When the color classes are more separated from each other, J is getting larger. • If all color classes are uniformly distributed over the entire image, J tends to be small. September, 2009 Kent State University 13

Color-Texture Segmentation A Criterion for Good Segmentation • Now let us recalculate J over each segmented region instead of the entire class-map and define the average by • A segmentation which can minimize J is considered a good segmentation. September, 2009 Kent State University 14

Color-Texture Segmentation -Spatial Segmentation • Seed Determination • Seed Growing • Region Merge September, 2009 Kent State University 15

Color-Texture Segmentation -Spatial Segmentation September, 2009 Kent State University 16

ROI Determination • Find ROI – Mechanism – Pixel closer to the center contributes more weight to the region it belongs to. – Region with more pixels tends to get higher weight September, 2009 Kent State University 17

Results of Image Domain Processing • Results of Color Quantization • Results of Finding ROI September, 2009 Kent State University 18

Results of Image Domain Processing V 1 = 500, V 2 = 1, V 3 = 0. 5 March, 2004 Kent State University Auto 19

Results of Image Domain Processing V 1 = 500, V 2 = 1, V 3 = 0. 5 March, 2004 Kent State University Auto 20

March, 2004 Kent State University 21

Interface with Feature Domain • Find the rectangle circumscribing the ROI • Store its coordinate information into to a temporary file for feature domain’s use. September, 2009 Kent State University 22

Feature Domain(Overview) • Two Stages: – Feature Extraction – Dimension Reduction (DR) Feature Domain Image Domain Interface Feature Extract Image & Feature Data Repository September, 2009 Dimension Reduction Store Feature Data back Kent State University 23

Implementations • Acquire ROI information from the image domain • Extract features based on Gabor Filter and color histogram on HSV space • Integrate two feature spaces • Reduce the high feature dimensions to a very low number September, 2009 Kent State University 24

Implementations (cont. ) • Calculate the similarity measurement between the query object and the objects in the image repository • Search the similar images in the repository based on similarity index • Output the corresponding retrieval images • Knowledge extraction September, 2009 Kent State University 25

Feature Extraction Algorithm • Gabor Filter Feature – One of the most important wavelets with multiscale and multi-resolution – Mainly reflect texture information • Color histogram on HSV space – Provide color features September, 2009 Kent State University 26

Gabor Filter Concept • A complete but non-orthogonal basis wavelet set • A significant aspect: localized frequency description – composed of space information September, 2009 Kent State University 27

Gabor(cont. ) • A two dimensional Gabor function g(x, y) and its Fourier transform G(u, v) can be written as: September, 2009 Kent State University 28

Gabor(cont. ) • Let g(x, y) be the mother Gabor wavelet, then this self-similar filter dictionary can be obtained by appropriate dilations and rotations of g(x, y) through the generating function September, 2009 Kent State University 29

Color Histogram in HSV Space • HSV color space includes – Hue (H) – Saturation (S) – Value (V or Lightness) • Only consider Hue and saturation information, since the lightness of pictures is very sensitive to the surrounding conditions. September, 2009 Kent State University 30

HSV space Figure September, 2009 Kent State University 31

HSV space bands • Design bands in the HSV space – 8 hue bands – 4 saturation bands, – Total 32 sub-spaces • Compute color histogram feature in each sub-space to form 32 feature dimensions eventually September, 2009 Kent State University 32

Feature Integration • Normalize both Gabor filter and HSV color histogram features • Set a weight factor to balance two feature spaces. Usually Gabor filter features will have the bigger weight value. September, 2009 Kent State University 33

DR Algorithm • Disadvantages in the high dimension space – The computational complexity arise sharply – The database indexing becomes difficult • Principal Component Analysis (PCA) – PCA seeks to reduce the dimension of the data by finding a few orthogonal linear combinations (Principal Component “PC”) September, 2009 Kent State University 34

DR implementation • Original feature dimensions – Gabor filter features: 6*5*2 = 60 – HSV color histogram features: 4*8 = 32 – Total dimensions: 92 • Feature dimensions after DR – 10 ~15 dimensions September, 2009 Kent State University 35

Simulation Results in the Feature Domain • We randomly select 11 query pictures as the test samples in this report. • At each query time, at most 14 retrieval pictures are retrieved. • The minimum square error method is served as the similarity measurement. • The value in the tables as below means the positive pictures out of the 14 retrieval pictures. September, 2009 Kent State University 36

Performance between different feature extraction techniques • the integration of Gabor Filter and HSV color Histogram gains the better performance. • See pictures in detail. Click here Query pic# 1 2 3 4 5 6 7 8 9 10 11 Gabor 6 7 7 4 12 1 1 2 4 3 2 HSV 8 2 9 1 2 3 1 1 4 2 3 Integrated 10 5 11 4 12 3 3 2 5 2 4 September, 2009 Kent State University 37

Performance between with and without DR applied • The performance after DR applied slightly degrades on average in comparison to the results before DR takes on stage • See pictures in detail. Click here Query pic# 1 2 3 4 5 6 7 8 9 10 11 Integrated 10 5 11 4 12 3 3 2 5 2 4 DR 9 6 5 5 12 2 1 1 4 3 2 September, 2009 Kent State University 38

More Simulations • Performance between different weight used • Performance between different dimensions retained after DR September, 2009 Kent State University 39

Final Integration Results • Simulation results when both the image domain and the feature domain are used • See the detail pictures, Click here September, 2009 Kent State University 40

Integration • UAV media capture and analysis • WWW based media analysis • Vehicle based media capture and analysis September, 2009 Kent State University 41

Future Research Extended to video objects • • Object based video coding Non-object based video coding Video indexing Knowledge extraction and management September, 2009 Kent State University 42

Future Research Data Fusion • • • Multimodality medical imaging CT – Structural information PET – Functional information Fusion Knowledge management September, 2009 Kent State University 43