ContentBased Video Retrieval System Presented by Edmund Liang

Content-Based Video Retrieval System Presented by: Edmund Liang CSE 8337: Information Retrieval

Introduction Ø Traditional Library search method

Introduction (cont. ) Ø Other search engines still using description search method. Ø Current image search method: by description.

Introduction (cont. ) Ø Sample of Google Video Search:

Introduction (cont. ) Ø Google Video Archive selections:

Introduction (cont. ) Ø Picture is worth a thousand words. Ø More than words can express. Ø Growing number video clips on My. Space and You. Tube, there is a need for a video search engine.

Introduction (cont. ) Ø Sample You. Tube Video page:

Introduction (cont. ) Ø Therefore, we need a better search technique – Content-Based Video Retrieval System (CBVR).

Introduction (cont. ) Ø What good is video retrieval? l l Historical Achieve Forensic documents Fingerprint & DNA matching Security usage

Overview (cont. ) Ø CBVR has two Approaches: l l Ø Attribute based Object based CBVR can be done by: l l l l l Color Texture Shape Spatial relationship Semantic primitives Browsing Objective Attribute Subjective Attribute Motion Text & domain concepts

Overview (cont. ) Ø CBVR has two phases: l Database Population phase • • • l Video shot boundary detection Key Frames selection Feature extraction Video Retrieval phase • Similarity measure

Overview (cont. ) Ø How CBVR works: [Wang, Li, Wiederhold, 2001]

Database Population Phase Ø Here are three major procedures: l Shot boundary detection – partition, segments [Luo, Hwang, Wu, 2004]

Database Population Phase (cont) l l Key frames selection – select characteristics Extracting low-level spatial features like color, texture, shape, etc. [Luo, Hwang, Wu, 2004]

Database Population Phase (cont. ) Video is complex data type – audio & video Ø Audio can be handled by query by humming. Ø Voice recognition system using Patricia-like tree to construct all possible substrings of a sentence. Ø Audio is categorized by: speech, music, and sound. Ø Audio retrieval methods: Hidden Markov Model, Boolean Search with multi-query using Fuzzy Logic. Ø

Database Population Phase (cont) Ø Most simple database storage: description of video as index along with the video. Ø Human effort is involved in this case. Ø We are searching for automatic video indexing and digital image storage method – Latent Semantic Indexing (LSI)

Database Population Phase (cont. ) LSI is using vector space model – low rank approximation of vector space represent image document collection. Ø Original matrix is replaced by an as close as possible matrix, where its column space is only the subspace of the original matrix column space. Ø By reducing the rank of the matrix, noises (duplicate frames) are reduce to improve storage and retrieval performance. Ø Term indexing is referred to the process of assigning terms to the content of the video. Ø

Database Population Phase (cont. ) Ø Closest terms in the database is returned based on the similarity measure between the query images and the resulting ones. Ø Cosine similarity measure is used in the vector space model. Ø Cosine similarity measure on Term-byvideo matrix:

Database Population Phase (cont. ) Ø Enterprise database like Oracle introduces new object type: ORDImage, which contains four different visual attributes: global color, local color, texture and shape. Ø ORDImage. Index provides multidimensional index structure to speed up stored feature vectors.

Database Population Phase (cont. ) Ø Oracle example of joining two images of Picture 1 and Picture 2: CREATE TABLE Picture 1( author VARCHAR 2(30), description VARCHAR 2(200), photo 1 ORDSYS. ORDImage, photo 1_sig ORDSYS. ORDImage. Signature ); CREATE TABLE Picture 2( mydescription VARCHAR 2(200), photo 2 ORDSYS. ORDImage, photo 2_sig ORDSYS. ORDImage. Signature ); SELECT p 1. description, p 2. mydescription FROM Picture p 1, Picture p 2, WHERE ORDSYS. IMGSimilar(p 1. photo 1_sig, p 2. photo 2_sig, ‘color=” 0, 6” texture=” 0, 2” shape=” 0, 1” location=” 0, 1”’, 20)=1; Note: Weighted sum of the distance of the visual attributes is less than or equal to the threshold, the image is matched.

Image Retrieval Phase Ø Query by example (QBE) l Allow to select sample image to search. [Wang, Li, Wiederhold, 2001]

Image Retrieval Phase (cont. ) Yet Another CBVR Application Interface [Li, Shapiro 2004]

Image Retrieval Phase (cont. ) Ø Query by color anglogram l l Histogram intersection measures is a fairly standard metric to analyze histogram base on features. Image is divided into 5 sub-images, upper right, upper left, lower right, lower left, and the center image.

Image Retrieval Phase (cont. ) Ø Query by color anglogram (cont. ) l l Convert RGB to HSV [wikipedia] Global and sub-image histogram forms LSI matrix. [Zhao & Grosky 2002]

Image Retrieval Phase (cont) Ø Sample results: Ancient Towers Ancient Columns Horses Figure [Zhao & Grosky 2002]

Image Retrieval Phase (cont. ) Ø Retrieve by shape anglogram l l Each image is divided into 256 block. Each block is approximated with hue and saturated value. Corresponding feature points are mapped perceptually base on the saturated value. Feature histogram is obtained by measure the largest angle of the nearest feature points.

Image Retrieval Phase (cont. ) Ø Query by shape anglogram (cont): Demo [Zhao & Grosky 2002]

Image Retrieval Phase (cont. ) Ø Query by shape anglogram sample output: [Zhao & Grosky 2002]

Image Retrieval Phase (cont. ) Ø Query by color and other category selection combination. l l Use training dataset: sky, sun, land, water, boat, grass, horse, rhino, bird, human, pyramid, column, tower, sphinx, and snow. Sun(5%), grass (15%), Sky(20%) combine with the LSI matrix to return better results.

Future Works Ø Handle multi-layer images Ø Include human-intractable relevance retrieval feedback system. Ø Eliminate bias objects but not affecting the performance.

Summary Ø Content-Based Video Retrieval system contains two phases: l Database population phase • • • l Shot boundary detection Key frames selection Extract low-level features Image retrieval phase • • Query by example Query by color anglogram Query by shape anglogram Query by color anglogram and category bit.

Conclusion Ø Content-based Video Retrieval system is not a sound system. Ø Video stream will become the main stream in the years to come. Ø Better off if we had a efficient CBVR search engine ready. Ø Still many area needs to be improved.

The End Thank you.