CSE 473573 Computer Vision and Image Processing CVIP

CSE 473/573 Computer Vision and Image Processing (CVIP) Ifeoma Nwogu Lecture 26 – Recognition 1

Schedule • Last class – We finished object detection • Today – Object recognition • Readings for today: – Forsyth and Ponce chapter 18 2

Slide Credits • All Darrell Trevor – UC Berkeley 3

Object recognition 1000+ descriptors per frame Shape adapted regions Slide credit: J. Sivic Maximally stable regions

Match regions between frames using SIFT descriptors and spatial consistency Multiple regions overcome problem of partial occlusion Shape adapted regions Maximally stable regions Slide credit: J. Sivic

Visual search using local regions Schmid and Mohr ’ 97 Sivic and Zisserman’ 03 Nister and Stewenius’ 06 Philbin et al. ’ 07 Chum et al. ’ 07 + Jegou and Schmid’ 07 Chum et al. ’ 08 – 1 k images – 50 k images (1 M) – 100 k images – 1 M images – 5 M images Index 1 billion (10^9) images – 200 servers each indexing 5 M images? Slide credit: J. Sivic

Beyond Nearest Neighbors… Indexing local features using inverted file index For text documents, an efficient way to find all pages on which a word occurs is to use an index… We want to find all images in which a feature occurs. To use this idea, we’ll need to map our features to “visual words”. K. Grauman, B. Leibe Slide credit: J. Sivic 7

Object Bag of ‘words’ Slide credit L. Fei-Fei

Analogy to documents Of all the sensory impressions proceeding to the brain, the visual experiences are the dominant ones. Our perception of the world around us is based essentially on the messages that reach the brain from our eyes. For a long time it was sensory, brain, thought that the retinal image was transmitted perception, point by pointvisual, to visual centers in the brain; the cerebral cortex was a movie screen, cortex, so to speak, retinal, cerebral upon which the image in the eye was projected. eye, cell, optical Through the discoveries of Hubel and Wiesel we now know that behind the origin of the visual nerve, image perception in the brain there is a considerably more complicated. Hubel, course of Wiesel events. By following the visual impulses along their path to the various cell layers of the optical cortex, Hubel and Wiesel have been able to demonstrate that the message about the image falling on the retina undergoes a step-wise analysis in a system of nerve cells stored in columns. In this system each cell has its specific function and is responsible for a specific detail in the pattern of the retinal image. China is forecasting a trade surplus of $90 bn (£ 51 bn) to $100 bn this year, a threefold increase on 2004's $32 bn. The Commerce Ministry said the surplus would be created by a predicted 30% jump in exports to $750 bn, compared with a 18% China, The trade, rise in imports to $660 bn. figures are likely to further annoysurplus, the US, which has long argued that commerce, China's exports are unfairly helped by a US, deliberatelyexports, undervaluedimports, yuan. Beijing agrees the surplus is too high, but says the yuan is only one yuan, bank, domestic, factor. Bank of China governor Zhou Xiaochuan foreign, increase, said the country also needed to do more to boost domestic demand so more goods stayed within trade, value the country. China increased the value of the yuan against the dollar by 2. 1% in July and permitted it to trade within a narrow band, but the US wants the yuan to be allowed to trade freely. However, Beijing has made it clear that it will take its time and tread carefully before allowing the yuan to rise further in value. Slide credit L. Fei-Fei

A clarification: definition of “Bo. W” Looser definition – Independent features Slide credit L. Fei-Fei

Visual words: main idea Extract some local features from a number of images … e. g. , SIFT descriptor space: each point is 128 -dimensional Slide credit: D. Nister K. Grauman, B. Leibe 11

Visual words: main idea Slide credit: D. Nister K. Grauman, B. Leibe 12

Visual words: main idea Slide credit: D. Nister K. Grauman, B. Leibe 13

Visual words: main idea Slide credit: D. Nister K. Grauman, B. Leibe 14

Slide credit: D. Nister K. Grauman, B. Leibe 15

Slide credit: D. Nister K. Grauman, B. Leibe 16

Visual words Example: each group of patches belongs to the same visual word Figure from Sivic & Zisserman, ICCV 2003 K. Grauman, Leibe Slide credit: J. B. Sivic 19

Visual words • First explored for texture and material representations • Texton = cluster center of filter responses over collection of images • Describe textures and materials based on distribution of prototypical texture elements. Leung & Malik 1999; Varma & Zisserman, 2002; Lazebnik, Schmid & Ponce, 2003; Slide credit: J. Sivic

Inverted file index for images comprised of visual words Word List of image numbers number • Score each image by the number of common visual words (tentative correspondences) • But: does not take into account spatial layout of regions Image credit: A. Zisserman K. Grauman, Leibe Slide credit: J. B. Sivic

Clustering / quantization methods • k-means (typical choice), agglomerative clustering, mean-shift, … • Hierarchical clustering: allows faster insertion / word assignment while still allowing large vocabularies – Vocabulary tree [Nister & Stewenius, CVPR 2006] K. Grauman, Leibe Slide credit: J. B. Sivic 22

Example: Recognition with Vocabulary Tree construction: [Nister & Stewenius, CVPR’ 06] K. Grauman, B. Leibe Slide credit: David Nister 23

Vocabulary Tree Training: Filling the tree [Nister & Stewenius, CVPR’ 06] K. Grauman, B. Leibe Slide credit: David Nister 24

Vocabulary Tree Training: Filling the tree [Nister & Stewenius, CVPR’ 06] K. Grauman, B. Leibe Slide credit: David Nister 25

Vocabulary Tree Training: Filling the tree [Nister & Stewenius, CVPR’ 06] K. Grauman, B. Leibe Slide credit: David Nister 26

Vocabulary Tree Training: Filling the tree [Nister & Stewenius, CVPR’ 06] K. Grauman, B. Leibe Slide credit: David Nister 27

Vocabulary Tree Training: Filling the tree [Nister & Stewenius, CVPR’ 06] K. Grauman, B. Leibe Slide credit: David Nister 28

Vocabulary Tree Recognition Verification on spatial layout [Nister & Stewenius, CVPR’ 06] K. Grauman, B. Leibe Slide credit: David Nister 29

Vocabulary Tree: Performance Evaluated on large databases – Indexing with up to 1 M images Online recognition for database of 50, 000 CD covers – Retrieval in ~1 s Find experimentally that large vocabularies can be beneficial for recognition [Nister & Stewenius, CVPR’ 06] K. Grauman, Leibe Slide credit: J. B. Sivic 30

“Bag of visual words” Slide credit: J. Sivic

Next class • Overview of probability models in vision • Readings for next lecture: – Lecture notes will be uploaded • Readings for today: – Forsyth and Ponce chapter 17 32

Questions 33