On Visual Recognition Jitendra Malik UC Berkeley University

On Visual Recognition Jitendra Malik UC Berkeley University of California Berkeley Computer Vision Group

From Pixels to Perception Water Tiger outdoor wildlife Grass Sand back Tiger head eye tail University of California Berkeley legs shadow mouse Computer Vision Group

Object Category Recognition University of California Berkeley Computer Vision Group

Defining Categories • What is a “visual category”? – Not semantic – Working hypothesis: Two instances of the same category must have “correspondence” (i. e. one can be morphed into the other) • e. g. Four-legged animals – Biederman’s estimate of 30, 000 basic visual categories University of California Berkeley Computer Vision Group

Facts from Biological Vision • Timing • Abstraction/Generalization • Taxonomy and Partonomy University of California Berkeley Computer Vision Group

Detection can be very fast • On a task of judging animal vs no animal, humans can make mostly correct saccades in 150 ms (Kirchner & Thorpe, 2006) – Comparable to synaptic delay in the retina, LGN, V 1, V 2, V 4, IT pathway. – Doesn’t rule out feed back but shows feed forward only is very powerful University of California Berkeley Computer Vision Group

As Soon as You Know It Is There, You Know What It Is University of California Berkeley Grill-Spector & Kanwisher, Psychological Science, 2005 Computer Vision Group

Abstraction/Generalization • Configurations of oriented contours • Considerable toleration for small deformations University of California Berkeley Computer Vision Group

Attneave’s Cat (1954) Line drawings convey most of the information University of California Berkeley Computer Vision Group

Taxonomy and Partonomy • Taxonomy: E. g. Cats are in the order Felidae which in turn is in the class Mammalia – Recognition can be at multiple levels of categorization, or be identification at the level of specific individuals , as in faces. • Partonomy: Objects have parts, they have subparts and so on. The human body contains the head, which in turn contains the eyes. • These notions apply equally well to scenes and to activities. • Psychologists have argued that there is a “basic-level” at which categorization is fastest (Eleanor Rosch et al). • In a partonomy each level contributes useful information fro Universityrecognition. of California Computer Vision Group Berkeley

Matching with Exemplars • Use exemplars as templates • Correspond features between query and exemplar • Evaluate similarity score Database of Templates Query Image University of California Berkeley Computer Vision Group

Matching with Exemplars • Use exemplars as templates • Correspond features between query and exemplar • Evaluate similarity score Database of Templates Query Image Best matching template is a helicopter University of California Berkeley Computer Vision Group

3 D objects using multiple 2 D views � View University of California Berkeley selection algorithm from Belongie, Malik & Puzicha (2001) Computer Vision Group

Error vs. Number of Views University of California Berkeley Computer Vision Group

Three Big Ideas • Correspondence based on local shape/appearance descriptors • Deformable Template Matching • Machine learning for finding discriminative features University of California Berkeley Computer Vision Group

Three Big Ideas • Correspondence based on local shape/appearance descriptors • Deformable Template Matching • Machine learning for finding discriminative features University of California Berkeley Computer Vision Group

Comparing Pointsets University of California Berkeley Computer Vision Group

Shape Context Count the number of points inside each bin, e. g. : Count = 4. . . Count = 10 F Compact representation of distribution of points relative to each point University of California Berkeley (Belongie, Malik & Puzicha, 2001) Computer Vision Group

Shape Context University of California Berkeley Computer Vision Group

Geometric Blur (Local Appearance Descriptor) Berg & Malik '01 Compute sparse channels from image Extract a patch in each channel ~ Geometric Blur Descriptor University of California Berkeley (Idealized signal) Apply spatially varying blur and sub-sample Descriptor is robust to small affine distortions Computer Vision Group

Three Big Ideas • Correspondence based on local shape/appearance descriptors • Deformable Template Matching • Machine learning for finding discriminative features University of California Berkeley Computer Vision Group

Modeling shape variation in a category • D’Arcy Thompson: On Growth and Form, 1917 – studied transformations between shapes of organisms University of California Berkeley Computer Vision Group

Matching Example model University of California Berkeley target Computer Vision Group

Handwritten Digit Recognition • MNIST 60 000: – – – – linear: 12. 0% 40 PCA+ quad: 3. 3% 1000 RBF +linear: 3. 6% K-NN: 5% K-NN (deskewed): 2. 4% K-NN (tangent dist. ): 1. 1% SVM: 1. 1% Le. Net 5: 0. 95% University of California Berkeley • MNIST 600 000 (distortions): – Le. Net 5: 0. 8% – SVM: 0. 8% – Boosted Le. Net 4: 0. 7% • MNIST 20 000: – K-NN, Shape Context matching: 0. 63% Computer Vision Group

University of California Berkeley Computer Vision Group

EZ-Gimpy Results • 171 of 192 images correctly identified: 92 % University of California Berkeley horse spade smile join canvas here Computer Vision Group

Three Big Ideas • Correspondence based on local shape/appearance descriptors • Deformable Template Matching • Machine learning for finding discriminative features University of California Berkeley Computer Vision Group

Discriminative learning (Frome, Singer, Malik, 2006) weights on patch features in training images 83/400 79/400 distance functions from training images to any other images University of California Berkeley browsing, retrieval, classification Computer Vision Group

triplets • learn from relative similarity image j image i image k compare image-to-image distances want: image-to-image distances based on feature-toimage distances Computer Vision Group Berkeley University of California

focal image version image k image i (focal) dik 0. 2 . . . 0. 8 0. 2 - 0. 8 0. 4 image j 0. 3 0. 4 dij . . . = University of California Berkeley 0. 5 -0. 2 x. . . ijk Computer Vision Group

large-margin formulation slack variables like soft-margin SVM w constrained to be positive Computer Vision Group Berkeley. L 2 regularization University of California

Caltech-101 [Fei-Fei et al. 04] • 102 classes, 31 -300 images/class University of California Berkeley Computer Vision Group

retrieval example query image retrieval results: University of California Berkeley Computer Vision Group

Caltech 101 classification results (see Manik Verma’s talks for the best yet. . ) University of California Berkeley Computer Vision Group

15 training/class, 63. 2% University of California Berkeley Computer Vision Group

Conclusion • Correspondence based on local shape/appearance descriptors • Deformable Template Matching • Machine learning for finding discriminative features • Integrating Perceptual Organization and Recognition University of California Berkeley Computer Vision Group