Shape Matching and Object Recognition using Shape Contexts
Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U. C. Berkeley (joint work with S. Belongie, J. Puzicha, G. Mori) University of California Berkeley Computer Vision Group
Outline • Shape matching and isolated object recognition • Scaling up to general object recognition University of California Berkeley Computer Vision Group
Biological Shape • D’Arcy Thompson: On Growth and Form, 1917 – studied transformations between shapes of organisms University of California Berkeley Computer Vision Group
Deformable Templates: Related Work • Fischler & Elschlager (1973) • Grenander et al. (1991) • Yuille (1991) • von der Malsburg (1993) University of California Berkeley Computer Vision Group
. . . Matching Framework model target • Find correspondences between points on shape • Estimate transformation • Measure similarity University of California Berkeley Computer Vision Group
Comparing Pointsets University of California Berkeley Computer Vision Group
Shape Context Count the number of points inside each bin, e. g. : Count = 4. . . Count = 10 F Compact representation of distribution of points relative to each point University of California Berkeley Computer Vision Group
Shape Context University of California Berkeley Computer Vision Group
Shape Contexts • Invariant under translation and scale • Can be made invariant to rotation by using local tangent orientation frame • Tolerant to small affine distortion – Log-polar bins make spatial blur proportional to r Cf. Spin Images (Johnson & Hebert) - range image registration University of California Berkeley Computer Vision Group
Comparing Shape Contexts Compute matching costs using Chi Squared distance: Recover correspondences by solving linear assignment problem with costs Cij [Jonker & Volgenant 1987] University of California Berkeley Computer Vision Group
. . . Matching Framework model target • Find correspondences between points on shape • Estimate transformation • Measure similarity University of California Berkeley Computer Vision Group
Thin Plate Spline Model • 2 D counterpart to cubic spline: • Minimizes bending energy: • Solve by inverting linear system • Can be regularized when data is inexact Duchon (1977), Meinguet (1979), Wahba (1991) University of California Berkeley Computer Vision Group
Matching Example model University of California Berkeley target Computer Vision Group
Outlier Test Example University of California Berkeley Computer Vision Group
Synthetic Test Results Fish - deformation + noise ICP University of California Berkeley Shape Context Fish - deformation + outliers Chui & Rangarajan Computer Vision Group
. . . Matching Framework model target • Find correspondences between points on shape • Estimate transformation • Measure similarity University of California Berkeley Computer Vision Group
Terms in Similarity Score • Shape Context difference • Local Image appearance difference – orientation – gray-level correlation in Gaussian window – … (many more possible) • Bending energy University of California Berkeley Computer Vision Group
Object Recognition Experiments • Kimia silhouette dataset • Handwritten digits • COIL 3 D objects (Nayar-Murase) • Human body configurations • Trademarks University of California Berkeley Computer Vision Group
Shape Similarity: Kimia dataset University of California Berkeley Computer Vision Group
Number correct Quantitative Comparison rank University of California Berkeley Computer Vision Group
Handwritten Digit Recognition • MNIST 60 000: – – – – linear: 12. 0% 40 PCA+ quad: 3. 3% 1000 RBF +linear: 3. 6% K-NN: 5% K-NN (deskewed): 2. 4% K-NN (tangent dist. ): 1. 1% SVM: 1. 1% Le. Net 5: 0. 95% University of California Berkeley • MNIST 600 000 (distortions): – Le. Net 5: 0. 8% – SVM: 0. 8% – Boosted Le. Net 4: 0. 7% • MNIST 20 000: – K-NN, Shape Context matching: 0. 63% Computer Vision Group
University of California Berkeley Computer Vision Group
Results: Digit Recognition 1 -NN classifier using: Shape context + 0. 3 * bending + 1. 6 * image appearance University of California Berkeley Computer Vision Group
COIL Object Database University of California Berkeley Computer Vision Group
Error vs. Number of Views University of California Berkeley Computer Vision Group
Prototypes Selected for 2 Categories Details in Belongie, Malik & Puzicha (NIPS 2000) University of California Berkeley Computer Vision Group
Editing: K-medoids • Input: similarity matrix • Select: K prototypes • Minimize: mean distance to nearest prototype • Algorithm: – iterative – split cluster with most errors • Result: Adaptive distribution of resources (cfr. aspect graphs) University of California Berkeley Computer Vision Group
Error vs. Number of Views University of California Berkeley Computer Vision Group
Human body configurations University of California Berkeley Computer Vision Group
Automatically Locating Keypoints • User marks keypoints on exemplars • Find correspondence with test shape • Transfer keypoint position from exemplar to the test shape. University of California Berkeley Computer Vision Group
Results University of California Berkeley Computer Vision Group
Trademark Similarity University of California Berkeley Computer Vision Group
Outline • Shape matching and isolated object recognition • Scaling up to general object recognition – Many objects (Mori, Belongie & Malik, CVPR 01) – Gray scale matching (Berg & Malik, CVPR 01) – Objects in scenes (scanning or segmentation) University of California Berkeley Computer Vision Group
Mori, Belongie, Malik (CVPR 01) • Fast Pruning – Given a query shape, quickly return a shortlist of candidate matches – Database of known objects will be large: ~30000 • Detailed Matching – Perform computationally expensive comparisons on only the few shapes in the shortlist University of California Berkeley Computer Vision Group
Representative Shape Contexts • Match using only a few shape contexts – Don’t need to compare every one University of California Berkeley Computer Vision Group
Snodgrass Results University of California Berkeley Computer Vision Group
Results University of California Berkeley Computer Vision Group
Conclusion • Introduced new matching algorithm matching based on shape contexts and TPS • Robust to outliers & noise • Forms basis of object recognition technique that performs well in a variety of domains using exactly the same algorithm University of California Berkeley Computer Vision Group
- Slides: 38