Numerical Geometry of NonRigid Shapes Featurebased methods shape

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval probl 1 Feature-based methods and shape retrieval problems © Alexander & Michael Bronstein, 2006 -2009 © Michael Bronstein, 2010 tosca. cs. technion. ac. il/book 048921 Advanced topics in vision Processing and Analysis of Geometric Shapes EE Technion, Spring 2010

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval probl 2 Structure Global Local Metric Feature descriptors

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval probl 3 Combining local and global structures Pair-wise stress (global) Local structure Point-wise stress (local) can be geometric or photometric BBK 2008; Keriven, Torstensen 2009; Dubrovina, Kimmel 2010; Wang, B 2010

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval probl 4 Photometric stress Thorstenstein & Keriven 2009

Numerical Geometry of Non-Rigid Shapes Diffusion Geometry Heat kernels, encore Brownian motion on X starting at point x, measurable set C Heat kernel represents transition probability of the Brownian motion to be in C at time t Coifman, Lafon, Lee, Maggioni, Warner & Zucker 2005 5

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval probl 6 Intrinsic descriptors can be interpreted as probability of Brownian motion to return to the same point after time (represents “stability” of the point) Multiscale local shape descriptor (Heat kernel signature) Time (scale) Sun, Ovsjanikov & Guibas 2009

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval probl 7 Relation to curvature Sun, Ovsjanikov & Guibas 2009 for small t

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval probl 8 Heat kernel signatures represented in RGB space Sun, Ovsjanikov & Guibas 2009 Ovsjanikov, BB & Guibas 2009

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval probl 9 Heat kernel descriptors Invariant to isometric deformations Localized sensitivity to topological noise J. Sun, M. Ovsjanikov, L. Guibas, SGP 2009 M. Ovsjanikov, BB, L. Guibas, 2009

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 10 probl Scale invariance Original shape Scaled by HKS= Not scale invariant!

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 11 probl Scale-invariant heat kernel signature Log scale-space 0 -5 log + d/d 0 4 -0. 01 3 -0. 02 2 -0. 03 -10 1 -0. 04 -15 0 Fourier transform magnitude 100 200 t 300 Scaling = shift and multiplicative constant in HKS B, Kokkinos CVPR 2010 0 100 200 t 300 Undo scaling 0 0 2 4 6 8 10 12 14 16 18 20 =2 k /T Undo shift

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 12 probl Scale invariance Heat Kernel Signature B, Kokkinos 2009 Scale-invariant Heat Kernel Signature

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 13 probl Bending invariance Heat Kernel Signature B, Kokkinos CVPR 2010 Scale-invariant Heat Kernel Signature

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 14 probl Bending invariance Geodesic+HKS Wang, B 2010 Diffusion+HKS Commute+SI-HKS

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 15 probl Topology invariance Geodesic+HKS Wang, B 2010 Diffusion+HKS

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 16 probl Scale invariance Geodesic+HKS Wang, B 2010 Commute+SI-HKS

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 17 probl Invariance Rigid Geodesic metric Diffusion metric Commute-time metric Heat kernel signature (HKS) Scale-invariant HKS (SI-HKS) Wang, B 2010 Scale Inelastic Topology

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 18 probl

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 19 probl 3 D warehouse Person Text search Man, person, human Tagged shapes Content-based search Shapes without metadata

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 20 probl Content-based search problems ? ? Shape classification Invariant shape retrieval Semantic Geometric Variability of shape within category under transformation

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 21 probl Image vs shape retrieval Illumination View Missing data Deformation Topology Missing data

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 22 probl Bags of words construction architecture Italy France cathedral church basilica Paris Rome Gothic Roman Notre Dame de Paris is a Gothic cathedral in the fourth quarter of Paris, France. It was the first Gothic architecture cathedral, and its construction spanned the Gothic period. St. Peter’s basilica is the largest church in world, located in Rome, Italy. As a work of architecture, it is regarded as the best building of its age in Italy.

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 23 probl Bags of features Visual vocabulary Feature detector + descriptor Invariant to changes of the image Discriminative (tells different images apart)

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 24 probl Advantages n “Shape signature” n Easy to store n Easy to compare n Partial similarity possible

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 25 probl Images vs shapes Images Shapes Many prominent features Few prominent features Affine transforms, illumination, occlusions, resolution Non-rigid deformations, topology, missing parts, triangulation SIFT, SURF, MSER, DAISY, … Curvature, conformal factor, local distance histograms

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 26 probl “Shape. Google” Geometric words Feature descriptor “ Bag of words ” “ ” Geometric expressions Spatially-sensitive bag of features

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 27 probl Geometric vocabulary M. Ovsjanikov, BB, L. Guibas, 2009

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 28 probl Bags of features Geometric vocabulary Nearest neighbor in the descriptor space M. Ovsjanikov, BB, L. Guibas, 2009

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 29 probl Bags of features Geometric vocabulary Weighted distance to words in the vocabulary M. Ovsjanikov, BB, L. Guibas, 2009

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 30 probl Bags of features Statistics of different geometric words over the entire shape Shape distance = distance between bags of features M. Ovsjanikov, BB, L. Guibas, 2009

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 31 probl Bags of features 1 M. Ovsjanikov, BB, L. Guibas, 2009 Index in vocabulary 64

Michael Bronstein Shape Google: geometric words and expressions for invariant shape 32 re Statistical weighting Query q syzygy in astronomy means alignment of three bodies of the solar system along a straight or nearly straight line. a planet is in syzygy with the earth and sun when it is in opposition or conjunction. the moon is in syzygy with the earth and sun when it is new or full. Frequent in document = important Sivic & Zisserman 2003 BB, Carmon & Kimmel 2009 Database D and of when with or the a in is syzygy Rare in database = discriminative

Michael Bronstein Shape Google: geometric words and expressions for invariant shape 33 re Statistical weighting Query q Database D Significance of a term t Term frequency Inverse document frequency Weight bags of features by tf-idf Reduce the influence of non-important terms in dense descriptor Sivic & Zisserman 2003 BB, Carmon & Kimmel 2009

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 34 probl Expressions M. Ovsjanikov, BB, L. Guibas, 2009 Matrix is a science fiction movie released in 1999. Matrix refers to a simulated reality created by machines in order to subdue the human population. matrix decomposition matrix factorization science fiction canonical form In biological science, decomposition is the process of organisms to break down into simpler form of matter. Usually, decomposition occurs after death. matrix decomposition is a the of in to by science form In math science, matrix decomposition is a factorization of a matrix into some canonical form. Each type of decomposition is used in a particular problem.

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 35 probl Expressions M. Ovsjanikov, BB, L. Guibas, 2009 In particular matrix used type a some science, decomposition form a factorization of is canonical. matrix math decomposition is in a Each problem. into of matrix decomposition matrix factorization science fiction canonical form matrix decomposition is a the of in to by science form In math science, matrix decomposition is a factorization of a matrix into some canonical form. Each type of decomposition is used in a particular problem.

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 36 probl Visual expressions “Inquisitor King” Inquisitor, King Giuseppe Verdi, Don Carlo, Metropolitan Opera “King Inquisitor”

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 37 probl Geometric expressions Yellow “Yellow” No total order between points (only “far” and “near”) Geometric expression = a pair of spatially close geometric words M. Ovsjanikov, BB, L. Guibas, 2009

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 38 probl Spatially-sensitive bags of features Distribution of pairs of geometric words is the probability to find word at point and word at point Proximity between points and is the statistic of geometric expressions of the form Shape distance M. Ovsjanikov, BB, L. Guibas, 2009

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 39 probl Spatially-sensitive bags of features M. Ovsjanikov, BB, L. Guibas, 2009

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 40 probl SHREC 2010 dataset

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 41 probl BB et al, 3 DOR 2010 SHREC 2010 dataset

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 42 probl Shape. Google with HKS descriptor (m. AP %) BB et al, 3 DOR 2010

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 43 probl Shape. Google with SI-HKS descriptor (m. AP %) BB et al, 3 DOR 2010

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 44 probl Scale-invariant retrieval ? Scale 0. 7 Heat Kernel Signature Scale-Invariant Heat Kernel Signature Kokkinos, B 2009

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 45 probl Scale-invariant retrieval Scale 1. 3 Heat Kernel Signature Scale-Invariant Heat Kernel Signature Kokkinos, B 2009

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 46 probl Scale-invariant retrieval Local scale Heat Kernel Signature Scale-Invariant Heat Kernel Signature Kokkinos, B 2009

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 47 probl Structure Global Local Metric Feature descriptors

Michael Bronstein Diffusion geometry for shape recognition Laplacian embedding Represent the shape using finite-dimensional Laplacian eigenmap Ambiguities! Beylkin & Niyogi 2003 Coifman, Lafon, Lee, Maggioni, Warner & Zucker 2005 Rustamov 2007 48

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 49 probl Global point signature (GPS) embedding Represent the shape using distribution of Euclidean distances in the Laplacian embedding space (=commute time distances) 0. 05 0. 15 0. 25 0. 35 Deformation- and scale-invariant No ambiguities related to eigenfunction permutations and sign No need to compare multidimensional embeddings Osada, Funkhouser, Chazelle & Dobkin 2002 Rustamov 2007 0. 45

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 50 probl Diffusion distance distributions Represent the shape using distribution of diffusion distances 0. 5 Deformation-invariant Mahmoudi & Sapiro 2009 1 1. 5 2 2. 5 3 How to select the scale? 3. 5 -3 x 10

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 51 probl Spectral shape distance Kernel Distance Distribution Dissimilarity Aggregation BB 2010

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 52 probl Spectral shape distance Kernel BB 2010 Distance Distribution Aggregation Dissimilarity

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 53 probl Spectral shape distance Kernel Distance Aggregation Distribution Diffusion distance BB 2010 Dissimilarity

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 54 probl Particular case I: Rustamov GPS embedding Kernel BB 2010 Distance Aggregation Distribution Dissimilarity

Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval 55 probl Particular case II: Mahmoudi&Sapiro Kernel BB 2010 Distance Aggregation Distribution Dissimilarity