Shape Representation and Similarity Longin Jan Latecki Computer

Shape Representation and Similarity Longin Jan Latecki Computer and Information Sciences Dept. , Temple Univ. , latecki@temple. edu With contributions from: Rolf Lakaemper, Zygmunt Pizlo, Eamonn Keogh, Remco Veltkamp Based on IS&T/SPIE EI 2006 Short Course #SC 754; Jan 2006

Overview 1. The role of shape in object recognition 2. Example applications 3. What is shape? 4. Shape similarity measures 5. Comparison based on MPEG-7 data set 6. Contour-based shape representation 7. Partial shape similarity

Why Shape ? • Shape is probably the most important property that is perceived about objects. It allows to predict more facts about an object than other features, e. g. color (Palmer 1999) • Thus, recognizing shape is crucial for object recognition. In some applications it may be the only feature present, e. g. logo recognition

Why Shape ? These objects are recognized by…

Why Shape ? These objects are recognized by… Texture Color X X Context X Shape X X X

Object Recognition Process: Source: 2 D image of a 3 D object Object Segmentation Contour Extraction Contour Cleaning, e. g. , Evolution Contour Segmentation Matching: Correspondence of Visual Parts

Why Shape ? Several applications use shape processing: • Object recognition • Object comparison • Image retrieval • Processing of pictorial information • Video compression (e. g. , MPEG-7) …

0 200 400 600 800 Courtesy of Eamonn Keogh, UCR 1000 1200 1400

Ve M M B R B G G W R D D rve e Br usta edta Hool Born Oran eard hite ray- Juven Olive Mand Red R Ring Mant ed HHom e g o o i n n l e e to a a r G zza's ched l mon ck G o Or utan d Sa faced ecke ile B Babo rill) uffed tail L ed H owle sapi m a a r o k k e G G ib j ree k ib n i Mo ns ens Le emu wle Sak d Ow boo on n M onke monk ueno ey bon guta uveni i m r n r M nke Skh i l. M ur fem ma n le n n ey on y y ul V on o o juv key ale le nk n key eni ey key le All these are in the genus Cercopithecus, except for the skull identified as being either a Vervet or Green monkey, both of which belong in the Genus of Chlorocebus which is in the same Tribe (Cercopithecini) as Cercopithecus. Tribe Cercopithecini Cercopithecus De Brazza's Monkey, Cercopithecus neglectus Moustached Guenon, Cercopithecus cephus Red-tailed Monkey, Cercopithecus ascanius Chlorocebus Green Monkey, Chlorocebus sabaceus Vervet Monkey, Chlorocebus pygerythrus These are in the same species Bunopithecus hooloc (Hoolock Gibbon) All these are in the tribe Papionini Tribe Papionini Genus Papio – baboons Genus Mandrillus- Mandrill These are in the Genus Pongo These are in the family Lemuridae All these are in the family Cebidae Family Cebidae (New World monkeys) Subfamily Aotinae Aotus trivirgatus Subfamily Pitheciinae sakis Black Bearded Saki, Chiropotes satanas White-nosed Saki, Chiropotes albinasus These are in the genus Alouatta These are in the same species Homo sapiens (Humans) Note: This is a phenogram, not a phylogenetic tree. For the current best phylogenetic tree, see the Tree of Life project http: //tolweb. org/tree/phylogeny. html

Lowland Gorilla (Gorilla graueri) Mountain Gorilla (Gorilla gorilla beringei)

Application: Human Activity Recognition Trajectory Rao and M. Shah, University of Central Florida

Activity Recognition: Typical Set of Trajectories

Another Shape Application

A, B: one individual tall male (bounce in B is truncated) C: relatively short female with a radically different style • Notice that DTW is forced to map A’s bounce section to the end of sequence B • MVM is free to ignore sections that do not have natural correspondence • Hence, MVM is able to produce the more natural clustering

Shape-Based Retrieval of Vertebra Sameer Antani, Daniel M. Krainak, et al. User text Query Shape User sketch Shape database search, compare, & retrieve similar segment Results User example image

Query § § Provide an image example Extract shape with the same process as in the original segmentation

Retrieval results ranked by shape similarity

Retrieval in multimedia databases Query by Shape / Texture / …(Color / Keyword)

Blobworld: http: //elib. cs. berkeley. edu/photos/blobworld/start. html Query by Shape / Texture / Location / Color Selected Blob Query: by Color and Texture of Blob Result: Blobs with similar Color and Texture Satisfying ?

Blobworld Selected Blob Query: by Shape of Blob Result: …are these shapes similar ? Satisfying ?

What is Shape ? Plato, "Meno", 380 BC: • "figure is the only existing that is found always following color“ • "figure is limit of solid"

What is Shape ? Shape is not only perceived by visual means: § § tactical sensors can also provide shape information that are processed in a similar way. robots’ range sensor provide shape information, too.

What is Shape ? Typical problems: • How to describe shape ? • What is the matching transformation? • No one-to-one correspondence • Occlusion • Noise

What is Shape ? … let’s start with some properties easier to agree on: • Shape describes a spatial region Shape is a (the ? ) specific part of spatial cognition • Typically addresses 2 D space why ?

What is Shape ? • the original 3 D (? ) object

What is Shape ? • 3 D => 2 D projection

What is Shape ? Moving on from the naive understanding, some questions arise: • Is there a maximum size for a shape to be a shape? • Can a shape have holes? • Does shape always describe a connected region? • How to deal with/represent partial shapes (occlusion / partial match) ?

What is Shape ? Shape or Not ? Continuous transformation from shape to no shape: Is there a point when it stops being a shape?

What is Shape ? Shape or Not ? Continuous transformation from shape to two shapes: Is there a point when it stops being a single shape?

What is Shape ? But there’s no doubt that a single, connected region is a shape. Right ?

What is Shape ? A single, connected region. But a shape ? A question of scale !

What is Shape ? • There’s no easy, single definition of shape • In difference to geometry, arbitrary shape is not covered by an axiomatic system • Different applications in object recognition focus on different shape related features • Special shapes can be handled • Typically, applications in object recognition employ a similarity measure to determine a plausibility that two shapes correspond to each other

Shape Similarity So the new question is: What is Shape Similarity ? or How to Define a Similarity Measure?

Shape Similarity which similarity measure, depends on which required properties, depends on which particular matching problem, depends on which application. However, there still may exist a single universal measure that we have build in our heads.

Shape Similarity …which application Simple Recognition (yes / no) . . . robustness Common Rating (best of. . . ) Analytical Rating (best of, but. . . ) . . . invariance to basic transformations

Shape Similarity …which problem • computation problem: d(A, B) • decision problem: d(A, B) <e ? • decision problem: is there g: d(g(A), B) <e ? • optimization problem: find g: min d(g(A), B)

Similarity Measure Requirements to a similarity measure • Should not incorporate context knowledge, thus computes generic shape similarity

Similarity Measure Requirements on a similarity measure • Must be in accord with human perception • Must be able to deal with noise • Must be invariant with respect to basic transformations Scaling (or resolution) Next: Strategy Rotation Rigid / non-rigid deformation

Similarity Measure Some other aspects worth consideration: • Similarity of structure • Similarity of area Can all these aspects be expressed by a single number?

Similarity Measure Desired Properties of a Similarity Function d (Basri et al. 1998) • d should be a metric • d should be continuous • d should be invariant (to…)

Properties Metric Properties S set of patterns Metric: d: S ´ S ® R satisfying 1. Nonnegativity: " x , yÎS, d(x, y)≥ 0 2. Self-identity: " xÎS, d(x, x)=0 3. Uniqueness: d(x, y)=0 implies x=y 4. Symmetry: " x, yÎS, d(x, y)= d(y, x) 5. Triangle inequality: " x, y, zÎS, d(x, z)£d(x, y)+d(y, z) • S with fixed metric d is called metric space

Properties

Properties

Properties In general: • a similarity measure in accordance with human perception is NOT a metric. This leads to deep problems in further processing, e. g. clustering, since most of these algorithms need metric spaces !

Properties

Properties

Similarity Measures Classes of Similarity Measures: Similarity Measure depends on • Shape Representation • Boundary • Area (discrete: = point set) • Structural (e. g. Skeleton) • Comparison Model • feature vector, also called shape signature • direct

Similarity Measures direct feature based Boundary Spring model, Cum. Angular Function, Chain Code, Arc Decomposition Central Dist. , Fourier Distance histogram … Area (point set) Hausdorff … Moments Zernike Moments … Structure Skeleton … ---

MPEG-7 • We focus on measures evaluated on MPEG-7 CE-Shape-1 data set. K. B. Sun and B. J. Super. Classification of Contour Shapes. . . CVPR 2005.

MPEG-7 Retrieval Rate in Part B of the MPEG-7 CE-Shape-1 data set: Each shape is as a query, and the retrieval rate is expressed by Bull’s Eye Percentage: the fraction of images that belong to the same class in the top 40 matches. Strong shape variations within the same classes imply that no shape similarity measure achieves a 100% retrieval rate.

Retrieval results obtained on the MPEG-7 CE-Shape-1 Part B data set

Improving shape retrieval by semi-supervised learning with graph transduction

Courtesy Remco C. Veltkamp

Contour-Based Similarity Measures • We focus on similarity measures based on contour representation. Contour is a 1 D manifold. It is mapped to a 1 D function from arc length to R: • Distance to the centroid • Tangent direction • Curvature

Comparison of object shape based on object contours Object contours are naturally obtained in Computer Vision, Robot Navigation, and other applications as polylines (polygonal curves). Shape similarity reduces to similarity of polylines. Shape similarity of polylines is not so simple: • simple 1 -1 vertex correspondence does not work • a scale problem

Cognitive Similarity Requirements Since polylines are obtained as boundary parts of objects in usually noisy sensor data (e. g. , digital images): 1. two similar polylines do not need to have the same number of vertices, i. e. , do not have to be of comparable level of detail, 2. do not have to be of comparable size, 3. may have only a subpart that is similar and that has a significant contribution to their shape = visual part

Vector Contour Comparison Representation It requires that the object is segmented or edges are extracted. Contour is given as list of Euclidean coordinates: 0, 0; 1, 0; 2, 1; 2, 2; 3, 3; …

Vector Chain Comparison Code A binary image can be converted into a ‘chain code’ representing the boundary. The boundary is traversed and a string representing the turns is constructed. 3 2 1 4 C 0 5 6 7 5, 6, 6, 3, 3, 4, 3, 2, 3, 4, 5, 3, …

Vector Chain Comparison Code Digital curves suffer from effects caused by digitalization, e. g. rotation:

Distance to the centroid as function of angle or arc length

0 200 400 600 800 1000 1200 Skull classification by Eamonn Keogh is based on distance to the centroid representation. 1400

Distance Histogram, example for a 3 D surface

Tangent or Turn Angle Space Transformation from image space to tangent or turn angle space The angle with x-axis of the tangent to the contour at each contour point s is denoted Ψ(s), where s is arc length normalized. angle with x-axis of the tangent arc length of the contour

Shape Similarity Measure: Arkin at al. PAMI 1991.

Shape Comparison: Measure Drawback: not adaptive to unequally distributed noise

Shape Comparison: Contour Segmentation Solution: use this measure only locally, i. e. , apply only to corresponding parts:

Shape Similarity

Correspondence of visual parts: Results

Visual parts = max. convex arcs obtained by Discrete Curve Evolution (DCE)

Visual Parts Play a Key Role in Human Shape Perception Divide a plane curve into parts at negative minima of curvature D. D. Hoffman and W. A. Richards (1984) Parts of Recognition, Cognition 18, 65– 96, 1984. D. D. Hoffman and M. Singh. (1997) Salience of Visual Parts. Cognition 63, p. 29 -78, 1997. K. Siddiqi, K. J. Tresness, and B. B. Kimia. Parts of visual form: Ecological and psychophysical aspects. Perception 25, p. 399 -424, 1996.

Correspondence of visual parts: non-rigid deformation L. J. Latecki and R. Lakämper: Shape Similarity Measure Based on Correspondence of Visual Parts. IEEE Trans. Pattern Analysis and Machine Intelligence 22, 2000.

ISS Database ISS-Database http: //knight. cis. temple. edu/~shape Rolf Lakaemper and Longin Jan Latecki

The Interface (JAVA – Applet)

The Sketchpad: Query by Shape

The First Guess: Different Shape - Classes

Selected shape defines query by shape – class

Result

ISS Database ISS: Query by Shape / Texture Sketch of Shape Query: by Shape only Result: Satisfying ?

Vector Comparison Curvature • Curvature is the rate of change of slope. • It is also the reciprocal of the radius of the local-fit circle.

Experimental Results (cont. ) Whole Sequence Matching (faces datasets) " Converting shapes into time series in a Face dataset: trace the head outline and the local turn angle is used to create a 1 -dimensional signal

Curvature Scale Space (CSS), included in MPEG 7 standard Mokhtarian, Abbasi, Kittler

CSS Inflection points (curvature zero crossings)

CSS • • • Smooth (continuous) boundary curve using convolution with an increasing Gaussian kernel. Use the run length position of curvature zero-crossings on the boundary as index set for each kernel size; creating the ‘Curvature Scale Space’. The maxima of the CSS are used for shape representation Similarity of shape is defined by difference between the maxima of the CSS representation

CSS • Shape Similarity is reduced to correspondence of points on the plane.

CSS • Similarity of shape is defined by difference between the maxima of the CSS representation

CSS Problems of CSS: • Convex shapes don’t have inflection points • Different shapes can have identical CSS !

Symmetry Axis Representation by Liu, Kohn and Geiger Shape Axis (SA) SA-Tree The idea goes back to Blum’s Medial Axis Transformation

Vector Partial. Comparison Shape Similarity All similarity measures shown cannot deal with occlusions, noise, segmentation errors, change of point of view, and non-rigid deformations. They are useful (and used) for specific applications, but are not sufficient to deal with these problems. Solution: Part – based similarity !

Partial Shape Similarity Partial shape similarity of contours is needed for object recognition!

Partial Shape Similarity Partial shape similarity

Partial Shape Similarity • Partial match: only part of query appears in part of database shape Courtesy of R. Veltkamp

Partial Shape Similarity Motivation § § § Once a significant visual part is recognized the whole recognition process is strongly constrained in possible topdown object models. (H 1) object recognition is preceded by, and based on recognition of visual parts. (H 2): contour extraction is driven by shape similarity to a known shape.

What do you see?

With grouping constraints we can see (i. e. , recognize the object).

Salient visual parts can influence the object recognition (Singh and Hoffman 2001)

Salient visual parts can influence the object recognition (Singh and Hoffman 2001)

Salient visual parts can influence the object recognition (Singh and Hoffman 2001)

Partial Shape Similarity Object contours § § Psychophysical and neurophysiological studies provide an abundance of evidence that contours of objects are extracted in early processing stages of human visual perception. Contours play a central role in the Gestalt-theory.

Partial Shape Similarity Contour detection is a difficult inverse problem § A given image could be produced by infinitely many possible 3 D scenes. In order to produce a unique, stable and accurate interpretation, the visual system must use a priori constraints (see Pizlo, 2001 for a review). § § The solution is obtained by optimizing a cost function which consists of two general terms: 1. how close the solution is to the visual data 2. how well the constraints are satisfied

Partial Shape Similarity

Challenges of Partial Shape Similarity Given only a part (of a shape ), find similar shapes (1) length problem, (2) scale problem, (3) distortion problem Query Shape Target Shape

Experimental results: 90% retrieval accuracy with single query part