Computer and Robot Vision II Chapter 18 Object
Computer and Robot Vision II Chapter 18 Object Models And Matching Presented by: 傅楸善 & 張博思 0911 246 313 r 94922093@ntu. edu. tw 指導教授: 傅楸善 博士 Digital Camera and Computer Vision Laboratory Department of Computer Science and Information Engineering National Taiwan University, Taipei, Taiwan, R. O. C.
18. 1 Introduction l object recognition: one of most important aspects of computer vision DC & CV Lab. CSIE NTU
18. 2 Two-Dimensional Object Representation l l 2 D shape analysis useful in machine vision application: medical image analysis aerial image analysis manufacturing DC & CV Lab. CSIE NTU
18. 2 Two-Dimensional Object Representation l l l 2 D shape representation classes: global features local features boundary description skeleton 2 D parts DC & CV Lab. CSIE NTU
18. 2. 1 Global Feature Representation l l 2 D object: can be thought of as binary image value 1: pixels of object value 0: pixels outside object 2 D shape features: area, perimeter, moments, circularity, elongation DC & CV Lab. CSIE NTU
18. 2. 1 Global Feature Representation l l Shape Recognition by Moments f: binary image function : 2 D shape digital th moment of S: l l area of S: number of pixels of S DC & CV Lab. CSIE NTU
18. 2. 1 Global Feature Representation l l l moment invariants: functions of moments invariant under shape transform prefer moment invariants: under translation, rotation, scaling skewing center of gravity of S: DC & CV Lab. CSIE NTU
18. 2. 1 Global Feature Representation l central moments: translation invariant normalized central moments of S: l th moment of S: DC & CV Lab. CSIE NTU
18. 2. 1 Global Feature Representation l seven functions that are rotation invariant DC & CV Lab. CSIE NTU
18. 2. 1 Global Feature Representation l l l Shape Recognition with Fourier Descriptors Fourier descriptors: another way for extracting features from 2 D shapes Fourier descriptors: defined to characterize boundary The main idea is to represent the boundary as a function of one variable , expand in its Fourier series, and use the coefficients of the series as Fourier descriptors (FDs). finite number of FDs: can be used to describe the shape DC & CV Lab. CSIE NTU
18. 2. 1 Global Feature Representation DC & CV Lab. CSIE NTU
18. 2. 1 Global Feature Representation DC & CV Lab. CSIE NTU
18. 2. 1 Global Feature Representation DC & CV Lab. CSIE NTU
18. 2. 2 Local Feature Representation l l l l 2 D object characterized by: local features, attributes, relationships most commonly used local features: holes, corners holes: found by connected component procedure followed by boundary tracing holes: detected by binary mathematical morphology, if hole shapes known hole properties: areas, shapes corner detection: can be performed on binary or gray tone image corner property: angle at which lines meet DC & CV Lab. CSIE NTU
l joke DC & CV Lab. CSIE NTU
18. 2. 3 Boundary Representation l l l boundary representation: most common representation for 2 D objects 3 main ways to represent object boundary: 1. sequence of points 2. chain code 3. sequence of line segments DC & CV Lab. CSIE NTU
18. 2. 3 Boundary Representation l l l The Boundary as a Sequence of Points boundary points from border-following or edgetracking algorithms interest points: boundary points with special property useful in matching DC & CV Lab. CSIE NTU
18. 2. 3 Boundary Representation l l l l The Chain Code Representation chain encoding: can be used at any level of quantization chain encoding: saves space required for row and column coordinates boundary encoded: first quantized by placing over square grid side length: determines resolution of encoding marked points: grid intersections closest to curve and used in encoding : marks starting point of curve DC & CV Lab. CSIE NTU
18. 2. 3 Boundary Representation l chain encoding of boundary curve DC & CV Lab. CSIE NTU
18. 2. 3 Boundary Representation l line segments: links: to be used to approximate the curve encoding scheme: eight possible directions assigned integer between 0, 7 chain: chain encoding: in the form or DC & CV Lab. CSIE NTU
18. 2. 3 Boundary Representation l l l length of chain code with n chains: can be simply estimated as n : number of odd chain codes : number of even chain codes : number of corners : unbiased estimate of perimeter length Freeman suggested: DC & CV Lab. CSIE NTU
18. 2. 3 Boundary Representation l l l The Boundary as a Sequence of Line Segments line segment sequence: after boundary segmented into near-linear portion line segment sequence: used in shape recognition or other matching tasks : coordinate location where pair of lines meet : angle magnitude where pair of lines meet sequence of junction points to represent line segment sequence DC & CV Lab. CSIE NTU
18. 2. 3 Boundary Representation sequence of junction points representing l test object T l l an association goal: given O, T, to find F satisfying i < j F(i) < F(j) or F(i) = missing or F(j) = missing DC & CV Lab. CSIE NTU
18. 2. 4 Skeleton Representation l l strokes: long, sometimes thin parts forming shapes line segments that characterize the strokes of set of characters DC & CV Lab. CSIE NTU
18. 2. 4 Skeleton Representation l l l symmetric axis transform: set of maximal circular disks inside object symmetric axis: locus of centers of these maximal disks symmetric axes of the characters DC & CV Lab. CSIE NTU
18. 2. 4 Skeleton Representation l l symmetric axis: one example of skeleton description of 2 D object symmetric axis of rectangle: consists of five line segments not single line symmetric axis: extremely sensitive to noise symmetric axis: difficult to use in matching DC & CV Lab. CSIE NTU
DC & CV Lab. CSIE NTU
18. 2. 4 Skeleton Representation l l axis of smoothed local symmetries: separate definition for skeleton local symmetry: midpoint P of line segment BA joining pair of points A, B : angle between BA and outward normal at A : angle between BA and inward normal at B DC & CV Lab. CSIE NTU
18. 2. 4 Skeleton Representation l point P that is local symmetry with respect to boundary points A and B DC & CV Lab. CSIE NTU
18. 2. 4 Skeleton Representation l l l axes: spines: loci of local symmetries maximal w. r. t. forming smooth curve cover of axis: portion of shape subtended by axis cover properly contained in another cover: second axis subsumes first DC & CV Lab. CSIE NTU
18. 2. 4 Skeleton Representation l symmetric axes of local symmetry of a rectangle DC & CV Lab. CSIE NTU
18. 2. 4 Skeleton Representation l axes of smoothed local symmetries of several objects DC & CV Lab. CSIE NTU
18. 2. 5 Two-Dimensional Part Representation l l l parts, attributes, interrelationships: form structural description of shape nuclei: regions where primary convex subset overlap nuclei: shaded areas of overlap DC & CV Lab. CSIE NTU
18. 2. 5 Two-Dimensional Part Representation l decomposition of shape into primary convex subsets and nuclei DC & CV Lab. CSIE NTU
18. 2. 5 Two-Dimensional Part Representation l l l near-convexity: allows noisy distorted instances to have same decompositions , : two points on object boundary relation: visibility relation if line completely interior to object boundary, the graph-theoretic clustering to determine clusters of visibility relation DC & CV Lab. CSIE NTU
18. 2. 5 Two-Dimensional Part Representation l decomposition of three similar shapes into nearconvex pieces DC & CV Lab. CSIE NTU
l joke DC & CV Lab. CSIE NTU
18. 3 Three-Dimensional Object Representations DC & CV Lab. CSIE NTU
18. 3. 1 Local Features Representation l l l range data: obtained from laser range finder, light striping, stereo, etc. from depth, try to infer surfaces, edges, corners, holes, other features 3 D matching more difficult than 2 D because of occlusion DC & CV Lab. CSIE NTU
18. 3. 2 Wire Frame Representation l wire frame model: 3 D object model with only edges of object DC & CV Lab. CSIE NTU
l two-color hyperboloid and its line drawing 18. 3. 2 Wire Frame Representation DC & CV Lab. CSIE NTU
18. 3. 2 Wire Frame Representation DC & CV Lab. CSIE NTU
18. 3. 2 Wire Frame Representation l l Necker cube: lower-vertical face or upper vertical face closer to viewer Schroder staircase: viewed either from above or from below DC & CV Lab. CSIE NTU
l two well-known ambiguous line drawings DC & CV Lab. CSIE NTU
l two well-known ambiguous line drawings DC & CV Lab. CSIE NTU
l two well-known ambiguous line drawings DC & CV Lab. CSIE NTU
l inherent ambiguity of line drawing owing to complete loss of depth DC & CV Lab. CSIE NTU
18. 3. 2 Wire Frame Representation l l general-viewpoint assumption: none of the following situations 1. two vertices of scene objects represented at same picture point 2. two scene edges seen as single line in picture 3. vertex seen exactly in line with unrelated edge DC & CV Lab. CSIE NTU
18. 3. 2 Wire Frame Representation l l l general-viewpoint assumption: heart of line-drawing interpretation viewpoint in perspective projection: center of projection viewpoint in orthographic projection: direction of projection DC & CV Lab. CSIE NTU
l subjective contours of Kanizsa: white occluding triangle in space DC & CV Lab. CSIE NTU
DC & CV Lab. CSIE NTU
18. 3. 2 Wire Frame Representation l line labels for visible projections of surface-normal discontinuities: DC & CV Lab. CSIE NTU
DC & CV Lab. CSIE NTU
l four basic ways in which three planar surfaces can form polyhedron vertex DC & CV Lab. CSIE NTU
l all possible distinct appearances of trihedral vertices of polyhedra DC & CV Lab. CSIE NTU
l complete junction catalog for line drawings of trihedral -vertex polyhedra DC & CV Lab. CSIE NTU
l complete junction catalog for line drawings of trihedral -vertex polyhedra DC & CV Lab. CSIE NTU
l joke DC & CV Lab. CSIE NTU
18. 3. 3 Surface-Edge-Vertex Representation l l VISIONS system: Visual Integration by Semantic Interpretation of Natural Scenes PREMIO system: Prediction in Matching Images to Objects PREMIO 3 D object model: hierarchical, relational model with five levels 5 levels: world, object, face/edge/vertex, surface/boundary, arc/2 D, 1 D piece DC & CV Lab. CSIE NTU
18. 3. 3 Surface-Edge-Vertex Representation l l world level: arrangement of different objects in world object level: arrangement of different faces, edges, vertices forming objects face level: describes face in terms of surfaces and boundaries surface level: specifies elemental pieces forming surfaces DC & CV Lab. CSIE NTU
18. 3. 3 Surface-Edge-Vertex Representation l l 2 D piece level: describes pieces and specifies arcs forming boundaries 1 D piece level: describes elemental pieces forming arcs SDS: spatial data structure A/V: attribute-value table DC & CV Lab. CSIE NTU
DC & CV Lab. CSIE NTU
18. 3. 4 Sticks, Plates, and Blobs l l sticks, plates, blobs model: rough models of 3 D objects used in rough-matching sticks: long, thin parts with only one significant dimension stick: cannot bend very much stick: has two logical endpoints, set of interior points, center of mass DC & CV Lab. CSIE NTU
18. 3. 4 Sticks, Plates, and Blobs l l l l plates: flattish wide parts with two nearly flat surfaces plates: have two significant dimensions plate surfaces: cannot fold very much plate: has set of edge points, set of surface points, center of mass blobs: parts with three significant dimensions blob: can be bumpy but cannot have concavities blob: set of surface points and center of mass sticks, plates, blobs: near-convex DC & CV Lab. CSIE NTU
DC & CV Lab. CSIE NTU
18. 3. 4 Sticks, Plates, and Blobs l l l l attribute-value table: contains global attributes simple-parts relation: lists the parts and their attributes connects-supports relation: gives connections between pairs of parts triples relation: specifies connections between three parts at a time parallel relation: lists pairs of parts that are parallel perpendicular relation: lists pairs of parts that are perpendicular TYPE: 1 for stick, 2 for plate, 3 for blob DC & CV Lab. CSIE NTU
l full relational structure of sticks-platesblobs model of chair object DC & CV Lab. CSIE NTU
DC & CV Lab. CSIE NTU
18. 3. 5 Generalized Cylinder Representation l l generalized cylinder: volumetric primitive defined by axis and cross-section cross section: swept along axis, creating a solid e. g. actual cylinder: generalized cylinder whose axis is straight-line segment and whose cross section is circle of constant radius e. g. cone: generalized cylinder whose axis is straight-line segment and cross section is circle with radius initially zero to maximum DC & CV Lab. CSIE NTU
18. 3. 5 Generalized Cylinder Representation l l l e. g. rectangular solid: generalized cylinder whose axis is straight line segment and cross section is constant rectangle e. g. torso: generalized cylinder whose axis is circle and whose cross section is constant circle generalized cylinder representation: uses generalized cylinders as primitives DC & CV Lab. CSIE NTU
18. 3. 5 Generalized Cylinder Representation l l l surface-edge-vertex model: very precise sticks-plates-and-blobs model: very rough generalized cylinder model: somewhere in between DC & CV Lab. CSIE NTU
18. 3. 5 Generalized Cylinder Representation l l person: modeled roughly as cylinders for head, torso, arms, legs dotted lines: axes of cylinders DC & CV Lab. CSIE NTU
18. 3. 6 Superquadric Representation l l superquadrics: lumps of clay deformable and can be glued into object models superquadric models: mainly used with range data DC & CV Lab. CSIE NTU
18. 3. 6 Superquadric Representation l Superquadrics are a flexible family of 3 -dimensional parametric objects, useful for geometric modeling. By adjusting a relatively few number of parameters, a large variety of shapes may be obtained. DC & CV Lab. CSIE NTU
l range data image of (a) a doll, (b) its superquadric fit (c), (d) wire frame DC & CV Lab. CSIE NTU
l joke DC & CV Lab. CSIE NTU
18. 3. 7 Octree Representation l l l octree encoding: geometric modeling technique used to represent 3 D objects octree encoding: used in computer vision, robotics, computer graphics octree hierarchical: 8 -ary tree structure each node in octree corresponds to cubic region of universe full: if cube is completely enclosed by 3 D object empty: if cube contains no part of object DC & CV Lab. CSIE NTU
18. 3. 7 Octree Representation l l partial: if cube partly intersects object full, empty, partial correspond to “black”, “white”, “gray” in quadtrees node with label full or empty: has no children partial: has eight children representing partition of cube into octants DC & CV Lab. CSIE NTU
DC & CV Lab. CSIE NTU
18. 3. 8 The Extended Gaussian Image l l l 3 D object: collection of surface normals, one at each point of object surface planar surface: all points on surface map to same surface normal convex with positive curvature everywhere: distinct surface normal everywhere Gaussian sphere: unit sphere set of surface normals mapped to Gaussian sphere: tail at center head outward Gaussian image of object: resultant set of points on Gaussian sphere DC & CV Lab. CSIE NTU
DC & CV Lab. CSIE NTU
18. 3. 8 The Extended Gaussian Image l l for planar objects: Gaussian image not invertible, not precise enough for use : small surface patch of object : corresponding surface patch on Gaussian sphere Gaussian curvature K: DC & CV Lab. CSIE NTU
18. 3. 8 The Extended Gaussian Image l l l : point on Gaussian sphere corresponding to point (u, v) on object surface extended Gaussian image: planar region: Gaussian curvature 0, point mass in extended Gaussian image DC & CV Lab. CSIE NTU
18. 3. 9 View-Class Representation l l l view classes: each representing set of viewpoints sharing some property: e. g. same object surfaces visible property: e. g. same line segments visible property: e. g. relational distances between relational structures are similar characteristic views: sets producing topologically isomorphic line drawings DC & CV Lab. CSIE NTU
18. 3. 9 View-Class Representation l three view classes of cube producing topologically isomorphic line drawings DC & CV Lab. CSIE NTU
DC & CV Lab. CSIE NTU
18. 3. 9 View-Class Representation l l l aspect graph of object: graph structure where 1. each node represents topologically distinct view of object 2. a node for each such view of object 3. each arc represents a visual event at transition 4. there is an arc for each such transition DC & CV Lab. CSIE NTU
DC & CV Lab. CSIE NTU
18. 4 General Frameworks for Matching l l matching: finding correspondence between two entities consistent labeling procedures: examples of matching algorithms DC & CV Lab. CSIE NTU
18. 4. 1 Relational-Distance Approach to Matching l l l l relational distance: compares two structures and determines similarity Relational-Distance Definition : relational description : sequence of relations : set of parts of entity being described : relation indicating various relationships among parts : relational description with part set A : relational description with part set B DC & CV Lab. CSIE NTU
18. 4. 1 Relational-Distance Approach to Matching l l assumption: |A| = |B|, otherwise add dummy parts to smaller set f: any one-one, onto mapping from A to B N: positive integer composition R f of relation R with function f: DC & CV Lab. CSIE NTU
18. 4. 1 Relational-Distance Approach to Matching l f: maps parts from set A to parts from set B structural error of f for ith pair of corresponding relations in , : l total error of f with respect to l relational distance l , : between DC & CV Lab. CSIE NTU , :
18. 4. 1 Relational-Distance Approach to Matching l best mapping from total error to DC & CV Lab. CSIE NTU : mapping f that minimizes
18. 4. 1 Relational-Distance Approach to Matching l Relational-Distance Examples best mapping from to l for this mapping: l DC & CV Lab. CSIE NTU is
18. 4. 1 Relational-Distance Approach to Matching l two digraphs whose relational distance is 3 DC & CV Lab. CSIE NTU
DC & CV Lab. CSIE NTU
18. 4. 1 Relational-Distance Approach to Matching l l l Relational Distance as a Metric relational distance: used to determine similarity of unknown object to model relational distance: used to compare object models to group models f relational isomorphism: if f one-one, onto from A to B and E(f) = 0 f: A B relational isomorphism: , isomorphic GD: relational-distance measure DC & CV Lab. CSIE NTU
18. 4. 1 Relational-Distance Approach to Matching l l l arbitrary relational descriptions , , : metric property of GD: 1. isomorphic 2. 3. DC & CV Lab. CSIE NTU
18. 4. 1 Relational-Distance Approach to Matching l l Attributed Relational Descriptions and Relational Distance extend relational description and relational distance to include properties of parts properties of the whole properties of these relationships DC & CV Lab. CSIE NTU
l joke DC & CV Lab. CSIE NTU
18. 4. 2 Ordered Structural Matching l definition of ordering on primitives: greatly reduces complexity of search DC & CV Lab. CSIE NTU
18. 4. 3 Hypothesizing and Testing with Viewpoint Consistency Constraint l viewpoint consistency constraint: The locations of all projected model features in an image must be consistent with projection from a single viewpoint. DC & CV Lab. CSIE NTU
18. 4. 4 View-Class Matching l l l if 3 D object represented by view-class model, matching divided into 2 stages: 1. determining view class of object 2. determining precise viewpoint within that view class DC & CV Lab. CSIE NTU
18. 4. 4 View-Class Matching l l l Determining View Class relational pyramid: hierarchical relational structure to represent view class Level-1 primitives: straight- and curved-line segments Level-2 relations: junctions and loops Level-3 relations: adjacency, collinearity, junction parallelness, loop-inside-loop DC & CV Lab. CSIE NTU
18. 4. 4 View-Class Matching l l Pose Determination within View Class relational pyramid: hierarchical, relational structure to constrain matching DC & CV Lab. CSIE NTU
18. 4. 5 Affine-Invariant Matching l l l set of interest points lying in plane rotation matrix relating model reference frame to camera reference frame: translation of object reference frame to camera reference frame: DC & CV Lab. CSIE NTU
18. 4. 5 Affine-Invariant Matching l l l f: distance between image plane and center of perspectivity : observed image data points by perspective projection: when translation : in z-direction large compared with DC & CV Lab. CSIE NTU
18. 4. 5 Affine-Invariant Matching l l l A: 2 x 2 (scaling, rotation, skewing) matrix b: 2 D (translation) vector affine 2 D correspondence: Aw + b DC & CV Lab. CSIE NTU
18. 4. 5 Affine-Invariant Matching l l Affine Transformation of Points in a Plane necessary and sufficient to define plane uniquely: 3 noncollinear points DC & CV Lab. CSIE NTU
18. 4. 5 Affine-Invariant Matching l l The Hummel-Wolfson-Lamdan Matching Algorithm to match noncollinear triplets in model interest points with scene: preprocessing: convert model interest points into affineinvariant model recognition: match model against image using affine representation DC & CV Lab. CSIE NTU
18. 4. 5 Affine-Invariant Matching l l l Shortcomings of the Affine-Invariant Matching Technique affine-invariant matching technique: mathematically sound in noiseless case shortcomings of affine-invariant matching in practice: 1. if three noncollinear points not numerically stable, points not reliable 2. coordinates of detected interest points: noisy in real image 3. partial object symmetries may cause wrong matching DC & CV Lab. CSIE NTU
18. 4. 5 Affine-Invariant Matching l An Explicit Noise Model and Optimal Voting DC & CV Lab. CSIE NTU
18. 5 Model Database Organization l l organize database of models: to allow rapid access to most likely candidate group similar relational models into clusters and choose representative arrows: indicate mapping from parts of object 2 to parts of other objects cluster of object models whose representative is object 2 DC & CV Lab. CSIE NTU
DC & CV Lab. CSIE NTU
l. END DC & CV Lab. CSIE NTU
- Slides: 115
