Rank Conditions in Multiple View Geometry by Yi

Rank Conditions in Multiple View Geometry by Yi Ma Rene Vidal (EECS. UCB), Kun Huang (ECE. UIUC) Jana Kosecka (CS. GMU), Robert Fossum (Math. UIUC) Perception & Decision Laboratory Decision & Control Group, CSL Image Formation & Processing Group, Beckman Electrical & Computer Engineering Dept. , UIUC http: //black. csl. uiuc. edu/~yima

FORMULATION: camera model and multiple images POINT FEATURE: multilinear constraints v. s. rank conditions GENERALIZATION: line, plane, space of higher dimensions APPLICATIONS: matching, transfer, structure from motion CONCLUSIONS AND ON-GOING WORK

FORMULATION - Fundamental Geometric Problem Input: Corresponding images (of “features”) in multiple images. Output: Camera motion, camera calibration, object structure. Jana’s apartment Image courtesy of Jana Kosecka

FORMULATION – Orthodox View (State of the Art) curve & surface plane line algorithm point algebra geometry projective affine Euclidean perspective orthographic 2 views omni-directional 3 views 4 views m views

FORMULATION – Literature Review “Multiple” view geometry theory • • • Two views: Three views: Four views: Multiple views: Higher dimension: Kruppa’ 13, Longuet-Higgins’ 81, Huang & Faugeras’ 89, … Spetsakis & Aloimonos’ 90, Shashua’ 94, Hartley’ 94, … Triggs’ 95, Shashua’ 00, … Heyden & Astrom’ 97’ 98, … Wolf & Shashua’ 01 Multiple view geometry algorithms • • Euclidean: Maybank’ 93, Weng, Ahuja & Huang’ 93, … Affine: Quan & Kanade’ 96, … Projective: Triggs’ 96, … Orthographic: Tomasi & Kanade’ 92, … Recent books on multiple view geometry 1. Multiple view geometry in computer vision, Hartley & Zisserman’ 00. 2. Geometry of multiple images, Faugeras, Luong & Papadopoulo’ 01.

FORMULATION – A Provocative Stand: One Theorem for All & More? curve & surface plane line algorithm point algebra geometry projective affine Euclidean Rank Conditions perspective orthographic 2 views omni-directional 3 views 4 views m views

FORMULATION – A Little Notation: Hat Operator

FORMULATION: camera model and multiple images POINT FEATURE: multilinear constraints v. s. rank conditions GENERALIZATION: line, plane, space of higher dimensions APPLICATIONS: matching, transfer, structure from motion CONCLUSIONS AND ON-GOING WORK

POINT FEATURE – Pinhole Camera Model Homogeneous coordinates of a 3 -D point Homogeneous coordinates of its 2 -D image Projection of a 3 -D point to an image plane

POINT FEATURE – Multiple View Structure From Motion Given corresponding images of points: recover everything else from equations associated to each: . . .

POINT FEATURE – Conventional Multilinear (Multifocal) Constraints For images of the same 3 -D point : (leading to the conventional approach) Multilinear constraints among 2, 3, 4 -wise views

POINT FEATURE – Rank Condition on the Multiple View Matrix WLOG, choose camera frame 1 as the reference Multiple View Matrix Lemma [Rank Condition for Point Features] (generic) (degenerate) Let then and are linearly dependent.

POINT FEATURE – Rank Condition Implies Bilinear Constraints Fact: Given non-zero vectors Hence, we have These constraints are only necessary but NOT sufficient!

POINT FEATURE – Rank Condition Implies Trilinear Constraints Fact: Given non-zero vectors Hence, we have • These constraints are only necessary but NOT sufficient! • However, there is NO further relationship among quadruple wise views. Quadrilinear constraints hence are redundant!

POINT FEATURE – Uniqueness of Pre-image (Bilinear Constraints) “Bilinear means pair-wise coplanar”: except in a rare coplanar case: Trifocal plane Rectilinear motion

POINT FEATURE – Uniqueness of Pre-image (Trilinear Constraints) “Trilinear means triple-wise incidental”: except in a rare collinear case:

POINT FEATURE – Uniqueness of Pre-image (Multiple View Matrix) Proposition [Uniqueness of Pre-image] Given vectors with respect to camera frames, they correspond to a unique point in the 3 -D space if the rank of the matrix is 1. If the rank is 0, the point is determined up to a line on which all the camera centers must lie. “point incidence condition” . . .

POINT FEATURE – Geometric Interpretation is the “depth” of the point that give the same relative to the camera center. Points matrix are on a sphere of radius

FORMULATION: camera model and multiple images POINT FEATURE: multilinear constraints v. s. rank conditions GENERALIZATION: line, plane, space of higher dimensions APPLICATIONS: matching, transfer, structure from motion CONCLUSIONS AND ON-GOING WORK

GENERALIZATION – Line Feature Homogeneous representation of a 3 -D line Homogeneous representation of its 2 -D co-image Projection of a 3 -D line to an image plane

GENERALIZATION – Multiple View Matrix: Line v. s. Point Features Line Features

GENERALIZATION – Rank Conditions: Line v. s. Point

GENERALIZATION – Incidence Relations Among Features Incidence conditions: inclusion, intersection, and restriction. “nothing but incidence conditions” . . .

GENERALIZATION – What is the Matrix? Theorem 1 [The Universal Rank Condition] for images of a point on a line: -Multi-nonlinear constraints among 3, 4 -wise images. -Multi-linear constraints among 2, 3 -wise images.

GENERALIZATION – Implications of the Rank Condition Examples: Case 1: a line reference Case 2: a point reference • All previously known constraints are theorem’s instantiations. • It implies more constraints and is now complete. • Degenerate configurations if and only if a drop of rank.

GENERALIZATION – Global Multiple-View Analysis (Example)

GENERALIZATION – A Family of Incidental Lines (A Corollary) each can randomly take the image of any of the lines: Nonlinear constraints among up to four views . . .

GENERALIZATION – Restriction to a Plane (A Corollary) Homogeneous representation of a 3 -D plane Corollary [Coplanar Features] Rank conditions on the new extended remain exactly the same!

GENERALIZATION – Multiple View Matrix: Coplanar Features Given that a point and line features lie on a plane in 3 -D space: In addition to previous constraints, it simultaneously gives homography:

GENERALIZATION – Coplanar Point/Line Duality On the plane any two points determine a line and any two lines determine a point. Corollary [Point/Line Duality] For planar features, points and lines are hence equivalent!

GENERALIZATION – Intrinsic Rank Condition for Planar Features Multi-quadratic equations in given images 4 coplanar points + 3 virtual points = 7 effective points

GENERALIZATION – Euclidean Imbedding of Dynamic Scenes Before: Now: Time base This is a perspective projection from <n to <2.

GENERALIZATION – Rank Condition in Space of High Dimension Theorem 2 [Generalized Rank Condition for from <n to <k]

GENERALIZATION – Rank Conditions for Curves & Surfaces Differentiating the matrix of a point (moving) along a curve: gives rise to a rank condition for curve. intensity level sets region boundaries . . .

GENERALIZATION – From Tangent to Point-wise Correspondence gives rise to a set of ordinary differential equations: • The rank deficiency condition for relates points and tangent lines of a curve. • Solving these equations establishes point-wise correspondence for image curves and in fact eventually for surface as well. • gives constraints on curvature and normals of image curves.

FORMULATION: camera model and multiple images POINT FEATURE: multilinear constraints v. s. rank conditions GENERALIZATION: line, plane, space of higher dimensions APPLICATIONS: matching, transfer, structure from motion CONCLUSIONS AND ON-GOING WORK

APPLICATIONS – Multiple View Matching Test Given the projection matrix Then for vectors associated to camera frames.

APPLICATIONS – Transferring Images to a New View Given the projection matrix associated to Then for given vectors camera frames. So given images, rank deficiency adds a linear constraint on the image. Computing the kernel of gives the new image.

APPLICATIONS – Motion and Structure from Multiple Views Given images of points:

APPLICATIONS – SVD Based Four Step Algorithm for SFM

APPLICATIONS – Utilize All Incidence Conditions (Example) Three edges intersect at each vertex. .

APPLICATIONS – Utilizing All Incidence Conditions (Simulations)

APPLICATIONS – Landing a Helicopter (Experiments) Images courtesy of Omid Shakernia, UCB

FORMULATION: camera model and multiple images POINT FEATURE: multilinear constraints v. s. rank conditions GENERALIZATION: line, plane, space of higher dimensions APPLICATIONS: matching, transfer, structure from motion CONCLUSIONS AND ON-GOING WORK

CONCLUSIONS AND ON-GOING WORK • Rank condition simplifies, unifies and completes existing algebraic results on multiview constraints (no tensor and algebraic geometry). • Rank condition is for all features, all incidence relations, all number of views, all types of projection, all (linear) spaces of arbitrary dimensions. • Rank condition intrinsically ties together geometry and algebra. • Metric multiple view geometry. • Consistent, optimal and robust algorithms for correspondence, image-based view synthesis, and reconstruction of motion & structure. • Apply to dynamical scenes: human motion; sensor networks… • Real-time algorithms for autonomous navigation and robotic control.

Rank Conditions in Multiple View Geometry by yima@uiuc. edu Rene Vidal (EECS. UCB), Kun Huang (ECE. UIUC) Jana Kosecka (CS. GMU), Robert Fossum (Math. UIUC) “Rank conditions on the multiple view matrix”, submitted to IJCV. “General rank conditions in multiple view geometry”, submitted to D&CG “An invitation to 3 -D vision”, Ma, Soatto, Kosecka, and Sastry, 2002.