Multiple View Geometry in Computer Vision Marc Pollefeys

Multiple View Geometry in Computer Vision Marc Pollefeys Comp 290 -089

Multiple View Geometry A a a f(a, b, c)=0 (a, b) A (a, b, c) (a, b) c c b b (reconstruction) (calibration) (transfer) c

Course objectives • To understand the geometric relations between multiple views of scenes. • To understand the general principles of parameter estimation. • To be able to compute scene and camera properties from real world images using state-of-the-art algorithms.

Relation to other vision/image courses • Focuses on geometric aspects • No image processing • Comp 254: Image Processing an Analysis Mostly orthogonal to this course, complementary • Comp 256: Computer Vision (fall 2003) Will be much broader, based on new book: “Computer Vision: a modern approach” David Forsyth and Jean Ponce

Material Textbook: Multiple View Geometry in Computer Vision by Richard Hartley and Andrew Zisserman Cambridge University Press Alternative book: The Geometry from Multiple Images by Olivier Faugeras and Quan-Tuan Luong MIT Press On-line tutorial: http: //www. cs. unc. edu/~marc/tutorial. pdf http: //www. cs. unc. edu/~marc/tutorial/

Learning approach • • • read the relevant chapters of the books and/or reading assignements before the course. In the course the material will then be covered in detail and motivated with real world examples and applications. Small hands-on assignements will be provided to give students a "feel" of the practical aspects. Students will also read and present some seminal papers to provide a complementary view on some of the covered topics. Finally, there will also be a project where students will implement an algorithm or approach using concepts covered by the course. Grade distribution • • • Class participation: 20% Hands-on assignments: 10% Paper presentation: 10% Implementation assignment/project: 40% Final: 20%

Applications • Match. Moving Compute camera motion from video (to register real an virtual object motion)

Applications • 3 D modeling

Content • Background: Projective geometry (2 D, 3 D), Parameter estimation, Algorithm evaluation. • Single View: Camera model, Calibration, Single View Geometry. • Two Views: Epipolar Geometry, 3 D reconstruction, Computing F, Computing structure, Plane and homographies. • Three Views: Trifocal Tensor, Computing T. • More Views: N-Linearities, Multiple view reconstruction, Bundle adjustment, autocalibration, Dynamic Sf. M, Cheirality, Duality

Multiple View Geometry course schedule (tentative) Jan. 7, 9 Intro & motivation Projective 2 D Geometry Jan. 14, 16 (no course) Projective 2 D Geometry Jan. 21, 23 Projective 3 D Geometry Parameter Estimation Jan. 28, 30 Parameter Estimation Algorithm Evaluation Camera Models Camera Calibration Feb. 11, 13 Single View Geometry Epipolar Geometry Feb. 18, 20 3 D reconstruction Fund. Matrix Comp. Feb. 25, 27 Structure Comp. Planes & Homographies Trifocal Tensor Three View Reconstruction Mar. 18, 20 Multiple View Geometry Multiple. View Reconstruction Mar. 25, 27 Bundle adjustment Papers Auto-Calibration Papers Dynamic Sf. M Papers Feb. 4, 6 Mar. 4, 6 Apr. 1, 3 Apr. 8, 10 Apr. 15, 17 Cheirality Papers Apr. 22, 24 Duality Project Demos

Fast Forward! • Quick overview of what is coming…

Background La reproduction interdite (Reproduction Prohibited), 1937, René Magritte.

Projective 2 D Geometry • Points, lines & conics • Transformations • Cross-ratio and invariants

Projective 3 D Geometry • Points, lines, planes and quadrics • Transformations • П∞, ω∞ and Ω ∞

Estimation How to compute a geometric relation from correspondences, e. g. 2 D trafo • Linear (normalized), non-linear and Maximum Likelihood Estimation • Robust (RANSAC)

Evaluation and error analysis How good are the results we get error • Bounds on performance residual • Covariance propagation & Monte-Carlo estimation

Single-View Geometry The Cyclops, c. 1914, Odilon Redon

Camera Models Mostly pinhole camera model but also affine cameras, pushbroom camera, …

Camera Calibration • Compute P given (m, M) (normalized) linear, MLE, … • Radial distortion

More Single-View Geometry • Projective cameras and planes, lines, conics and quadrics. • Camera center and camera rotation • Camera calibration and vanishing points, calibrating conic and the IAC

Single View Metrology Antonio Criminisi

Two-View Geometry The Birth of Venus (detail), c. 1485, Sandro Botticelli

Epipolar Geometry Fundamental matrix Essential matrix

Two-View Reconstruction

Epipolar Geometry Computation (normalized) linear: minimal: MLE: RANSAC … and automated two view matching

Rectification Warp images to simplify epipolar geometry

Structure Computation • Points: Linear, optimal, direct optimal • Also lines and vanishing points

Planes and Homographies Relation between plane and H given P and P’ Relation between H and F, H from F, F from H The infinity homography H∞

Three-View Geometry The Birth of Venus (detail), c. 1485, Sandro Botticelli

Trifocal Tensor

Three View Reconstruction • (normalized) linear • minimal (6 points) • MLE (Gold Standard)

Multiple-View Geometry The Birth of Venus (detail), c. 1485, Sandro Botticelli

Multiple View Geometry Quadrifocal tensor 81 parameters, but only 29 DOF!

Multiple View Reconstruction • Affine factorization • Projective factorization

Multiple View Reconstruction • Sequential reconstruction

Bundle Adjustment Maximum Likelyhood Estimation for complete structure and motion P 1 P 2 P 3 M U 1 U 2 W U 3 WT 12 xm 3 xn (in general much larger) V

Bundle Adjustment Maximum Likelyhood Estimation for complete structure and motion U-WV-1 WT WT V 11 xm 3 xn

Bundle adjustment No bundle adjustment Bundle adjustment needed to avoid drift of virtual object throughout sequence Bundle adjustment (including radial distortion)

Auto-calibration * * projection constraints

Dynamic Structure from Motion

Cheirality Oriented projective geometry Allows to use fact that points are in front of camera • to recover quasi-affine reconstruction • to determine order for image warping • to determine orientation for rectification with epipoles in images • etc.

Duality Gives possibility to interchange role of P and X in algorithms

Contact information Marc Pollefeys, Room 205 marc@cs. unc. edu Tel. 962 1845