Stanford CS 223 B Computer Vision Winter 2006

Stanford CS 223 B Computer Vision, Winter 2006 Lecture 8 Structure From Motion Professor Sebastian Thrun CAs: Dan Maynes-Aminzade, Mitul Saha, Greg Corrado Slides by: Gary Bradski, Intel Research and Stanford SAIL

Structure From Motion features camera Recover: structure (feature locations), motion (camera extrinsics) Sebastian Thrun Stanford University CS 223 B Computer Vision

Structure From Motion (1) [Tomasi & Kanade 92] Sebastian Thrun Stanford University CS 223 B Computer Vision

Structure From Motion (2) [Tomasi & Kanade 92] Sebastian Thrun Stanford University CS 223 B Computer Vision

Structure From Motion (3) [Tomasi & Kanade 92] Sebastian Thrun Stanford University CS 223 B Computer Vision

Structure From Motion (4 a): Images Marc Pollefeys Sebastian Thrun Stanford University CS 223 B Computer Vision

Structure From Motion (4 b) Marc Pollefeys Sebastian Thrun Stanford University CS 223 B Computer Vision

Structure From Motion n Problem 1: – Given n points pij =(xij, yij) in m images – Reconstructure: 3 -D locations Pj =(xj, yj, zj) – Reconstruct camera positions (extrinsics) Mi=(Aj, bj) n Problem 2: – Establish correspondence: c(pij) Sebastian Thrun Stanford University CS 223 B Computer Vision

SFM: General Formulation O X Z Sebastian Thrun -x f Stanford University CS 223 B Computer Vision

SFM: Bundle Adjustment O X Z Sebastian Thrun -x f Stanford University CS 223 B Computer Vision

Bundle Adjustment n n SFM = Nonlinear Least Squares problem Minimize through – – n Gradient Descent Conjugate Gradient Gauss-Newton Levenberg Marquardt (!) Prone to local minima Sebastian Thrun Stanford University CS 223 B Computer Vision

Count # Constraints vs #Unknowns n n n m camera poses n points 2 mn point constraints 6 m+3 n unknowns Suggests: need 2 mn 6 m + 3 n But: Can we really recover all parameters? ? ? Sebastian Thrun Stanford University CS 223 B Computer Vision

How Many Parameters Can’t We Recover? We can recover all but… 0 3 6 7 8 10 12 n m nm Place Your Bet! Sebastian Thrun Stanford University CS 223 B Computer Vision

Count # Constraints vs #Unknowns n n n m camera poses n points 2 mn point constraints 6 m+3 n unknowns Suggests: need 2 mn 6 m + 3 n But: Can we really recover all parameters? ? ? – Can’t recover origin, orientation (6 params) – Can’t recover scale (1 param) n Thus, we need 2 mn 6 m + 3 n - 7 Sebastian Thrun Stanford University CS 223 B Computer Vision

Are done? n No, bundle adjustment has many local minima. Sebastian Thrun Stanford University CS 223 B Computer Vision

The “Trick Of The Day” n n n Replace Perspective by Orthographic Geometry Replace Euclidean Geometry by Affine Geometry Solve SFM linearly (“closed” form, globally optimal) Post-Process to make solution Euclidean Post-Process to make solution perspective By Tomasi and Kanade, 1992 Sebastian Thrun Stanford University CS 223 B Computer Vision

Orthographic Camera Model Limit of Pinhole Model: Extrinsic Parameters Rotation Orthographic Projection Sebastian Thrun Stanford University CS 223 B Computer Vision

Orthographic Projection Limit of Pinhole Model: Orthographic Projection Sebastian Thrun Stanford University CS 223 B Computer Vision

The Orthographic SFM Problem subject to Sebastian Thrun Stanford University CS 223 B Computer Vision

The Affine SFM Problem subjectdrop to the constraints Sebastian Thrun Stanford University CS 223 B Computer Vision

Count # Constraints vs #Unknowns n n n m camera poses n points 2 mn point constraints 8 m+3 n unknowns Suggests: need 2 mn 8 m + 3 n But: Can we really recover all parameters? ? ? Sebastian Thrun Stanford University CS 223 B Computer Vision

How Many Parameters Can’t We Recover? We can recover all but… 0 3 6 7 8 10 12 n m nm Place Your Bet! Sebastian Thrun Stanford University CS 223 B Computer Vision

The Answer is (at least): 12 Sebastian Thrun Stanford University CS 223 B Computer Vision

Points for Solving Affine SFM Problem n m camera poses n points n Need to have: 2 mn 8 m + 3 n-12 n Sebastian Thrun Stanford University CS 223 B Computer Vision

Affine SFM Fix coordinate system by making p 0=origin Rank Theorem: Q has rank 3 Proof: Sebastian Thrun Stanford University CS 223 B Computer Vision

2 m elements The Rank Theorem n elements Sebastian Thrun Stanford University CS 223 B Computer Vision

Singular Value Decomposition Sebastian Thrun Stanford University CS 223 B Computer Vision

Affine Solution to Orthographic SFM Gives also the optimal affine reconstruction under noise Sebastian Thrun Stanford University CS 223 B Computer Vision

Back To Orthographic Projection Find C and d for which constraints are met Search in 12 -dim space (instead of 8 m + 3 n-12) Sebastian Thrun Stanford University CS 223 B Computer Vision

Back To Projective Geometry Orthographic (in the limit) Projective Sebastian Thrun Stanford University CS 223 B Computer Vision

Back To Projective Geometry O X Z -x f Optimize Using orthographic solution as starting point Sebastian Thrun Stanford University CS 223 B Computer Vision

The “Trick Of The Day” n n n Replace Perspective by Orthographic Geometry Replace Euclidean Geometry by Affine Geometry Solve SFM linearly (“closed” form, globally optimal) Post-Process to make solution Euclidean Post-Process to make solution perspective By Tomasi and Kanade, 1992 Sebastian Thrun Stanford University CS 223 B Computer Vision

Structure From Motion n Problem 1: – Given n points pij =(xij, yij) in m images – Reconstructure: 3 -D locations Pj =(xj, yj, zj) – Reconstruct camera positions (extrinsics) Mi=(Aj, bj) n Problem 2: – Establish correspondence: c(pij) Sebastian Thrun Stanford University CS 223 B Computer Vision

The Correspondence Problem View 1 Sebastian Thrun View 2 Stanford University View 3 CS 223 B Computer Vision

Correspondence: Solution 1 n Track features (e. g. , optical flow) n …but fails when images taken from widely different poses Sebastian Thrun Stanford University CS 223 B Computer Vision

Correspondence: Solution 2 n n Start with random solution A, b, P Compute soft correspondence: p(c|A, b, P) Plug soft correspondence into SFM Reiterate See Dellaert/Seitz/Thorpe/Thrun, Machine Learning Journal, 2003 Sebastian Thrun Stanford University CS 223 B Computer Vision

Example Sebastian Thrun Stanford University CS 223 B Computer Vision

Results: Cube Sebastian Thrun Stanford University CS 223 B Computer Vision

Animation Sebastian Thrun Stanford University CS 223 B Computer Vision

Tomasi’s Benchmark Problem Sebastian Thrun Stanford University CS 223 B Computer Vision

Reconstruction with EM Sebastian Thrun Stanford University CS 223 B Computer Vision

3 -D Structure Sebastian Thrun Stanford University CS 223 B Computer Vision

Correspondence: Alternative Approach n Ransac [Fisher/Bolles] = Random sampling and consensus Sebastian Thrun Stanford University CS 223 B Computer Vision

Summary SFM n Problem – Determine feature locations (=structure) – Determine camera extrinsic (=motion) n Two Principal Solutions – Bundle adjustment (nonlinear least squares, local minima) – SVD (through orthographic approximation, affine geometry) n Correspondence – (RANSAC) – Expectation Maximization Sebastian Thrun Stanford University CS 223 B Computer Vision