TwoView Multibody Structure from Motion Ren Vidal and
Two-View Multibody Structure from Motion René Vidal and Xiaodong Fan Center for Imaging Science Johns Hopkins University
Structure and motion recovery n n Input: Corresponding points in multiple images Output: camera motion, scene structure, camera calibration Structure = 3 D surface Motion = camera position and orientation Copyright @ Rene Vidal 2
2 -D and 3 -D motion segmentation n n A static scene: multiple 2 -D motion models n A dynamic scene: multiple 3 -D motion models Given an image sequence, determine n n Number of motion models (affine, Euclidean, etc. ) Motion model: affine (2 -D) or Euclidean (3 -D) Segmentation: model to which each pixel belongs n Copyright @ Rene Vidal 3
Taxonomy of problems n 2 -D Layered representation n Multiple rigid motions in two perspective views n n Multibody trifocal tensor: Hartley-Vidal: CVPR’ 04 Multiple rigid motions in multiple affine views n n n Two transparent motions: Izawa’ 92 Complex polynomial differentiation: Vidal-Ma ECCV’ 04 Multiple rigid motions in three perspective views n n Two-body fundamental matrix: Wolf-Shashua CVPR’ 01 Multibody fundamental matrix: Vidal et al. ECCV’ 02, IJCV’ 04, CVPR’ 03 Multiple 2 -D and 3 -D motion models in two perspective views n n Probabilistic approaches: Jepson-Black’ 93, Ayer-Sawhney ’ 95, Darrel. Pentland’ 95, Weiss-Adelson’ 96, Weiss’ 97, Torr-Szeliski-Anandan ’ 99 Initialization: Wang-Adelson ’ 94, Irani-Peleg ’ 92, Shi-Malik ‘ 98 Factorization-based: Costeira-Kanade’ 98, Gear’ 98 Han-Kanade’ 01, Kanatani’ 01 -02 -04 Power. Factorization + GPCA: Vidal-Hartley: CVPR’ 04 Multiple rigid motions in multiple perspective views Copyright @ Rene Vidal 4
Prior work on 2 -D motion segmentation n Probabilistic approaches (Jepson-Black’ 93, Ayer-Sawhney ’ 95, Darrel. Pentland’ 95, Weiss-Adelson’ 96, Weiss’ 97, Torr-Szeliski-Anandan ’ 99) n n Generative model: data membership + motion model Obtain motion models using Expectation Maximization n E-step: Given motion models, segment image data M-step: Given data segmentation, estimate motion models How to initialize iterative algorithms? n Local methods n n n One model per window + Kmeans Aperture, motion across boundaries Global methods n n n (Wang-Adelson ’ 94) (Irani-Peleg ‘ 92) Dominant motion: fit one motion model to all pixels Look for misaligned pixels & fit a new model to them Spectral clustering: normalized cuts n (Shi-Malik ‘ 98) Similarity matrix based on motion profile Copyright @ Rene Vidal 5
Prior work on 3 -D motion segmentation n Affine cameras, multiple views n n (Costeira-Kanade ’ 98, Gear ’ 98, Han. Kanade ’ 01, Kanatani ’ 01 -02, Han-Kanade ‘ 00) Perspective cameras (statistical approaches) n n n Factorization methods: Motion segmentation & model selection (Torr’ 98) Multiple rigid motions using NCuts+EM (Feng-Perona ’ 98) Perspective cameras (geometric approaches) n n n Points in a line (Shashua-Levin ‘ 01) Points in a plane moving linearly at constant speed Points in moving in planes (Sturm ’ 02) Segmentation of two rigid motions (Wolf-Shashua ‘ 01) n Copyright @ Rene Vidal 6
Data: point correspondences n Given point correspondences in multiple views, estimate n n Number of motion models Motion models: affine, projective, fundamental matrices, etc. Segmentation: motion model associated which each Mathematics of the problem depends on n n Number of frames (2, 3, multiple) Projection model (affine, perspective) Motion model (affine, translational, planar motion, rigid motion) 3 -D structure (planar or not) Copyright @ Rene Vidal 7
Segmentation of 3 -D fundamental matrices n Rotation: Translation: Epipolar constraint n Multiple motions n n Multibody epipolar constraint Copyright @ Rene Vidal 8
Multibody Fundamental Matrix Multibody epipolar constraint Given rank condition Embedding Lifting Copyright @ Rene Vidal n linear system F Embedding Multibody Fundamental matrices 9 epipolar transfer epipole
Multibody Fundamental Matrix 1 -body motion n-body motion Copyright @ Rene Vidal 10
Number of Independent Motions n Theorem: Given image points corresponding to motions, if at least 8 points correspond to each object, then Minimum number of points Copyright @ Rene Vidal 1 2 8 35 99 225 3 4 11
Epipoles in the nullspace of F n Embedded epipoles are in the null space of the multibody fundamental matrix n n Copyright @ Rene Vidal Are all vectors in Null(F) embedded epipoles? Are embedded epipoles linearly independent? 12
Dimension of Null(F) n Lemma: embedded epipoles are linearly independent if the epipoles are distinct n Lemma: if repeated n Theorem: Copyright @ Rene Vidal 13
Estimation of fundamental matrices Multibody epipolar constraint Given rank condition Embedding Lifting n linear system F Embedding Theorem: Multibody structure from motion [Vidal et al. ] n n Copyright @ Rene Vidal Factorization of bilinear forms can be reduced to factorization of linear forms Estimation of fundamental matrices can be reduced to GPCA 15
3 D Motions: Multibody epipolar transfer Number of motions Lifting Multibody epipolar line Polynomial factorization Copyright @ Rene Vidal 16
3 D Motions: Multibody epipole Lifting n The multibody epipole is the solution of the linear system n n Copyright @ Rene Vidal Number of distinct epipoles Epipoles are obtained using polynomial factorization 17
3 D Motions: Fundamental matrices n Columns of n Polynomial factorization to compute them up to scale n Scales can be computed linearly Copyright @ Rene Vidal are epipolar lines 18
Multibody SFM Algorithm Image point Embedded image point Veronese map Multibody epipolar transfer Multibody epipolar line Polynomial Factorization Epipolar lines Linear system Multibody epipole Epipoles Fundamental matrix Copyright @ Rene Vidal Polynomial Factorization Linear system 19
Comparison of 1 and n bodies Copyright @ Rene Vidal 20
Segmentation of 3 -D fundamental matrices n Multibody epipolar constraint n Multibody fundamental matrix n n Epipolar lines: derivatives of at a correspondence Epipoles are derivatives of at epipolar lines Copyright @ Rene Vidal n Fundamental matrices 21
- Slides: 20