Reconstruction of Multiple Motions using Generalized Principal Component

Reconstruction of Multiple Motions using Generalized Principal Component Analysis (GPCA) René Vidal Center for Imaging Science Department of Biomedical Engineering Johns Hopkins University

Structure and motion recovery n n Input: Corresponding points in multiple images Output: camera motion, scene structure, camera calibration Structure = 3 D surface Motion = camera position and orientation 2

Motion segmentation from 2 views n n A static scene: multiple 2 -D motion models n A dynamic scene: multiple 3 -D motion models Given an image sequence, determine n n n Number of motion models (affine, Euclidean, etc. ) Motion model: affine (2 -D) or Euclidean (3 -D) Segmentation: model to which each pixel belongs 3

Main challenges in motion segmentation n Mathematics of the problem depends on n n Number of frames (2, 3, multiple) Projection model (affine, perspective) Motion model (affine, translational, planar motion, rigid motion) 3 -D structure (planar or not) “Chicken-and-egg” problem n n If segmentation is known, apply standard SFM If motion is known, segmentation is easy 4

A unified approach to motion segmentation n Applies to most motion models in computer vision n All motion models can be segmented algebraically by n n Fitting multibody model: real or complex polynomial to all data Fitting individual model: differentiate polynomial at a data point 5

A unified approach to motion segmentation n Estimation of multiple motion models equivalent to estimation of one multibody motion model chicken-and-egg n Eliminate feature clustering: multiplication n Estimate a single multibody motion model: polynomial fitting n Segment multibody motion model: polynomial differentiation 6

Example: segmentation of optical flow n Scene having multiple optical flows n n n Brightness constancy constraint (BCC) gives Multiple BCCs Multibody brightness constancy constraint 7

How to segment motions in general? n One motion – one subspace: Component Analysis (PCA) Principal Basis for S n Multiple motions – multiple subspaces Generalized Principal Component Analysis (PCA) 8

Generalized Principal Component Analysis n Given points on multiple subspaces, identify n n “Chicken-and-egg” problem n n n Given segmentation, estimate subspaces Given subspaces, segment the data Prior work n n n The number of subspaces and their dimensions A basis for each subspace The segmentation of the data points Iterative algorithms: e. g. K-plane clustering (Bradley’ 00) Probabilistic approaches (Tipping-Bishop ‘ 99): learn the parameters of a mixture model using e. g. EM Initialization? 9

Generalized Principal Component Analysis n Clustering hyperplanes using Generalized PCA (GPCA) Veronese map n One can linearly fit the multibody motion model 10

Generalized Principal Component Analysis Segmenting multiple hyperplanes n Estimate multibody motion model: fit complex polynomial n Estimate motion models: the polynomial differentiate 11

Generalized Principal Component Analysis n n With moderate noise and outliers n Polynomials may not be a perfect union of subspaces n Normals can estimated correctly by choosing points optimally Distance to closest subspace without knowing segmentation? 12

Segmentation of 2 -D motion models n 2 -D translational motion model: n 2 -D similarity motion model: n 2 -D affine motion model: 13

Segmentation results: 2 -D translational 14

Segmentation of 3 -D translational motions n Multiple epipoles (translation) n Epipolar constraint: plane in n Plane normal = epipoles Data = epipolar lines Multibody epipolar constraint n Epipoles are derivatives of at epipolar lines 15

Segmentation results: 3 -D translational 16

Segmentation of 3 -D fundamental matrices n Multibody epipolar constraint n Multibody fundamental matrix n n Epipolar lines: derivatives of at a correspondence Epipoles are derivatives of at epipolar lines n Fundamental matrices 17

Segmentation results: 3 -D fundamental N = 44 + 48 + 81 = 173 18

Segmentation of 3 -D homographies n Homography model useful for n n Special motion: pure rotation Special structure: planar scene n Complex homography constraint n Can re-write equation in bilinear form as n Multibody homography constraint 19

Segmentation results: 3 -D homographies n n Two rigid motions: cube and plane Number of correspondences: 147 Number of homographies = 3 Number of groups depends on both the number of motions and the number of planes 20

Multiple affine views with missing data n n n Affine camera model Motion of 1 rigid-body lives in a subspace of dimension 4 Motion segmentation is equivalent to segmenting subspaces of dimension 2, 3, 4 in n n Project to 5 -D subspace: Power Factorization Estimate multiple subspaces in : GPCA 23

Segmentation results: 3 D multiframe affine n Sequence A Sequence B Sequence C n Percentage of correct classification 24

Mutiframe results 25

Mutiframe results 26

Omnidirectional Vision n n Motion Estimation and Segmentation Formation control of nonholonomic autonomous vehicles 27

Conclusions n Unified algebraic approach to motion segmentation n Fit a polynomial to all image data Differentiate the polynomial to obtain motion parameters Applies to most motion models in vision n Two views n n Three views n Multibody trifocal tensor Multiple views n n 2 -D: translational, similarity, affine 3 -D: translational, fundamental matrices, homographies Affine cameras Open problems n n n Multiple views perspective and central panoramic Dealing with noise (bilinear and trilinear models) Dealing with outliers (all models) 28

References n n n R. Vidal, Y. Ma, S. Soatto and S. Sastry. Two-view multibody structure from motion, International Journal of Computer Vision, 2004 R. Vidal and Y. Ma. A unified algebraic approach to 2 -D and 3 -D motion segmentation, European Conf. on Computer Vision 2004 R. Vidal and R. Hartley. segmentation using Power Factorization and GPCA, IEEE Motion Conference on Computer Vision and Pattern Recognition 2004 n n R. Vidal and S. Sastry. Optimal segmentation of dynamic scenes from two perspective views, International Conference on Computer Vision and Pattern Recognition, 2003 R. Vidal and S. Sastry. Segmentation of dynamic scenes from image intensities, IEEE Workshop on Vision and Motion Computing, 2002 29

Minimum number of correspondences 30