Segmentation of Dynamic Scenes from Image Intensities Ren

Segmentation of Dynamic Scenes from Image Intensities René Vidal Shankar Sastry Department of EECS, UC Berkeley

Motivation and problem statement n A static scene: multiple 2 D motion models n A dynamic scene: multiple 3 D motion models n Given 3 -D Motion an image Segmentation sequence, (Vidal-Soatto-Ma-Sastry, determine ECCV’ 02) n Number Generalization of motion of the models 8 -point (affine, algorithm Euclidean, etc. ) n n Motion Multibody model: fundamental affine (2 D) matrix or Euclidean (3 D) n Segmentation: model to. Segmentation which each pixel belongs This Talk: 2 -D Motion n n 2

Previous work n Probabilistic approaches (Jepson-Black’ 93, Ayer-Sawhney ’ 95, Darrel-Pentland’ 95, Weiss-Adelson’ 96, Weiss’ 97, Torr-Szeliski-Anandan ’ 99) n n Generative model: data membership + motion model Obtain motion models using Expectation Maximization n E-step: Given motion models, segment image data M-step: Given data segmentation, estimate motion models How to initialize iterative algorithms? n Spectral clustering: normalized cuts n n n Similarity matrix based on motion profile Local methods n (Wang-Adelson ’ 94) Estimate one model per pixel using a data in a window Global methods n n (Shi-Malik ‘ 98) (Irani-Peleg ‘ 92) Dominant motion: fit one motion model to all pixels Look for misaligned pixels & fit a new model to them 3

Our Approach to Motion Segmentation n n Can we estimate ALL motion models simultaneously using ALL the image measurements? When can we do so analytically? In closed form? Is there a formula for the number of models? We propose an algebraic geometric approach to affine motion segmentation n = degree of a polynomial ≈ roots of a polynomial = polynomial factorization In the absence of noise n n Number of models Groups Derive a constraint that is independent on the segmentation There exists a unique solution which is closed form iff n<5 The exact solution can be computed using linear algebra In the presence of noise n Derive a maximum likelihood algorithm for zero-mean Gaussian noise in which the E-step is algebraically eliminated 6

One-dimensional Segmentation n with a. ames Number of models? 7

One-dimensional Segmentation n For n groups n Number of groups n Groups with a. ames 8

Motion segmentation: the affine model n Constant brightness constraint n Affine motion model for the optical flow n Bilinear affine constraint n Mixture of n affine motion models 9

The multibody affine constraint n n n 1 -dimensional case n Affine segmentation case Multibody affine constraint Veronese map 10

The multibody affine matrix Multibody affine constraint Multibody affine matrix Embedding Lifting Embedding 11

Affine motion segmentation algorithm n 1 -dimensional case n Affine segmentation case Estimate all models: coefficients of a polynomial Estimate number models: rank of a matrix Estimate individual models: roots/factors of the polynomial 12

Estimation of individual affine models Can be reduced to scalar case!! Factorization of affine motion models n n Factorization of bilinear forms can be reduced to factorization of linear forms Factorization of linear forms corresponds to segmentation of mixtures of subspaces Generalized PCA: mixture of subspaces n Find roots of polynomial of degree n in one variable n Solve one linear systems in n variables 13

Optimal affine motion segmentation n n Zero-mean Gaussian noise Minimize distance error in image intensities subject to affine constraints n Using Langrange optimization n After some algebra multibody 14

Experimental results: flower sequence 17

Experimental results n Two motions n n Camera panning to the right Car translating to the right http: //www. cs. otago. ac. nz/research/vision/Research/ 18

Conclusions n There is an analytic solution to affine motion segmentation based on n n A similar technique also applies to n n Multibody affine constraint: segmentation independent Polynomial factorization: linear algebra Solution is closed form iff n<5 Eigenvector segmentation: from similarity matrices Generalized PCA: mixtures of subspaces 3 -D motion segmentation: of fundamental matrices Future work n Reduce data complexity, sensitivity analysis, robustness 19