The Brightness Constraint Brightness Constancy Equation Linearizing assuming

The Brightness Constraint Brightness Constancy Equation: Linearizing (assuming small (u, v)): Where: I t = I ( x, y) - J( x, y) Each pixel provides 1 equation in 2 unknowns (u, v). Insufficient info. Another constraint: Global Motion Model Constraint

Global Motion Models 2 D Models: • Affine • Quadratic • Planar projective transform (Homography) 3 D Models: • Rotation, Translation, 1/Depth • Instantaneous camera motion models • Plane+Parallax

Example: Affine Motion Substituting into the B. C. Equation: Each pixel provides 1 linear constraint in 6 global unknowns (minimum 6 pixels necessary) Least Square Minimization (over all pixels):

Coarse-to-Fine Estimation warp refine Jw pixels u=1. 25 + u=2. 5 pixels ==> small u and v. . . u=5 pixels image J Pyramid of image J u=10 pixels image I Pyramid of image I

Other 2 D Motion Models Quadratic – instantaneous approximation to planar motion Projective – exact planar motion (Homography)

Panoramic Mosaic Image Original video clip Alignment accuracy (between a pair of frames): error < 0. 1 pixel Generated Mosaic image

Video Removal Original Outliers Synthesized

Video Enhancement ORIGINAL ENHANCED

Direct Methods: Methods for motion and/or shape estimation, which recover the unknown parameters directly from measurable image quantities at each pixel in the image. Minimization step: Direct methods : Error measure based on dense measurable image quantities (Confidence-weighted regression; Exploits all available information) Feature-based methods: Error measure based on distances of a sparse set of distinct features.

Benefits of Direct Methods • Subpixel accuracy. • Does not need distinct features. • Locking property.

Limitations • Limited search range (up to 10% of the image size). • Brightness constancy.

Handling Varying Brightness Preprocessing: • Mean and contrast normalization. • Laplacian pyramids. Measurable image quantities: • brightness values • correlation surfaces [Irani-Anandan: iccv 98, Mandelbaum-et-al: iccv 99] • mutual information [Viola-et-al]

Video Indexing and Editing

The 2 D/3 D Dichotomy Camera motion Camera induced motion Image motion = = + Scene structure + Independent motions 2 D techniques Do not model “ 3 D scenes” = + Independent motions 3 D techniques Singularities in “ 2 D scenes”

The Plane+Parallax Decomposition Original Sequence The residual parallax lies on a radial (epipolar) field: epipole Plane-Stabilized Sequence

Benefits of the P+P Decomposition 1. Reduces the search space: • Eliminates effects of rotation • Eliminates changes in camera parameters / zoom • Camera parameters: Need to estimate only epipole. (gauge ambiguity: unknown scale of epipole) • Image displacements: Constrained to lie on radial lines (1 -D search problem) A result of aligning an existing structure in the image.

Benefits of the P+P Decomposition 2. Scene-Centered Representation: Translation or pure rotation ? ? ? Focus on relevant portion of info Remove global component which dilutes information !

Benefits of the P+P Decomposition 2. Scene-Centered Representation: Shape = Fluctuations relative to a planar surface in the scene STAB_RUG SEQ

Benefits of the P+P Decomposition 2. Scene-Centered Representation: Shape = Fluctuations relative to a planar surface in the scene • Height vs. Depth (e. g. , obstacle avoidance) • Appropriate units for shape - fewer bits, progressive encoding • A compact representation total distance [97. . 103] camera center scene global (100) component local [-3. . +3] component

Benefits of the P+P Decomposition 3. Stratified 2 D-3 D Representation: • Start with 2 D estimation (homography). • 3 D info builds on top of 2 D info. Avoids a-priori model selection.

Dense 3 D Reconstruction (Plane+Parallax) Original sequence Plane-aligned sequence Recovered shape

Dense 3 D Reconstruction (Plane+Parallax) Original sequence Plane-aligned sequence Recovered shape

Results Original sequence Recovered shape Plane-aligned sequence

Multi-Frame vs. 2 -Frame Estimation 1. Eliminating Aperture Problem Brig htne ss C onst ancy r ola ine l cons train t p ip p E epipole The intersection of the two line constraints uniquely defines the displacement.

Multi-Frame vs. 2 -Frame Estimation 1. Eliminating Aperture Problem cy n ta s n o c nt i a tr other epipolar line s another epipole h g i r B s s e tn n Co r ola ine l p ip p E epipole The other epipole resolves the ambiguity ! two line constraints are parallel ==> do NOT intersect

3 D Motion Models Instantaneous camera motion: Global parameters: Local Parameter: Residual Planar Parallax Motion Global parameters: Local Parameter: