EECS 274 Computer Vision Affine Structure from Motion

  • Slides: 38
Download presentation
EECS 274 Computer Vision Affine Structure from Motion

EECS 274 Computer Vision Affine Structure from Motion

Affine structure from motion • Structure from motion (SFM) • Elements of affine geometry

Affine structure from motion • Structure from motion (SFM) • Elements of affine geometry • Affine SFM from two views – Geometric approach – Affine epipolar geometry – Affine SFM from multiple views – From affine to Euclidean images • Reading: FP Chapter 12

Affine structure from motion Given a sequence of images • Find out feature points

Affine structure from motion Given a sequence of images • Find out feature points in 2 D images • Find out corresponding features • Find out their 3 D positions • Find out their affine motion

Affine Structure from Motion Reprinted with permission from “Affine Structure from Motion, ” by

Affine Structure from Motion Reprinted with permission from “Affine Structure from Motion, ” by J. J. (Koenderink and A. J. Van Doorn, Journal of the Optical Society of America A, 8: 377 -385 (1990). 1990 Optical Society of America. Given m pictures of n points, can we recover • the three-dimensional configuration of these points? • the camera configurations (projection matrices)? (structure) (motion)

Scene relief • When the scene relief is small (compared with the overall distance

Scene relief • When the scene relief is small (compared with the overall distance separating it from the observing camera), affine projection models can be used to approximate the imaging process

Orthographic Projection R is a scene reference point Parallel Projection consider the points off

Orthographic Projection R is a scene reference point Parallel Projection consider the points off optical axis viewing rays are parallel

Affine projection models Weak-Perspective Projection (generalizes orthographic projection) R is a scene reference point

Affine projection models Weak-Perspective Projection (generalizes orthographic projection) R is a scene reference point Paraperspective Projection (generalizes parallel projection) consider the distortions for points off the optical axis

Affine projection equations • Consider weak perspective projection and let zr denote the depth

Affine projection equations • Consider weak perspective projection and let zr denote the depth of a reference point R, then P P’ p p is the non-homogenous coordinate R 2 is a 2 × 3 matrix of the first 2 rows of R and t 2 is a vector formed by first 2 elements of t

Weak perspective projection • k and s denote the aspect ratio and skew of

Weak perspective projection • k and s denote the aspect ratio and skew of the camera • M is a 2 × 4 matrix defined by – 2 intrinsic parameters – 5 extrinsic parameters – 1 scene-dependent structure parameter zr See Chapter 2. 3 of FP

The Affine Structure-from-Motion Problem Given m images of n matched points Pj we can

The Affine Structure-from-Motion Problem Given m images of n matched points Pj we can write Here pij is 2 × 1 non-homogenous coordinate, and Mi = (Ai bi) Problem: estimate the m 2 × 4 affine projection matrices Mi and the n positions Pj from the mn correspondences pij 2 mn equations in 8 m+3 n unknowns Overconstrained problem, that can be solved using (non-linear) least squares!

The Affine Ambiguity of Affine SFM When the intrinsic and extrinsic parameters are unknown

The Affine Ambiguity of Affine SFM When the intrinsic and extrinsic parameters are unknown If M i and P are solutions, j So are M’i and P’ where j and Q is an affine transformation. C is a 3 × 3 non-singular matrix and d is in R 3

Affine Structure from Motion • Any solution of the affine structure from motion (sfm)

Affine Structure from Motion • Any solution of the affine structure from motion (sfm) can only be defined up to an affine transformation ambiguity • Taking into account the 12 parameters define general affine transformation, for 2 views (m=2), we need at least 4 point correspondences to determine the projection matrices and 3 D points 2 mn ≥ 8 m + 3 n - 12

With known intrinsic parameters • Exploit constraints of Mi = (Ai bi) (See Chapter

With known intrinsic parameters • Exploit constraints of Mi = (Ai bi) (See Chapter 2. 3) to eliminate ambiguity • First find affine shape • Use additional views and constraints to determine Euclidean structure

2 D planar transformations Preserve parallelism and ratio of distance between colinear points

2 D planar transformations Preserve parallelism and ratio of distance between colinear points

Affine Spaces: (Semi-Formal) Definition

Affine Spaces: (Semi-Formal) Definition

Example: R 2 as an Affine Space

Example: R 2 as an Affine Space

In General The notation is justified by the fact that choosing some origin O

In General The notation is justified by the fact that choosing some origin O in X allows us to identify the point P with the vector OP, i. e. u=OP , Φu(O)=P Warning: P+u and Q-P are defined independently of O!!

Barycentric Combinations • Can “add” a vector to a point and “subtract” two points

Barycentric Combinations • Can “add” a vector to a point and “subtract” two points • Can we add points? R=P+Q • But, when NO! we can define • Note by introducing an arbitrary origin O:

Affine Subspaces defined by y a point O and a vector subspace U Can

Affine Subspaces defined by y a point O and a vector subspace U Can be defined purely in terms of points m+1 points define a m-dimensional subspace

Affine Coordinates • Coordinate system for U: • Coordinate system for Y=O+U: • Affine

Affine Coordinates • Coordinate system for U: • Coordinate system for Y=O+U: • Affine coordinates: • Coordinate system for Y: • Barycentric coordinates: Affine coordinates of P in the basis formed by points Ai

Affine Transformations Bijections from X to Y that: • Bijections map m-dimensional subspaces of

Affine Transformations Bijections from X to Y that: • Bijections map m-dimensional subspaces of X onto m-dimensional • subspaces map linesofof. Y; X onto lines of Y; and preserve thesubspaces ratios of signed lengthssubspaces; of • • map parallel onto parallel and line segments. • preserve affine (or barycentric) coordinates. • The affine coordinates of D in the basis of A, B, C are the same as those of D’ in the basis of A’, B’, and C’ – namely 2/3 and ½. • In E 3 they are combinations of rigid transformations, non-uniform scalings and shears

Affine Transformations II • Given two affine spaces X and Y of dimension m,

Affine Transformations II • Given two affine spaces X and Y of dimension m, and two coordinate frames (A) and (B) for these spaces, there exists a unique affine transformation mapping (A) onto (B). • Given an affine transformation from X to Y, one can always write: • When coordinate frames have been chosen for X and Y, this translates into:

Affine projections induce affine transformations from planes onto their images. Preserve ratio of distance

Affine projections induce affine transformations from planes onto their images. Preserve ratio of distance between colinear points, parallelism, and affine coordinates Weak- and paraperspective projections are affine transformations

Affine Shape Two point sets S and S’ in some affine space X are

Affine Shape Two point sets S and S’ in some affine space X are affinely equivalent when there exists an affine transformation y: X X such that X’ = y ( X ). Affine structure from motion = affine shape recovery. = recovery of the corresponding motion equivalence classes.

Geometric affine scene reconstruction from two images (Koenderink and Van Doorn, 1991). • 4

Geometric affine scene reconstruction from two images (Koenderink and Van Doorn, 1991). • 4 points define 2 affine views • Affine projection of a plane onto • another plane is an affine transformation • Affine coordinates in π can be measured by other two images affine coordinates of P in the basis (A, B, C, D)

Affine Structure from Motion Reprinted with permission from “Affine Structure from Motion, ” by

Affine Structure from Motion Reprinted with permission from “Affine Structure from Motion, ” by J. J. (Koenderink and A. J. Van Doorn, Journal of the Optical Society of America A, 8: 377 -385 (1990). 1990 Optical Society of America. (Koenderink and Van Doorn, 1991) Given 2 affine views of 4 non-coplanar ponits, the affine shape of the scene is uniquely determined

Algebraic motion estimation using affine epipolar constraint A, A’, b, b’ are known α,

Algebraic motion estimation using affine epipolar constraint A, A’, b, b’ are known α, β, α’, β’ are constants of A, A’, b, b’ Note: the epipolar lines are parallel.

Affine Epipolar Geometry Given point p=(u, v)T, the matching point p’=(u’, ’v)T lies on

Affine Epipolar Geometry Given point p=(u, v)T, the matching point p’=(u’, ’v)T lies on α’u’+β’v’+(αu+βv+δ)=0

The Affine Fundamental Matrix where

The Affine Fundamental Matrix where

Algebraic Scene Reconstruction Method

Algebraic Scene Reconstruction Method

An Affine Trick. . Algebraic Scene Reconstruction Method

An Affine Trick. . Algebraic Scene Reconstruction Method

The Affine Structure of Affine Images Suppose we observe a scene with m fixed

The Affine Structure of Affine Images Suppose we observe a scene with m fixed cameras. . The set of all images of a fixed scene is a 3 D affine space!

has rank 4!

has rank 4!

From Affine to Vectorial Structure Idea: pick one of the points (or their center

From Affine to Vectorial Structure Idea: pick one of the points (or their center of mass) as the origin.

What if we could factorize D? (Tomasi and Kanade, 1992) Affine SFM is solved!

What if we could factorize D? (Tomasi and Kanade, 1992) Affine SFM is solved! Singular Value Decomposition We can take

From uncalibrated to calibrated cameras Weak-perspective camera: Calibrated camera: Problem: what is Q ?

From uncalibrated to calibrated cameras Weak-perspective camera: Calibrated camera: Problem: what is Q ? Note: Absolute scale cannot be recovered. The Euclidean shape (defined up to an arbitrary similitude) is recovered.

Reconstruction Results (Tomasi and Kanade, 1992) Reprinted from “Factoring Image Sequences into Shape and

Reconstruction Results (Tomasi and Kanade, 1992) Reprinted from “Factoring Image Sequences into Shape and Motion, ” by C. Tomasi and T. Kanade, Proc. IEEE Workshop on Visual Motion (1991). 1991 IEEE.

Photo tourism/photosynth

Photo tourism/photosynth