3 D Computational Vision CSc 83029 Optical Flow

Optical Flow & Motion § Finding the movement of scene objects from time-varying images.

CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Computing Time-to-Impact τ v l(t)=f L/D(t) L l(t) D 0 f D 0 -vt

Sudden change in viewing position/direction: Hard to compute motion field/optical flow. CSc 83029 –

Views from a sequence of spatially close viewpoints: Motion Field/ Optical Flow CSc 83029

Sub problems of Motion Analysis § Correspondence. § Reconstruction. § Segmentation. CSc 83029 –

Motion Field 2 -D vector field of velocities of image points, induced by relative

Special Case 1: Pure Translation 2 -D vector field of velocities of image points,

Pure Translation: Radial Motion Field p 0=(x 0, y 0) CSc 83029 – 3

Pure Translation: Radial Motion Field p 0=(x 0, y 0) 1. 2. 3. 4.

Special Case 2: Moving Plane n P Plane moves: n, d are functions of

The notion of Optical Flow: Estimation of the motion field from a sequence of

Optical Flow CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Optical Flow (x+uδt, y+vδt) u=δx/δt v=δy/δt (x, y) t t+δt Image brightness constancy equation:

The Aperture Problem Aperture problem The component of the motion field in the direction

Optical Flow At each point we know d. E/dx d. E/dy and d. E/dt.

Computing Optical Flow Assumption: The motion field is well approximated by a constant vector

Computing Optical Flow global approach [Schunck and Horn 81] Minimize the error in the

Optical Flow If the scene is planar the motion Is described by Using Solve

Tracking Rigid Bodies B B Random Sampling Algorithm Step 1: Find corners Step 2:

Structure and Motion Recovery from Video 1. Use multiple image stream to compute the

Structure and Motion Recovery from Video Computed model 3 D coordinates of the feature

Factorization Method ji ki ii FRAMES: i=1…N CSc 83029 – 3 -D Computer Vision/

Factorization Method Pj ji ki (xij, yij) ii World Points: j=1…n CSc 83029 –

Factorization Method Pj Z ji ki (xij, yij) Ti X Y ii World Points:

ASSUMPTIONS: The camera model is orthographic! The positions of n image points have been

Measurement Matrix 2*N (Frames) n points per frame Registered Measurement Matrix 2*N (Frames) n

Factorization Method World Points: j=1…n Pj 3 D Centroid Z ji ki Ti X

Rank Theorem 2*N (Frames) n points per frame R: “Rotation Matrix” S: Shape Matrix

The algorithm 2*N (Frames) n points per frame Decompose into R and S. Is

Slides: 48

Download presentation

3 -D Computational Vision CSc 83029 Optical Flow & Motion The Factorization Method CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Optical Flow & Motion § Finding the movement of scene objects from time-varying images. § Motion Field § Optical Flow § Computing Optical Flow CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Computing Time-to-Impact τ v l(t)=f L/D(t) L l(t) D 0 f D 0 -vt CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Computing Time-to-Impact τ v l(t)=f L/D(t) L l(t) D 0 f D 0 -vt l(t) / l’(t) = τ Quantities measured from image sequence. CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Sudden change in viewing position/direction: Hard to compute motion field/optical flow. CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Views from a sequence of spatially close viewpoints: Motion Field/ Optical Flow CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Sub problems of Motion Analysis § Correspondence. § Reconstruction. § Segmentation. CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Motion Field 2 -D vector field of velocities of image points, induced by relative motion between the viewing camera and the observed points. Image plane f ri v dt p Scene Point Velocity V=dro/dt Image Velocity v =dri/dt CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos V dt ro P

Motion Field 2 -D vector field of velocities of image points, induced by relative motion between the viewing camera and the observed points. Image plane f ri v dt p V dt Perspective projection ro P Velocity of point P as a function of translation and rotation Motion field equations CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Motion Field 2 -D vector field of velocities of image points, induced by relative motion between the viewing camera and the observed points => SUM of 2 COMPONENTS: CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Motion Field 2 -D vector field of velocities of image points, induced by relative motion between the viewing camera and the observed points => SUM of 2 COMPONENTS: + CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Special Case 1: Pure Translation 2 -D vector field of velocities of image points, induced by relative motion between the viewing camera and the observed points => SUM of 2 COMPONENTS: + CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Pure Translation: Radial Motion Field p 0=(x 0, y 0) CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Pure Translation: Radial Motion Field p 0=(x 0, y 0) 1. 2. 3. 4. Tz < 0 : FOCUS OF EXPANSION. Tz > 0: FOCUS OF CONTRACTION. Tz = 0: PARALLEL MOTION FIELD. Vanishing point (epipole) p 0. CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Special Case 2: Moving Plane n P Plane moves: n, d are functions of time. Motion field=? CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Special Case 2: Moving Plane n P Plane moves: n, d are functions of time. Motion field=? Motion field: quadratic polynomial in (x, y, f) at any time t. The same motion field can be produced by 2 different planes undergoing 2 different 3 -D motions. CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

The notion of Optical Flow: Estimation of the motion field from a sequence of images. CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Optical Flow CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Optical Flow (x+uδt, y+vδt) u=δx/δt v=δy/δt (x, y) t t+δt Image brightness constancy equation: E(x, y, t)=E(x+uδt, y+vδt, t+δt) or CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

The Aperture Problem Aperture problem The component of the motion field in the direction orthogonal to the spatial image gradient is not constrained by the image brightness constancy equation. Image brightness constancy equation: E(x, y, t)=E(x+uδt, y+vδt, t+δt) 1 constraint 2 unknowns CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Optical Flow At each point we know d. E/dx d. E/dy and d. E/dt. How can we obtain dx/dt and dy/dt? CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Computing Optical Flow Assumption: The motion field is well approximated by a constant vector field within any small patch of the image plane. Each provides one constraint Solution is CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Computing Optical Flow Assumption: The motion field is well approximated by a constant vector field within any small patch of the image plane. Q Minimize CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Computing Optical Flow Assumption: The motion field is well approximated by a constant vector field within any small patch of the image plane. Q Minimize =>Solve the linear system CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Computing Optical Flow global approach [Schunck and Horn 81] Minimize the error in the image brightness constancy constraint. CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Computing Optical Flow global approach [Schunck and Horn 81] Minimize the error in the image brightness constancy constraint. Minimize the deviation from smoothness of the motion vectors. CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Optical Flow CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Optical Flow If the scene is planar the motion Is described by Using Solve for the 8 unknowns a, b, c, d, e, f, g and h. CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Tracking Rigid Bodies B B Random Sampling Algorithm Step 1: Find corners Step 2: Search for correspondence Step 3: Randomly choose small set of matches. Step 4: Estimate F matrix Step 5: Find total number of matches close to epipolar lines Step 6: Go to step 3 Step 7: Choose F with largest number of matches CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Structure and Motion Recovery from Video 1. Use multiple image stream to compute the information about camera motion and 3 D structure of the scene 2. Tracking image features over time Original sequence From Jana Kosecka Tracked Features CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Structure and Motion Recovery from Video Computed model 3 D coordinates of the feature points From Jana Kosecka Original picture CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Factorization Method ji ki ii FRAMES: i=1…N CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Factorization Method Pj ji ki (xij, yij) ii World Points: j=1…n CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos FRAMES: i=1…N

Factorization Method Pj Z ji ki (xij, yij) Ti X Y ii World Points: j=1…n CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos World Reference Frame FRAMES: i=1…N

ASSUMPTIONS: The camera model is orthographic! The positions of n image points have been tracked. Pj Z ji ki (xij, yij) Ti X Y ii World Points: j=1…n CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos World Reference Frame FRAMES: i=1…N

Measurement Matrix 2*N (Frames) n points per frame Registered Measurement Matrix 2*N (Frames) n points per frame CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Factorization Method World Points: j=1…n Pj 3 D Centroid Z ji ki Ti X ii FRAMES: i=1…N Y 2 D Centroid CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos World Reference Frame

Rank Theorem 2*N (Frames) n points per frame R: “Rotation Matrix” S: Shape Matrix CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

Rank Theorem 2*N (Frames) n points per frame R: “Rotation Matrix” S: Shape Matrix Rank of is 3. CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

The algorithm 2*N (Frames) n points per frame Decompose into R and S. Is the decomposition unique? Translation estimation? CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos

CSc 83029 – 3 -D Computer Vision/ Ioannis Stamos