Computer and Robot Vision II Chapter 15 Motion

Computer and Robot Vision II Chapter 15 Motion and Surface Structure from Time Varying Image Sequences Presented by: 傅楸善 & 王林農 0917 533843 r 94922081@ntu. edu. tw 指導教授: 傅楸善 博士

15. 1 Introduction l Motion analysis involves estimating the relative motion of objects with respect to each other and the camera given two or more perspective projection images in a time sequence. DC & CV Lab. CSIE NTU

15. 1 Introduction (cont’) l Real-world applications: industrial automation and inspection, robot assembly, autonomous vehicle navigation, biomedical engineering, remote sensing, general 3 D-scene understanding DC & CV Lab. CSIE NTU

15. 1 Introduction (cont’) l object motion and surface structure recovery from: l l observed optic flow point correspondences DC & CV Lab. CSIE NTU

15. 2 The Fundamental Optic Flow Equation l l (x, y, z): 3 D point on moving rigid body (u, v): perspective projection on the image plane f: camera constant (u, v): velocity of the point (u, v) DC & CV Lab. CSIE NTU

15. 2 The Fundamental Optic Flow Equation (cont’) l take time derivatives of both sides l yields the fundamental optic flow equation: . . . DC & CV Lab. CSIE NTU

15. 2 The Fundamental Optic Flow Equation (cont’) l general solution: (λ is a free variable) . . . DC & CV Lab. CSIE NTU

15. 2. 1 Translational Motion l Known: l l Unknown: l l N-point optic flow field: . . corresponding unknown 3 D points: all points moving with same but unknown velocity (x, y, z) . . . can be solved up to a multiplicative constant DC & CV Lab. CSIE NTU

15. 2. 2 Focus of Expansion and Contraction l Known: l l l 3 D motion is translational one 2 D projected point (u, v) has no motion: . . thus translational motion is in a direction along the ray of sight DC & CV Lab. CSIE NTU

15. 2. 2 Focus of Expansion and Contraction (cont’) l l focus of expansion (FOE): if 3 D point field moving toward camera FOE: motion-field vectors radiate outward from that point focus of contraction (FOC): if 3 D point field moving away from camera FOC: vectors radiate inward toward diametrically opposite point flow pattern of the motion field of a forward-moving observer DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

15. 2. 3 Moving Line Segment l l Known: fixed distance between two unknown 3 D points translational motion with common velocity. . . (x, y, z) corresponding optic flow: . . DC & CV Lab. CSIE NTU

15. 2. 3 Moving Line Segment (cont’) l Unknown: l l : two unknown 3 D points . . . common velocity: (x, y, z) DC & CV Lab. CSIE NTU

15. 2. 4 Optic Flow Acceleration Invariant. l l Since differentiating general solution in Sec 15. 2 and. . . solve for (x, y, z) . . . DC & CV Lab. CSIE NTU . .

15. 3 Rigid-Body Motion l l l Rigid-body motion: no relative motion of points w. r. t. one another Rigid-body motion: points maintain fixed position relative to one another Rigid-body motion: all points move with the body as a whole DC & CV Lab. CSIE NTU

15. 3 Rigid-Body Motion (cont’) l l l R(t): rotation matrix T(t): translation vector p(0): initial position of given point R(0)=I, T(0)=0 p(t): position of given point at time t DC & CV Lab. CSIE NTU

15. 3 Rigid-Body Motion (cont’) l Rigid-body motion in displacement vectors: l velocity vector: time derivative of its position: . . . DC & CV Lab. CSIE NTU

15. 3 Rigid-Body Motion (cont’) l Since l (a) translational-motion field under projection onto hemispherical surface only translationalcomponent motion useful in determining scene structure (b) rotational-motion field under projection onto hemispherical surface rotational-motion field provides no information about scene structure l . . DC & CV Lab. CSIE NTU .

DC & CV Lab. CSIE NTU

15. 3 Rigid-Body Motion (cont’) l we can describe rigid-body motion in instantaneous velocity by . DC & CV Lab. CSIE NTU

15. 3 Rigid-Body Motion (cont’) l l l : angular velocities in three axes : translational velocities in three axes from rigid-body-motion equation . . . DC & CV Lab. CSIE NTU

15. 3 Rigid-Body Motion (cont’) l and perspective projection equation l . we can determine an expression for z: . DC & CV Lab. CSIE NTU

15. 3 Rigid-Body Motion (cont’) l after simplification . . DC & CV Lab. CSIE NTU

15. 3 Rigid-Body Motion (cont’) l l l image velocity: expressed as sum of translational field and rotational field (x, y, z): 3 D coordinate before rigid-body motion in displacement vectors (x’, y’, z’): 3 D coordinate after rigid-body motion in displacement vectors : rotation angles in three axes : translation in three axes DC & CV Lab. CSIE NTU

15. 3 Rigid-Body Motion (cont’) l Rigid-body motion in displacement vectors: DC & CV Lab. CSIE NTU

15. 3 Rigid-Body Motion (cont’) l l motion in displacement vector and instantaneous velocity is different: e. g. moon encircling earth instantaneous velocity: first order approximation of displacement vector first order approximation: when small, DC & CV Lab. CSIE NTU

15. 3 Rigid-Body Motion (cont’) l l l first order approximation: when time=1 thus. x=(x’ - x)/1 first order approximation: DC & CV Lab. CSIE NTU

joke DC & CV Lab. CSIE NTU

15. 4 Linear Algorithms for Motion and Surface Structure from Optic Flow l l l 15. 4. 1 The Planar Patch Case : arbitrary object point on planar patch at time t : central projective coordinates of p(t) onto image plane z= f DC & CV Lab. CSIE NTU

15. 4. 1 The Planar Patch Case l l . . : instantaneous velocity of moving image point. . : optic flow image point : instantaneous rotational angular velocity : instantaneous translational velocity DC & CV Lab. CSIE NTU

15. 4. 1 The Planar Patch Case (cont’) l l unit vector n(t): orthogonal to moving planar patch rigid planar patch motion represented by rigidmotion constraint: . DC & CV Lab. CSIE NTU

15. 4. 1 The Planar Patch Case (cont’) l from above two equations: l Let l Rigid-motion constraint could be written as DC & CV Lab. CSIE NTU

15. 4. 1 The Planar Patch Case (cont’) l l l denote the 3 x 3 matrix by W and its three row vectors by W: called planar motion parameter matrix since skew symmetric DC & CV Lab. CSIE NTU

15. 4. 1 The Planar Patch Case (cont’) l above equation. can be written as . . l from perspective projection equations: l taking time derivatives of these equations we. . . have . . . DC & CV Lab. CSIE NTU

15. 4. 1 The Planar Patch Case (cont’) l substitute equations into above equations: l from third row l . . . substitute equation . . z to obtain optical flow-planar motion. . DC & CV Lab. CSIE NTU

15. 4. 1 The Planar Patch Case (cont’) l we have 2 N linear equations: n=1, …, N: . . . l optic flow-planar motion recovery: first solve W then find DC & CV Lab. CSIE NTU

15. 4. 2 General Case Optic Flow. Motion Equation l l 1. set up optic flow-motion equation not involving depth information 2. solve it by using linear least-squares technique DC & CV Lab. CSIE NTU

15. 4. 3 A Linear Algorithm for Solving Optic Flow. Motion Equations DC & CV Lab. CSIE NTU

15. 5. 4 Mode of Motion, Direction of Translation, and Surface Structure l l l mode of motion: whether translation k=0 or not direction of translation: direction of k surface structure: relative depth when k 0 DC & CV Lab. CSIE NTU

15. 4. 5 Linear Optic Flow-Motion Algorithm and Simulation Results l l motion and shape recovery algorithms should answer three questions: minimum number of points to compute motion and shape what set of optic flow points violate rank assumption e. g. collinearity… What’s the accuracy of estimated motion from noisy optic flow? DC & CV Lab. CSIE NTU

joke DC & CV Lab. CSIE NTU

15. 5 The Two View-Linear Motion Algorithm DC & CV Lab. CSIE NTU

15. 5. 1 Planar Patch Motion Recovery from Two Perspective Views: A Brief Review l l l Two View-Planar Motion Equation imaging geometry for two view-planar motion rigid planar patch in motion in half-space z< 0 DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

15. 5. 1 Planar Patch Motion Recovery from Two Perspective Views: A Brief Review (cont’) l l : arbitrary object point before motion : same object point after motion : central projective coordinates of f : camera constant l DC & CV Lab. CSIE NTU

15. 5. 1 Planar Patch Motion Recovery from Two Perspective Views: A Brief Review (cont’) l l l R 0: 3 X 3 rotational matrix, R 0’R 0=I, |R 0|=1 t 0: 3 X 1 translational vector n 0: 3 X 1 normal vector DC & CV Lab. CSIE NTU

15. 5. 1 Planar Patch Motion Recovery from Two Perspective Views: A Brief Review (cont’) l Rigid-body-motion equation relates p 1 to p 2 as follows: l planarity constrains p 1 by l combining two equations produces planar rigidbody-motion-equation DC & CV Lab. CSIE NTU

15. 5. 2 General Curved Patch Motion Recovery from Two Perspective Views A Simplified Linear Algorithm l discard planar patch assumption, consider general curved patch DC & CV Lab. CSIE NTU

15. 5. 3 Determining Translational Orientation DC & CV Lab. CSIE NTU

15. 5. 4 Determining Mode of Motion and Relative Depths DC & CV Lab. CSIE NTU

15. 5. 5 A Simplified Two View. Motion Linear Algorithm DC & CV Lab. CSIE NTU

15. 5. 6 Discussion and Summary l l when no noise appears: algorithm extremely accurate when small noise appears: it works well except mode of motion incorrect DC & CV Lab. CSIE NTU

15. 6 Linear Algorithm for Motion and Structure from Three Orthographic Views l Ullman (1979) showed that for the orthographic case four-point correspondences over three views are sufficient to determine the motion and structure of the four-point rigid configuration DC & CV Lab. CSIE NTU

l l Shimon Ullman, The Interpretation of Visual Motion The MIT Press Cambridge MA. 1979 DC & CV Lab. CSIE NTU

15. 6 Linear Algorithm for Motion and Structure from Three Orthographic Views l l to infer depth information: translation needed in perspective projection to infer depth information: rotation useless in perspective projection to infer depth information: rotation needed in orthographic projection to infer depth information translation useless in orthographic projection DC & CV Lab. CSIE NTU

15. 6. 1 Problem Formulation l l l l image plane stationary three orthographic views at time (x, y, z): object-space coordinates of point P at t 1 (x’, y’, z’): object-space coordinates of point P at t 2 (x”, y”, z”): object-space coordinates of point P at t 3 (u, v): image-space coordinates of P at t 1 (u’, v’): image-space coordinates of P at t 2 (u”, v”): image-space coordinates of P at t 3 DC & CV Lab. CSIE NTU

15. 6. 1 Problem Formulation (cont’) l l : rotation matrix : translation vector (x’, y’, z’)’ = R(x’, y’, z’)+Tr (x”, y”, z”)” = S(x”, y”, z”)+Ts DC & CV Lab. CSIE NTU

15. 6. 1 Problem Formulation (cont’) l Known: four image-point correspondences l Unkown: l DC & CV Lab. CSIE NTU

15. 6. 1 Problem Formulation (cont’) l note that with orthographic projections l therefore it is obvious that tr 3, ts 3 can never be determined we are trying to determine: l DC & CV Lab. CSIE NTU

15. 6. 2 Determining DC & CV Lab. CSIE NTU

15. 6. 3 Solving a Unique Orthonormal Matrix R DC & CV Lab. CSIE NTU

15. 6. 4 Linear Algorithm to Uniquely Solve R, s, a 3 DC & CV Lab. CSIE NTU

15. 6. 5 Summary l Given two orthographic views, one cannot finitely determine the motion and structure of a rigid body, no matter how many point correspondences are used, as shown by Huang. DC & CV Lab. CSIE NTU

15. 7 Developing a Highly Robust Estimator for General Regression DC & CV Lab. CSIE NTU

15. 7. 1 Inability of the Classical Robust MEstimator to Render High Robustness l l l Classical robust estimator, such as M-, L-, or Restimator: 1. optimal or nearly optimal at assumed noise distribution 2. relatively small performance degradation with small number of outliers 3. larger deviations from assumed distribution do not cause catastrophe MF-estimator with new property much stronger than property 3 relatively small performance degradation with larger deviations from assumed distribution DC & CV Lab. CSIE NTU

15. 7. 2 Partially Modeling Log Likelihood Function by Using Heuristics l l MF-estimator: combine Bayes statistical decision rule with heuristics MF-estimator: robust regression more appropriate model-fitting DC & CV Lab. CSIE NTU

15. 7. 3 Discussion l M-, L-, R and MF-estimator: all residual based DC & CV Lab. CSIE NTU

15. 7. 4 MF-Estimator DC & CV Lab. CSIE NTU

15. 8 Optic Flow-Instantaneous Rigid. Motion Segmentation and Estimation l formulate optic flow-single rigid-motion estimation into general regression DC & CV Lab. CSIE NTU

15. 8. 1 Single Rigid Motion DC & CV Lab. CSIE NTU

15. 8. 2 Multiple Rigid Motions DC & CV Lab. CSIE NTU

l joke DC & CV Lab. CSIE NTU

15. 9 Experimental Protocol DC & CV Lab. CSIE NTU

15. 9. 1 Simplest Location Estimation DC & CV Lab. CSIE NTU

15. 9. 2 Optic Flow-Rigid-Motion Segmentation and Estimation DC & CV Lab. CSIE NTU

15. 10 Motion and Surface Structure from Line Correspondences DC & CV Lab. CSIE NTU

15. 10. 1 Problem Formulation l Cartesian reference system-central projection DC & CV Lab. CSIE NTU

15. 10. 1 Problem Formulation (cont’) DC & CV Lab. CSIE NTU

15. 10. 1 Problem Formulation (cont’) l l l l l: line in D space L: projection of the line on image plane z = f : image frame : known plane line L is in; projective plane of l : set of lines in 3 D space : lines moved by rigid motion (R’ , T’)’ at time t’ : lines moved by rigid motion (R” , T”)” at time t” DC & CV Lab. CSIE NTU

15. 10. 1 Problem Formulation (cont’) : projections of lines l ; l respective projective planes DC & CV Lab. CSIE NTU

15. 10. 1 Problem Formulation (cont’) l l l Known: K triples of line correspondences in three views Unkown: rotations and translations: 3 D lines DC & CV Lab. CSIE NTU

15. 10. 2 Solving Rotation Matrices R’, R” and Translations T’, R” DC & CV Lab. CSIE NTU

15. 10. 3 Solving Three-Dimensional Line Structure DC & CV Lab. CSIE NTU

15. 11 Multiple Rigid Motions from Two Perspective Views l l 15. 11. 1 Problem Statement imaging geometry for two-view-motion DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

15. 11. 1 Problem Statement l How many good point correspondences are needed in order to apply the nonlinear leastsquares estimator? DC & CV Lab. CSIE NTU

15. 11. 2 Simulated Experiments DC & CV Lab. CSIE NTU

15. 12 Rigid Motion from Three Orthographic Views DC & CV Lab. CSIE NTU

15. 12. 1 Problem Formulation and Algorithm l same as Sec. 15. 6, instead of linear algorithms, formulate model-fitting problem DC & CV Lab. CSIE NTU

15. 12. 2 Simulated Experiments DC & CV Lab. CSIE NTU

15. 12. 3 Further Research on the MFEstimator l l l two problems to be solved for MF-estimator to be practically useful: distance problem requirement for a good initial approximation DC & CV Lab. CSIE NTU

l l difficulty of motion and shape recovery: ambiguity of displacement field Fuh. Ph. D. Thesis, Fig 4. 1 DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

15. 13 Literature Review l 15. 13. 1 Inferring Motion and Surface Structure DC & CV Lab. CSIE NTU

15. 13. 1 Inferring Motion and Surface Structure l classifications for methods of inferring 3 D motion and shape l l l use of individual sets of feature points use of local optic flow information about a single point use of the entire optic flow field DC & CV Lab. CSIE NTU

15. 13. 1 Inferring Motion and Surface Structure l Despite all the results obtained over the years, almost none of these inference techniques have been successfully applied to feature-point correspondences calculated from real imagery DC & CV Lab. CSIE NTU

15. 13. 2 Computing Optic Flow or Image-Point Correspondences l problem source contains abundant information l l occlusion boundaries specular points near a focus of expansion noise and digitization effects in image formation DC & CV Lab. CSIE NTU

15. 13. 2 Computing Optic Flow or Image -Point Correspondences (cont’) l l motion parallax: apparent relative motion between objects and observer points in observer’s direction of translation remain relatively unchanged information available to a moving observer DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

15. 13. 2 Computing Optic Flow or Image -Point Correspondences (cont’) l l impart time dimension to image data spatiotemporal image data block DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

15. 13. 2 Computing Optic Flow or Image -Point Correspondences (cont’) l l motion field: assignment of vectors to image points representing motion angular velocity of fixed scene: inversely proportional to distance pilot in straight-ahead level flight on an overcast day DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

15. 13. 2 Computing Optic Flow or Image -Point Correspondences (cont’) l l motion field of pilot looking straight ahead in motion direction zero image velocity: at approach point and at infinity (along horizon) DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

15. 13. 2 Computing Optic Flow or Image -Point Correspondences (cont’) l l l motion field of pilot looking to the right in level flight focus of expansion here: at infinity to the left focus of contraction here: at infinity to the right of the figure DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

15. 13. 2 Computing Optic Flow or Image -Point Correspondences (cont’) l l spatiotemporal image data acquired by a camera, - caption straight streaks at block top due to translating parallel to image plane DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

B. K. P, Horn, Robot Vision, The MIT Press, Cambridge, MA, 1986 l l Chapter 12 Motion Field & Optical Flow optic flow: apparent motion of brightness patterns during relative motion DC & CV Lab. CSIE NTU

12. 1 Motion Field l l l motion field: assigns velocity vector to each point in the image Po: some point on the surface of an object Pi: corresponding point in the image vo: object point velocity relative to camera vi: motion in corresponding image point DC & CV Lab. CSIE NTU

12. 1 Motion Field (cont’) l ri : l l distance between perspectivity center and image point ro: distance between perspectivity center and object point f’: camera constant z: depth axis, optic axis object point displacement causes corresponding image point displacement DC & CV Lab. CSIE NTU

12. 1 Motion Field (cont’) DC & CV Lab. CSIE NTU

12. 1 Motion Field (cont’) l Velocities: l where ro and ri are related by DC & CV Lab. CSIE NTU

12. 1 Motion Field (cont’) l differentiation of this perspective projection equation yields DC & CV Lab. CSIE NTU

l joke DC & CV Lab. CSIE NTU

12. 2 Optical Flow l l l optical flow need not always correspond to the motion field (a) perfectly uniform sphere rotating under constant illumination: no optical flow, yet nonzero motion field (b) fixed sphere illuminated by moving light source: nonzero optical flow, yet zero motion field DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

12. 2 Optical Flow (cont’) l not easy to decide which P’ on contour C’ corresponds to P on C DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

12. 2 Optical Flow (cont’) l l l optical flow: not uniquely determined by local information in changing irradiance at time t at image point (x, y) components of optical flow vector DC & CV Lab. CSIE NTU

12. 2 Optical Flow (cont’) l assumption: irradiance the same at time l fact: motion field continuous almost everywhere DC & CV Lab. CSIE NTU

12. 2 Optical Flow (cont’) l expand above equation in Taylor series l e: second- and higher-order terms in cancelling E( x, y, t), dividing through by l DC & CV Lab. CSIE NTU

12. 2 Optical Flow (cont’) l which is actually just the expansion of the equation l abbreviations: DC & CV Lab. CSIE NTU

12. 2 Optical Flow (cont’) l we obtain optical flow constraint equation: l flow velocity (u, v): lies along straight line perpendicular to intensity gradient DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

12. 2 Optical Flow (cont’) l rewrite constraint equation: l aperture problem: cannot determine optical flow along isobrightness contour DC & CV Lab. CSIE NTU

12. 3 Smoothness of the Optical Flow l l motion field: usually varies smoothly in most parts of image try to minimize a measure of departure from smoothness DC & CV Lab. CSIE NTU

12. 3 Smoothness of the Optical Flow (cont’) l error in optical flow constraint equation should be small l overall, to minimize DC & CV Lab. CSIE NTU

12. 3 Smoothness of the Optical Flow (cont’) l l large if brightness measurements are accurate small if brightness measurements are noisy DC & CV Lab. CSIE NTU

12. 4 Filling in Optical Flow Information l l regions of uniform brightness: optical flow velocity cannot be found locally brightness corners: reliable information is available DC & CV Lab. CSIE NTU

12. 5 Boundary Conditions l l Well-posed problem: solution exists and is unique partial differential equation: infinite number of solution unless with boundary DC & CV Lab. CSIE NTU

12. 6 The Discrete Case l first partial derivatives of u, v: can be estimated using difference DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

12. 6 The Discrete Case (cont’) l measure of departure from smoothness: l error in optical flow constraint equation: l to seek set of values DC & CV Lab. CSIE NTU that minimize

12. 6 The Discrete Case (cont’) l dieffrentiating e with respect to DC & CV Lab. CSIE NTU

12. 6 The Discrete Case (cont’) l l where are local average of u, v (9 neighbors? ) extremum occurs where the above derivatives of e are zero: DC & CV Lab. CSIE NTU

12. 6 The Discrete Case (cont’) l determinant of 2 x 2 coefficient matrix: l so that DC & CV Lab. CSIE NTU

12. 6 The Discrete Case (cont’) l suggests iterative scheme such as l new value of (u, v): average of surrounding values minus adjustment DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

12. 6 The Discrete Case (cont’) l first derivatives estimated using first differences in 2 x 2 x 2 cube DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

12. 6 The Discrete Case (cont’) l consistent estimates of three first partial derivatives: DC & CV Lab. CSIE NTU

12. 6 The Discrete Case (cont’) l four successive synthetic images of rotating sphere DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

12. 6 The Discrete Case (cont’) l estimated optical flow after 1, 4, 16, and 64 iterations DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

12. 6 The Discrete Case (cont’) l l (a) estimated optical flow after several more iterations (b) computed motion field DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

12. 7 Discontinuities in Optical Flow l discontinuities in optical flow: on silhouettes where occlusion occurs DC & CV Lab. CSIE NTU

l Joke DC & CV Lab. CSIE NTU

Project due April 18 l l l implementing Horn & Schunck optical flow estimation as above synthetically translate lena. im one pixel to the right and downward Try DC & CV Lab. CSIE NTU