Binocular Stereo 1 Topics 1 Principle 2 binocular
Binocular Stereo #1
Topics 1. Principle 2. binocular stereo basic equation 3. epipolar line 4. features and strategies for matching
Binocular stereo single image is ambiguous A a’ a” another image taken from a different direction gives the unique 3 D point
Epipolar line constraints Epipolar line One image point Possible line of sight Base line Epipolar plane Corresponding points lie on the Epipolar lines Epipolar line constratints
Epipolar geometry (multiple points) C 1 e 2 Epipoles: • intersections of baseline with image planes • projection of the optical center in another image • the vanishing points of camera motion direction C 2
Examples of epipolar geometry
Examples of epipolar geometry
Examples of epipolar geometry
Characteristics of epipolar line • rectification
Basic binocular stereo equation A physical point left image point right image plane left image center focal length right image center z World coordinate system base line length
Camera Model Pinhole camera
Camera Model geometry Y Perspective projection X (x, y) View point (Optical center) y x (X, Y, Z) (s. X, s. Y, s. Z) f : focal length -Z Image plane
Basic binocular stereo equation d+x d-x -z x’ x” d d z=-2 df/(x”-x’) x”-x’: disparity 2 d : base line length z f
Classic algorithms for binocular Stereo Marr-Poggio-Grimson Nishihara-Poggio Lucas-Kanade Ohta-Kanade Matthie-Kanade Okutomi-Kanade Baker Hannah Moravec Barnard-Thompson MIT group CMU group Stanford group
Features for matching a. brightness b. edges c. edge intervals d. interest points 10 11 12 11 15 16
Strategies for matching a. relaxation 10 10 5 10 10 10 10 10 b. coarse to fine c. dynamic programming global optimam local optimam
Main purpose of development Marr-Poggio-Grimson Nishihara-Poggio simulate human stereo Lucas-Kanade Ohta-Kanade Matthie-Kanade Okutomi-Kanade map making Baker Hannah Moravec map making navigation Barnard-Thompson navigation
Features for matching Marr-Poggio-Grimson Nishihara-Poggio points(random dots) edges intervals Lucas-Kanade Ohta-Kanade Matthie-Kanade Okutomi-Kanade brightness(gradient) intervals brightness Baker Hannah Moravec edges interest points Barnard-Thompson interest points
Strategies for matching Marr-Poggio-Grimson Nishihara-Poggio relaxation coarse to fine Lucas-Kanade Ohta-Kanade Matthie-Kanade Okutomi-Kanade relaxation dynamic programming Relaxation (Kalman filter) relaxation Baker Hannah Moravec dynamic programming coarse to fine Barnard-Thompson relaxation
Summary 1. binocular stereo takes two images of 3 D point from two different positions and determines its 3 D coordinate system. 2. Epipolar line 2 D matching ↓ 1 D matching 3. Features for matching ---brightness, edge interval, interest point 4. Strategies for matching ---relaxation, coarse to fine, dynamic programming 5. Read B&B pp. 88 -93 Horn pp. 299 -303
Binocular Stereo #2
Topics case study area-based stereo Marr-poggio stereo simulate human visual system Ohta-Kanade stereo aerial image analysis Moravec stereo navigation
Classification of stereo method 1. Features for matching a. brightness value b. point c. edge d. region 2. Strategies for matching a. brute-force (not a strategy ? ? ? ) b. coarse-to-fine c. relaxation d. dynamic programming 3. Constraints for matching a. epipolar lines b. disparity limit c. continuity d. uniqueness
Area-based stereo 1. method b c 2. problem a. trade-off of window size and resolution b. dull peak b c
Area-based stereo 1. Features for matching a. brightness value b. point c. edge d. region 2. Strategies for matching a. brute-force (not a strategy ? ? ? ) b. coarse-to-fine c. relaxation d. dynamic programming 3. Constraints for matching a. epipolar lines b. disparity limit c. continuity d. uniqueness
Marr-Poggio Stereo(`76) Simulating human visual system (random dot stereo gram) Marr, Poggio “Coopertive computation of stereo disparity” Science 194, 283 -287
Input : random dot stereo left image random dot shift the catch pat right image we can see the height different between the central and peripheral area
Constraints – Epipolar line constraint – Uniqueness constraint » each point in a image has only one depth value O. K. No. – Continuity constraint » each point is almost sure to have a depth value near the values of neighbors O. K. No.
D E F A A B C B D C E F Uniqueness constraint prohibits two or more matching points on one horizontal or vertical lines (E-A) A (E-B) B prohibit C (E-C) continuity constraint attracts more matching on a diagonal line (D-A) attract (E-B) (F-C) Same depth
relaxation 10 10 5 10 10 10 10 10 n+1
Marr-Poggio Stereo(`76) 1. Features for matching a. brightness value b. point c. edge d. region 2. Strategies for matching a. brute-force (not a strategy ? ? ? ) b. coarse-to-fine c. relaxation d. dynamic programming 3. Constraints for matching a. epipolar lines b. disparity limit c. continuity d. uniqueness simulate the human visual system (MIT)
Ohta-Kanade Stereo(`85) Map making Ohta, Kanade “Stereo by intra- and inter-scanline search using dynamic programming” , IEEE Trans. , Vol. PAMI-7, No. 2, pp. 139 -14
now matching become 1 D to 1 D L 1 L 2 L 3 L 4 L 5 L 6 R 1 R 2 R 3 R 4 R 5 R 6 L disparity R yet, N line * ML * MR (512 * 100 * 10 m sec = 15 hours)
Path Search u Matching problem can be considered as a path search problem u define a cost at each candidate of path segment based some ad-hoc function 10 100
Dynamic programming We can formalize the path finding problem as the following iterative formula optimum cost to K 3 0 2 1 cost between M and K Optimum costs are known
stereo pair edges
path disparity depth
stereo pair edges depth
Ohta-Kanade Stereo(`85) 1. Features for matching a. brightness value b. point Brightness of interval c. edge d. region 2. Strategies for matching a. brute-force (not a strategy ? ? ? ) b. coarse-to-fine c. relaxation d. dynamic programming 3. Constraints for matching a. epipolar lines b. disparity limit c. continuity d. uniqueness aerial image analysis (CMU)
Moravec Stereo(`79) navigation Moravec “Visual mapping by a robot rover” Proc 6 th IJCAI, pp. 598 -600 (1979)
Moravec’s cart Slide stereo Motion stereo
Slider stereo (9 eyes stereo) u = 36 stereo pairs!!! each stereo has an uncertainty measure uncertainty = 1 / base-line u each stereo has a confidence measure u u 9 C 2 long base line large uncertainty
Coarse to fine matching expand matching
area: confidence measure σ estimated distance σ: uncertainty measure 9 C 2 = 36 curves Interest point
Moravec Stereo(`81) 1. Features for matching a. brightness value b. point c. edge interest point d. region 2. Strategies for matching a. brute-force (not a strategy ? ? ? ) b. coarse-to-fine c. relaxation d. dynamic programming 3. Constraints for matching a. epipolar lines b. disparity limit c. continuity d. uniqueness navigation (Stanford)
Summary 1. Two images from two different positions give depth information 2. Epipolar line and plane 3. Basic equation Z=-2 df/(x”-x’) x”-x’: disparity 2 d : base line length 4. case study area-based stereo Marr-poggio stereo simulate human visual system Ohta-Kanade stereo aerial image analysis Moravec stereo navigation 5. Read Horn pp. 299 -303
F matrix
Camera Model Pinhole camera
Camera Model geometry Y Perspective projection X (x, y) View point (Optical center) y x (X, Y, Z) (s. X, s. Y, s. Z) f : focal length -Z Image plane
Camera Model Perspective projection formularization Projection matrix Perspective projection Affine projection (Non-linear) (Linear)
Affine Camera Models General formularization • Orthographic • Perspective • Affine camera
Affine Cameras perspective orthographic Focal length Distance from camera
Intrinsic parameters CCD elements CCD : an actual picture Not equal ! Image plane : an ideal image
Intrinsic parameters An ideal image on the Image plane y v (x, y) v 0 x θ An actual picture u 0 u (u, v)
Intrinsic parameters e. g. perspective projection Intrinsic matrix Projection matrix (normalized)
Extrinsic parameters Z Y X
Extrinsic parameters Z Y X
Extrinsic parameters
Extrinsic parameters R : rotation matrix t : translation vector
Summary (intrinsic & extrinsic parameters) (X, Y, Z) Z picture (u, v) Y R, t World coordinate Camera coordinate X World coordinate
Summary (intrinsic & extrinsic parameters) (X, Y, Z) Z picture (u, v) Y R, t World coordinate X 3 × 4 matrix
Epipolar geometry Essential matrix : E C 2 C 1 R
Essential & Fundamental matrix Image planes (ideal) Image 1 Image 2 Pictures (actual) Fundamental matrix : F
F matrix picture 1 picture 2 (u, v, 1) (u’, v’, 1) F & (u, v) known Calculate the epipolar line
Computing F matrix (Linear solution)
Corner detector x Extract interest points in each images y Harris corner detector
Matching or
Computing F matrix (Linear solution) Suppose we found 8 pairs of corresponding points ·····
Computing F matrix (Singularity constraint) Epipolar pencil by linear solution (due to noise and error)
Computing F matrix (Singularity constraint) Singular value decomposition (SVD) Without noise, σ3 must be 0 modification
Computing F matrix (Singularity constraint)
Summary u Pinhole camera and Affine camera u Intrinsic and extrinsic camera parameter u Epipolar geometry u Fundamental matrix
- Slides: 74