Stereo and Epipolar geometry Jana Kosecka CS 223

Stereo and Epipolar geometry Jana Kosecka CS 223 b Geyer, Daniilidis, UPenn ICRA 2003

Previously n n n Image Primitives (feature points, lines, contours) Today: How to match primitives between two (multiple) views) Goals: 3 D reconstruction, recognition Stereo matching and reconstruction (canonical configuration) Epipolar Geometry (general two view setting) CS 223 b 2

Why Stereo Vision? n 2 D images project 3 D points into 2 D: P Q P’=Q’ O n center of projection 3 D points on the same viewing line have the same 2 D image: n 2 D imaging results in depth information loss CS 223 b 3

Canonical Stereo Configuration n Assumes (two) cameras Known positions and focal lengths P Recover depth O 1 f O 2 T CS 223 b 4

n n Depth can be recovered by triangulation Prerequisite: Finding Correspondences Given an image point in left image, the (corresponding) point in the right image, is the point which is the projection of the same 3 -D point CS 223 b 5

Random dot stereograms B. Julesz: Showed that correspondence is not needed for stereo. CS 223 b 6

Autostereograms n Depth perception from one image n Viewing trick the brain by focusing at the plane behind - match can be established perception of 3 D CS 223 b 7

Correspondence Problem n Two classes of algorithms: n Correlation-based algorithms n n Produce a DENSE set of correspondences Feature-based algorithms n Produce a SPARSE set of correspondences CS 223 b 8

Stereo – Photometric Constraint • Same world point has same intensity in both images. • Lambertian fronto-parallel • Issues (noise, specularities, foreshortening) • Difficulties – ambiguities, large changes of appearance, due to change Of viewpoint, non-uniquess CS 223 b 9

Stereo Matching What if ? For each scanline , for each pixel in the left image • compare with every pixel on same epipolar line in right image • pick pixel with minimum match cost • This will never work, so: improvement match windows CS 223 b (slides O. Camps) 10

? = Comparing Windows: f g Most popular For each window, match to closest window on epipolar line in other image. CS 223 b (slides O. Camps) 11

Comparing Windows: Minimize Sum of Squared Differences Maximize Cross correlation It is closely related to the SSD: CS 223 b 12

Region based Similarity Metrics • Sum of squared differences • Normalize cross-correlation • Sum of absolute differences CS 223 b 13

NCC score for two widely separated views NCC score CS 223 b 14

Window size W=3 n W = 20 Effect of window size Better results with adaptive window • • T. Kanade and M. Okutomi, A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment, , Proc. International Conference on Robotics and Automation, 1991. D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2): 155 -174, July 1998 CS 223 b (S. Seitz) 15

Stereo results n Data from University of Tsukuba Scene Ground truth (Seitz) CS 223 b 16

Results with window correlation Ground truth Window-based matching (best window size) (Seitz) CS 223 b 17

Results with better method State of the art Ground truth Boykov et al. , Fast Approximate Energy Minimization via Graph Cuts, International Conference on Computer Vision, September 1999. CS 223 b (Seitz) 18

More of advanced stereo n n n Ordering constraint Dynamic programming Global optimization – next lecture CS 223 b 19

Two view – General Configuration • Motion between the two views is not known Given two views of the scene recover the unknown camera displacement and 3 D scene structure CS 223 b 20

Pinhole Camera Imaging Model • 3 D points • Image points • Perspective Projection • Rigid Body Motion + Persp. projection CS 223 b 21

Rigid Body Motion – Two Views CS 223 b 22

3 D Structure and Motion Recovery Euclidean transformation Measurements Corresponding points unknowns Find such Rotation and Translation and Depth that the reprojection error is minimized Two views ~ 200 points 6 unknowns – Motion 3 Rotation, 3 Translation - Structure 200 x 3 coordinates - (-) universal scale Difficult optimization problem CS 223 b 23

Notation n Cross product between two vectors in where CS 223 b 24

Epipolar Geometry Image correspondences • Algebraic Elimination of Depth • Essential matrix [Longuet-Higgins ’ 81]:

Epipolar Geometry • Epipolar lines • Epipoles Image correspondences Epipolar transfer • Additional constraints CS 223 b 26

Characterization of Essential Matrix Essential matrix special 3 x 3 matrix (Essential Matrix Characterization) A non-zero matrix is an essential matrix iff its SVD: satisfies: with and CS 223 b 27

Estimating Essential Matrix • Find such Rotation and Translation that the epipolar error is minimized • Space of all Essential Matrices is 5 dimensional • 3 DOF Rotation, 2 DOF – Translation (up to scale !) • Denote • Rewrite • Collect constraints from all points CS 223 b 28

Estimating Essential Matrix Solution • Eigenvector associated with the smallest eigenvalue of • If degenerate configuration estimated using linear least squares unstack -> Projection on to Essential Space (Project onto a space of Essential Matrices) If the SVD of a matrix is given by then the essential matrix which minimizes the Frobenius distance is given by with

Pose Recovery from Essential Matrix Essential matrix (Pose Recovery) There are two relative poses with and corresponding to a non-zero matrix essential matrix. • Twisted pair ambiguity CS 223 b 30

Two view linear algorithm - summary • Solve the LLSE problem: • Solution eigenvector associated with smallest eigenvalue of • Compute SVD of F recovered from data E is 5 diml. sub. mnfld. in • Project onto the essential manifold: • 8 -point linear algorithm • Recover the unknown pose: CS 223 b 31

Pose Recovery • There are two pairs corresponding to essential matrix . . • Positive depth constraint disambiguates the impossible solutions • Translation has to be non-zero • Points have to be in general position - degenerate configurations – planar points - quadratic surface • Linear 8 -point algorithm • Nonlinear 5 -point algorithms yields up to 10 solutions CS 223 b 32

3 D Structure Recovery unknowns • Eliminate one of the scale’s • Solve LLSE problem If the configuration is non-critical, the Euclidean structure of the points and motion of the camera can be reconstructed up to a universal scale. • Alternatively recover each point depth separately CS 223 b 33

Example Two views Point Feature Matching CS 223 b 34

Example Epipolar Geometry Camera Pose and Sparse Structure Recovery CS 223 b 35

Epipolar Geometry - Planar case • Plane in first camera coordinate frame Image correspondences Planar homography Linear mapping relating two corresponding planar points in two views

Decomposition of H (into motion and plane normal) n n Algebraic elimination of depth can be estimated linearly Normalization of Decomposition of H into 4 solutions n CS 223 b 37

Uncalibrated Camera calibrated coordinates Linear transformation pixel coordinates CS 223 b 38

Uncalibrated Epipolar Geometry • Epipolar constraint • Fundamental matrix CS 223 b 39

Properties of the Fundamental Matrix • Epipolar lines • Epipoles Image correspondences CS 223 b 40

Epipolar Geometry for Parallel Cameras f Ol T Or f el er P Epipoles are at infinite Epipolar lines are parallel to the baseline CS 223 b 41

Properties of the Fundamental Matrix A nonzero matrix is a fundamental matrix if a singular value decomposition (SVD) with for some has . There is little structure in the matrix CS 223 b except that 42

Estimating Fundamental Matrix • Find such F that the epipolar error is minimized Pixel coordinates • Fundamental matrix can be estimated up to scale • Denote • Rewrite • Collect constraints from all points CS 223 b 43

Two view linear algorithm – 8 -point algorithm • Solve the LLSE problem: • Solution eigenvector associated with smallest eigenvalue of • Compute SVD of F recovered from data • Project onto the essential manifold: • cannot be unambiguously decomposed into pose and calibration CS 223 b 44

Projective Reconstruction n Euclidean Motion Cannot be obtained in uncalibrated setting (F cannot be uniquely decomposed into R, T and K matrix) Can we still say something about 3 D ? Notion of the projective 3 D structure (study of projective geometry) Euclidean reconstruction Projective reconstruction CS 223 b 45

Euclidean vs Projective reconstruction n n Euclidean reconstruction – true metric properties of objects lenghts (distances), angles, parallelism are preserved Unchanged under rigid body transformations => Euclidean Geometry – properties of rigid bodies under rigid body transformations, similarity transformation Projective reconstruction – lengths, angles, parallelism are NOT preserved – we get distorted images of objects – their distorted 3 D counterparts --> 3 D projective reconstruction => Projective Geometry CS 223 b 46

Image rectification n Given general displacement how to warp the views Such that epipolar lines are parallel to each other How to warp it back to canonical configuration (more details later) (Seitz)

Epipolar rectification • Rectified Image Pair • Corresponding epipolar lines are aligned with the scan-lines • Search for dense correspondence is a 1 D search CS 223 b 48

Epipolar rectification Rectified Image Pair CS 223 b 49

Special Motions – Pure Rotation n Calibrated Two views related by rotation only n Uncalibrated Case Mapping to a reference view – H can be estimated n Mapping to a cylindrical surface - applications – image mosaics n CS 223 b 50

Epipolar rectification • Rectified Image Pair • Corresponding epipolar lines are aligned with the scan-lines • Search for dense correspondence is a 1 D search CS 223 b 51

Using H p’ p • Image based rectification (given some points in 3 D world) compute H which would map them into a square • Use H to rectify the entire image • In calibrated case inter-image homography CS 223 b 52

Single view modeling in projective setting Vermeer’s Music Lesson Planes and distances from the planes can be computed Single view Rendering (A. Crimisi) CS 223 b 53

More on stereo CS 223 b 54

Ordering constraint n n Usually, order of points in two images is same. Is this always true? If we match pixel i in image 1 to pixel j in image 2, no matches that follow will affect which are the best preceding matches. Example with pixels (a la Cox et al. ). CS 223 b 55

The Ordering Constraint Points on the epipolar lines appear in the same order But it is not always the case. . . This enables dynamic programming CS 223 b 56

Dynamic Programming (Baker and Binford, 1981) Find the minimum-cost path going monotonically down and right from the top-left corner of the graph to its bottom-right corner. • Nodes = matched feature points (e. g. , edge points). • Arcs = matched intervals along the epipolar lines. • Arc cost = discrepancy between intervals. CS 223 b 57

Dynamic Programming (Baker and Binford, 1981) Find the minimum-cost path going monotonically down and right from the top-left corner of the graph to its bottom-right corner. • Nodes = matched feature points (e. g. , edge points). • Arcs = matched intervals along the epipolar lines. • Arc cost = discrepancy between intervals. CS 223 b 58

Dynamic Programming (Ohta and Kanade, 1985) Reprinted from “Stereo by Intra- and Intet-Scanline Search, ” by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and Machine CS 223 b Intelligence, 7(2): 139 -154 (1985). 1985 IEEE. 59

Other constraints n n n Smoothness: disparity usually doesn’t change too quickly. n Unfortunately, this makes the problem 2 D again. n Solved with a host of graph algorithms, Markov Random Fields, Belief Propagation, …. Uniqueness constraint (each feature can at most have one match) Occlusion and disparity are connected. CS 223 b 60