Computer Vision Multiple View Geometry Stereo Marc Pollefeys

Computer Vision Multiple View Geometry & Stereo Marc Pollefeys COMP 256

Computer Vision Last class: epipolar geometry p Underlying structure in set of matches for rigid scenes C 1 L 2 p 1 M L 1 l T 1 l 2 e 1 e 2 Fundamental matrix (3 x 3 rank 2 matrix) Canonical representation: 2 p 2 l 2 C 2 1. Computable from corresponding points 2. Simplifies matching 3. Allows to detect wrong matches 4. Related to calibration

Tentative class schedule Computer Vision Jan 16/18 - Introduction Jan 23/25 Cameras Radiometry Sources & Shadows Color Feb 6/8 Linear filters & edges Texture Feb 13/15 Multi-View Geometry Stereo Feb 20/22 Optical flow Project proposals Affine Sf. M Projective Sf. M Camera Calibration Silhouettes and Photoconsistency Mar 13/15 Springbreak Mar 20/22 Segmentation Fitting Mar 27/29 Prob. Segmentation Project Update Tracking Apr 10/12 Object Recognition Apr 17/19 Range data Final project Jan 30/Feb 1 Feb 27/Mar 1 Mar 6/8 Apr 3/5 3 Apr 24/26

Computer Vision 4 Multiple Views (Faugeras and Mourrain, 1995)

Computer Vision Two Views Epipolar Constraint 5

Computer Vision Three Views Trifocal Constraint 6

Computer Vision Four Views Quadrifocal Constraint (Triggs, 1995) 7

Computer Vision 8 Geometrically, the four rays must intersect in P. .

Computer Vision 9 Quadrifocal Tensor and Lines

Computer Vision Quadrifocal tensor • determinant is multilinear thus linear in coefficients of lines ! • There must exist a tensor with 81 coefficients containing all possible combination of x, y, w coefficients for all 4 images: the quadrifocal tensor 10

from perspective to omnidirectional cameras Computer Vision 3 constraints allow to reconstruct 3 D point perspective camera (2 constraints / feature) more constraints also tell something about cameras l=(y, -x) (0, 0) (x, y) radial camera (uncalibrated) multilinear constraints known as epipolar, trifocal and quadrifocal constraints (1 constraints / feature) 11

Computer (x, y) Vision Radial quadrifocal tensor • Linearly compute radial quadrifocal tensor Qijkl from 15 pts in 4 views (2 x 2 x 2 x 2 tensor) • Reconstruct 3 D scene and use it for calibration Not easy for real data, hard to avoid degenerate cases (e. g. 3 optical axes intersect in single point). However, degenerate case leads to simpler 3 view algorithm for pure rotation • Radial trifocal tensor Tijk from 7 points in 3 views (2 x 2 x 2 tensor) • Reconstruct 2 D panorama and use it for calibration 12

Computer Vision Non-parametric distortion calibration (Thirthala and Pollefeys, ICCV’ 05) angle • Models fish-eye lenses, cata-dioptric systems, etc. normalized radius 13

Computer Vision Non-parametric distortion calibration (Thirthala and Pollefeys, ICCV’ 05) • Models fish-eye lenses, cata-dioptric systems, etc. angle 90 o normalized radius 14

Computer Vision STEREOPSIS • The Stereopsis Problem: Fusion and Reconstruction • Human Stereopsis and Random Dot Stereograms • Cooperative Algorithms • Correlation-Based Fusion • Multi-Scale Edge Matching • Dynamic Programming • Using Three or More Cameras Reading: Chapter 11. 15

Computer Vision An Application: Mobile Robot Navigation The INRIA Mobile Robot, 1990. The Stanford Cart, H. Moravec, 1979. 16 Courtesy O. Faugeras and H. Moravec.

Computer Vision 17 Reconstruction / Triangulation

Computer Vision 18 (Binocular) Fusion

Computer Vision Reconstruction • Linear Method: find P such that • Non-Linear Method: find Q minimizing 19

Computer Vision Rectification All epipolar lines are parallel in the rectified image plane. 20

Computer Vision Image pair rectification simplify stereo matching by warping the images Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole e to (1, 0, 0) try to minimize image distortion 21 problem when epipole in (or close to) the image

Computer Vision 23

Computer Vision 24

Computer Vision Polar rectification (Pollefeys et al. ICCV’ 99) Polar re-parameterization around epipoles Requires only (oriented) epipolar geometry Preserve length of epipolar lines Choose so that no pixels are compressed original image Works for all relative motions Guarantees minimal image size 25 rectified image

Computer Vision 26 polar rectification: example

Computer Vision 27 polar rectification: example

Computer Vision Example: Béguinage of Leuven Does not work with standard Homography-based approaches 28

Computer Vision 29 Example: Béguinage of Leuven

Computer Vision Reconstruction from Rectified Images Disparity: d=u’-u. 30 Depth: z = -B/d.

Computer Vision Stereopsis Figure from US Navy Manual of Basic Optics and Optical Instruments, prepared by Bureau of Naval Personnel. Reprinted by Dover Publications, Inc. , 1969. 31

Computer Vision Human Stereopsis: Reconstruction d=0 Disparity: 32 d = r-l = D-F. In 3 D, the horopter. d<0

Computer Vision 33 Human Stereopsis: experimental horopter…

Computer Vision Iso-disparity curves: planar retinas X 0 X 1 Xi Xj C 1 C 2 X∞ 34 the retina act as if it were flat!

Computer Vision Human Stereopsis: Reconstruction What if F is not known? Helmoltz (1909): • There is evidence showing the vergence angles cannot be measured precisely. • Humans get fooled by bas-relief sculptures. • There is an analytical explanation for this. • Relative depth can be judged accurately. 35

Computer Vision Human Stereopsis: Binocular Fusion How are the correspondences established? Julesz (1971): Is the mechanism for binocular fusion a monocular process or a binocular one? ? • There is anecdotal evidence for the latter (camouflage). • Random dot stereograms provide an objective answer 36 BP!

Computer Vision A Cooperative Model (Marr and Poggio, 1976) Excitory connections: continuity Inhibitory connections: uniqueness Iterate: C = S Ce - w. S C i + C 0. 37 Reprinted from Vision: A Computational Investigation into the Human Representation and Processing of Visual Information by David Marr. 1982 by David Marr. Reprinted by permission of Henry Holt and Company, LLC.

Computer Vision Correlation Methods (1970 --) Slide the window along the epipolar line until w. w’ is maximized. 38 Normalized Correlation: minimize instead. Minimize |w-w’|. 2

Computer Vision Correlation Methods: Foreshortening Problems Solution: add a second pass using disparity estimates to warp the correlation windows, e. g. Devernay and Faugeras (1994). 39 Reprinted from “Computing Differential Properties of 3 D Shapes from Stereopsis without 3 D Models, ” by F. Devernay and O. Faugeras, Proc. IEEE Conf. on Computer Vision and Pattern Recognition (1994). 1994 IEEE.

Computer Vision Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979 -81) • Edges are found by repeatedly smoothing the image and detecting the zero crossings of the second derivative (Laplacian). • Matches at coarse scales are used to offset the search for matches at fine scales (equivalent to eye movements). 40

Computer Vision Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979 -81) One of the two input images Image Laplacian Zeros of the Laplacian 41 Reprinted from Vision: A Computational Investigation into the Human Representation and Processing of Visual Information by David Marr. 1982 by David Marr. Reprinted by permission of Henry Holt and Company, LLC.

Computer Vision 42 Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979 -81) Reprinted from Vision: A Computational Investigation into the Human Representation and Processing of Visual Information by David Marr. 1982 by David Marr. Reprinted by permission of Henry Holt and Company, LLC.

Computer Vision The Ordering Constraint In general the points are in the same order on both epipolar lines. But it is not always the case. . 43

Computer Vision Dynamic Programming (Baker and Binford, 1981) Find the minimum-cost path going monotonically down and right from the top-left corner of the graph to its bottom-right corner. • Nodes = matched feature points (e. g. , edge points). • Arcs = matched intervals along the epipolar lines. • Arc cost = discrepancy between intervals. 44

Computer Vision 45 Dynamic Programming (Ohta and Kanade, 1985) Reprinted from “Stereo by Intra- and Intet-Scanline Search, ” by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and Machine Intelligence, 7(2): 139 -154 (1985). 1985 IEEE.

Computer Vision Three Views The third eye can be used for verification. . 46

Computer Vision More Views (Okutami and Kanade, 1993) Pick a reference image, and slide the corresponding window along the corresponding epipolar lines of all other images, using inverse depth relative to the first image as the search parameter. Reprinted from “A Multiple-Baseline Stereo System, ” by M. Okutami and T. Kanade, IEEE Trans. on Pattern Analysis and Machine Intelligence, 15(4): 353 -363 (1993). copyright 1993 IEEE. 47 Use the sum of correlation scores to rank matches.

Computer Vision Stereo matching Similarity measure (SSD or NCC) Optimal path (dynamic programming ) 48 Constraints • epipolar • ordering • uniqueness • disparity limit • disparity gradient limit Trade-off • Matching cost (data) • Discontinuities (prior) (Cox et al. CVGIP’ 96; Koch’ 96; Falkenhagen´ 97; Van Meerbergen, Vergauwen, Pollefeys, Van. Gool IJCV‘ 02)

Computer Vision Hierarchical stereo matching Allows faster computation 49 Disparity propagation (Gaussian pyramid) Downsampling Deals with large disparity ranges (Falkenhagen´ 97; Van Meerbergen, Vergauwen, Pollefeys, Van. Gool IJCV‘ 02)

Computer Vision image I(x, y) Disparity map D(x, y) (x´, y´)=(x+D(x, y) 50 image I´(x´, y´)

Computer Vision 51 Example: reconstruct image from neighboring images

Computer Vision I 1 52 I 10 Reprinted from “A Multiple-Baseline Stereo System, ” by M. Okutami and T. Kanade, IEEE Trans. on Pattern Analysis and Machine Intelligence, 15(4): 353 -363 (1993). copyright 1993 IEEE.

Computer Vision Multi-view depth fusion (Koch, Pollefeys and Van Gool. ECCV‘ 98) • Compute depth for every pixel of reference image – Triangulation – Use multiple views – Up- and down sequence – Use Kalman filter Allows to compute robust texture 53

Computer Vision Real-time stereo on graphics hardware Yang and Pollefeys CVPR 03 • Computes Sum-of-Square-Differences • Hardware mip-map generation used to aggregate results over support region • Trade-off between small and large support window Shape of a kernel for summing up 6 levels 54 140 M disparity hypothesis/sec on Radeon 9700 pro e. g. 512 x 20 disparities at 30 Hz

Computer Vision near 55 Sample Re-Projections far

Computer Vision Combine multiple aggregation windows using hardware mipmap and multiple texture units in single pass (1 x 1) (1 x 1+2 x 2) video 56 (1 x 1+2 x 2 +4 x 4+8 x 8) (1 x 1+2 x 2 +4 x 4+8 x 8 +16 x 16)

Computer Vision Cool ideas • Space-time stereo (varying illumination, not shape) 57

Computer Vision More on stereo … The Middleburry Stereo Vision Research Page http: //cat. middlebury. edu/stereo/ Recommended reading D. Scharstein and R. Szeliski. A Taxonomy and Evaluation of Dense Two. Frame Stereo Correspondence Algorithms. IJCV 47(1/2/3): 7 -42, April-June 2002. PDF file (1. 15 MB) - includes current evaluation. Microsoft Research Technical Report MSR-TR 2001 -81, November 2001. PDF file (1. 27 MB). 58

Computer Vision 59 Next class: Optical Flow: where do pixels move to?