Stereo STEREOPSIS The Stereopsis Problem Fusion and Reconstruction

Stereo

STEREOPSIS • The Stereopsis Problem: Fusion and Reconstruction • Human Stereopsis and Random Dot Stereograms • Cooperative Algorithms • Correlation-Based Fusion • Multi-Scale Edge Matching • Dynamic Programming • Using Three or More Cameras Reading: Chapter 11.

An Application: Mobile Robot Navigation The INRIA Mobile Robot, 1990. The Stanford Cart, H. Moravec, 1979. Courtesy O. Faugeras and H. Moravec.

Reconstruction / Triangulation

(Binocular) Fusion

Reconstruction • Linear Method: find P such that • Non-Linear Method: find Q minimizing

Rectification All epipolar lines are parallel in the rectified image plane.

Image pair rectification simplify stereo matching by warping the images Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole e to (1, 0, 0) try to minimize image distortion problem when epipole in (or close to) the image

Polar rectification (Pollefeys et al. ICCV’ 99) Polar re-parameterization around epipoles Requires only (oriented) epipolar geometry Preserve length of epipolar lines Choose so that no pixels are compressed original image Works for all relative motions Guarantees minimal image size rectified image

polar rectification: example

Reconstruction from Rectified Images Disparity: d=u’-u. Depth: z = -B/d.

Human Stereopsis: Binocular Fusion How are the correspondences established? Julesz (1971): Is the mechanism for binocular fusion a monocular process or a binocular one? ? • There is anecdotal evidence for the latter (camouflage). • Random dot stereograms provide an objective answer BP!

A Cooperative Model (Marr and Poggio, 1976) Excitory connections: continuity Inhibitory connections: uniqueness Iterate: C = S Ce - w. S C i + C 0. Reprinted from Vision: A Computational Investigation into the Human Representation and Processing of Visual Information by David Marr. 1982 by David Marr. Reprinted by permission of Henry Holt and Company, LLC.

Correlation Methods (1970 --) Slide the window along the epipolar line until w. w’ is maximized. Normalized Correlation: minimize instead. Minimize |w-w’|. 2

Correlation Methods: Foreshortening Problems Solution: add a second pass using disparity estimates to warp the correlation windows, e. g. Devernay and Faugeras (1994). Reprinted from “Computing Differential Properties of 3 D Shapes from Stereopsis without 3 D Models, ” by F. Devernay and O. Faugeras, Proc. IEEE Conf. on Computer Vision and Pattern Recognition (1994). 1994 IEEE.

Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979 -81) • Edges are found by repeatedly smoothing the image and detecting the zero crossings of the second derivative (Laplacian). • Matches at coarse scales are used to offset the search for matches at fine scales (equivalent to eye movements).

Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979 -81) One of the two input images Image Laplacian Zeros of the Laplacian Reprinted from Vision: A Computational Investigation into the Human Representation and Processing of Visual Information by David Marr. 1982 by David Marr. Reprinted by permission of Henry Holt and Company, LLC.

Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979 -81) Reprinted from Vision: A Computational Investigation into the Human Representation and Processing of Visual Information by David Marr. 1982 by David Marr. Reprinted by permission of Henry Holt and Company, LLC.

The Ordering Constraint In general the points are in the same order on both epipolar lines. But it is not always the case. .

Dynamic Programming (Baker and Binford, 1981) Find the minimum-cost path going monotonically down and right from the top-left corner of the graph to its bottom-right corner. • Nodes = matched feature points (e. g. , edge points). • Arcs = matched intervals along the epipolar lines. • Arc cost = discrepancy between intervals.

Dynamic Programming (Ohta and Kanade, 1985) Reprinted from “Stereo by Intra- and Intet-Scanline Search, ” by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and Machine Intelligence, 7(2): 139 -154 (1985). 1985 IEEE.

Three Views The third eye can be used for verification. .

More Views (Okutami and Kanade, 1993) Pick a reference image, and slide the corresponding window along the corresponding epipolar lines of all other images, using inverse depth relative to the first image as the search parameter. Reprinted from “A Multiple-Baseline Stereo System, ” by M. Okutami and T. Kanade, IEEE Trans. on Pattern Analysis and Machine Intelligence, 15(4): 353 -363 (1993). copyright 1993 IEEE. Use the sum of correlation scores to rank matches.

Stereo matching Similarity measure (SSD or NCC) Optimal path (dynamic programming ) Constraints • epipolar • ordering • uniqueness • disparity limit • disparity gradient limit Trade-off • Matching cost (data) • Discontinuities (prior) (Cox et al. CVGIP’ 96; Koch’ 96; Falkenhagen´ 97; Van Meerbergen, Vergauwen, Pollefeys, Van. Gool IJCV‘ 02)

Hierarchical stereo matching Allows faster computation Disparity propagation (Gaussian pyramid) Downsampling Deals with large disparity ranges (Falkenhagen´ 97; Van Meerbergen, Vergauwen, Pollefeys, Van. Gool IJCV‘ 02)

Disparity map image I(x, y) Disparity map D(x, y) (x´, y´)=(x+D(x, y) image I´(x´, y´)

Example: reconstruct image from neighboring images

I 1 I 2 I 10 Reprinted from “A Multiple-Baseline Stereo System, ” by

Real-time stereo on graphics hardware Ruigang Yang and Marc Pollefeys l l l Computes Sum-of-Square-Differences Hardware mip-map generation used to aggregate results over support region Trade-off between small and large support window Shape of a kernel for summing up 6 levels 140 M disparity hypothesis/sec on Radeon 9700 pro e. g. 512 x 20 disparities at 30 Hz

Combine multiple aggregation windows using hardware mipmap and multiple texture units in single pass (1 x 1) (1 x 1+2 x 2 +4 x 4+8 x 8) (1 x 1+2 x 2 +4 x 4+8 x 8 +16 x 16)

Live stereo demo PC with ATI Radeon 9700 pro Bumblebee (Point Grey) Open. GL 1. 4 (Fragment programs) Code available at: http: //vis. uky. edu/~ryang/research/View. Syn/

More on stereo … The Middleburry Stereo Vision Research Page http: //cat. middlebury. edu/stereo/ Recommended reading D. Scharstein and R. Szeliski. A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms. IJCV 47(1/2/3): 7 -42, April-June 2002. PDF file (1. 15 MB) - includes current evaluation. Microsoft Research Technical Report MSR-TR-2001 -81, November 2001. PDF file (1. 27 MB).