3 D Urban Modeling Marc Pollefeys JanMichael Frahm

3 D Urban Modeling Marc Pollefeys Jan-Michael Frahm, Philippos Mordohai Fall 2006 / Comp 790 -089 Tue & Thu 14: 00 -15: 15 SN 252

3 D Urban Modeling • Basics: sensor, techniques, … • Video, LIDAR, GPS, IMU, … • Sf. M, Stereo, … • Robot Mapping, GIS, … • Review state-of-the-art • Video and LIDAR-based systems • Projects to experiment • Virtual UNC-Chapel Hill • DARPA Urban Challenge • …

3 D Urban Modeling course schedule (tentative) Sep. 5, 7 Introduction Video-based Urban 3 D Capture Sep. 12, 14 Cameras Epi. Geom. and Triangulation Sep. 19, 21 Feature Tracking & Matching Stereo matching (Sudipta) (Philippos) GPS/INS/LIDAR Project proposals Sep. 26, 28 (Brian) (Jan-Michael) Oct. 3, 5 Oct. 10, 12 Oct. 17, 19 Oct. 24, 26 (fall break) Project Status Update Oct. 31, Nov. 2 Nov. 7, 9 Nov. 14, 16 Nov. 21, 23 (Thanksgiving) Nov. 28, 30 Dec. 5 Project demonstrations Note: Dec. 3 is CVPR deadline (classes ended)

Course material Slides, notes, papers and references see course webpage/wiki (later) On-line “shape-from-video” tutorial: http: //www. cs. unc. edu/~marc/tutorial. pdf http: //www. cs. unc. edu/~marc/tutorial/

Epipolar geometry? courtesy Frank Dellaert

Epipolar geometry Underlying structure in set of matches for rigid scenes p L 2 C 1 m 1 l T 1 l 2 M l 1 e 2 Fundamental matrix (3 x 3 rank 2 matrix) Canonical representation: L 1 m 2 l 2 C 2 1. Computable from corresponding points 2. Simplifies matching 3. Allows to detect wrong matches 4. Related to calibration

Epipolar geometry P P L 2 C 1 x 1 M L 1 l T 1 l 2 e 1 e 2 Fundamental matrix (3 x 3 rank 2 matrix) x 2 C 2 l 2

The projective reconstruction theorem If a set of point correspondences in two views determine the fundamental matrix uniquely, then the scene and cameras may be reconstructed from these correspondences alone, and any two such reconstructions from these correspondences are projectively equivalent allows reconstruction from pair of uncalibrated images!

Computation of F • • Linear (8 -point) Minimal (7 -point) Robust (RANSAC) Non-linear refinement (MLE, …) • Practical approach

Epipolar geometry: basic equation separate known from unknown (data) (unknowns) (linear)

the NOT normalized 8 -point algorithm ~10000 ~1000 0 ! ~10000 ~100 ~10 0 1 Orders of magnitude difference between column of data matrix least-squares yields poor results

the normalized 8 -point algorithm Transform image to ~[-1, 1]x[-1, 1] (0, 500) (700, 500) (-1, 1) (0, 0) (700, 0) (-1, -1) normalized least squares yields good results (Hartley, PAMI´ 97) (1, -1)

the singularity constraint SVD from linearly computed F matrix (rank 3) Compute closest rank-2 approximation

the minimum case – 7 point correspondences one parameter family of solutions but F 1+l. F 2 not automatically rank 2

the minimum case – impose rank 2 3 (obtain 1 or 3 solutions) F 7 pts F F 1 F 2 (cubic equation) Compute possible l as eigenvalues of (only real solutions are potential solutions) Minimal solution for calibrated cameras: 5 -point

Robust estimation • What if set of matches contains gross outliers? (to keep things simple let’s consider line fitting first)

RANSAC (RANdom Sampling Consensus) Objective Robust fit of model to data set S which contains outliers Algorithm (i) Randomly select a sample of s data points from S and instantiate the model from this subset. (ii) Determine the set of data points Si which are within a distance threshold t of the model. The set Si is the consensus set of samples and defines the inliers of S. (iii) If the subset of Si is greater than some threshold T, reestimate the model using all the points in Si and terminate (iv) If the size of Si is less than T, select a new subset and repeat the above. (v) After N trials the largest consensus set Si is selected, and the model is re-estimated using all the points in the subset Si

Distance threshold Choose t so probability for inlier is α (e. g. 0. 95) • Often empirically • Zero-mean Gaussian noise σ then follows distribution with m=codimension of model (dimension+codimension=dimension space) Codimension Model t 2 1 line, F 3. 84σ2 2 H, P 5. 99σ2 3 T 7. 81σ2

How many samples? Choose N so that, with probability p, at least one random sample is free from outliers. e. g. p=0. 99 proportion of outliers e s 2 3 4 5 6 7 8 5% 2 3 3 4 4 4 5 10% 25% 30% 40% 50% 3 5 6 7 11 17 4 7 9 11 19 35 5 9 13 17 34 72 6 12 17 26 57 146 7 16 24 37 97 293 8 20 33 54 163 588 9 26 44 78 272 1177 Note: Assumes that inliers allow to identify other inliers

Acceptable consensus set? • Typically, terminate when inlier ratio reaches expected ratio of inliers

Adaptively determining the number of samples e is often unknown a priori, so pick worst case, i. e. 0, and adapt if more inliers are found, e. g. 80% would yield e=0. 2 • N=∞, sample_count =0 • While N >sample_count repeat • • Choose a sample and count the number of inliers Set e=1 -(number of inliers)/(total number of points) Recompute N from e Increment the sample_count by 1 • Terminate

Other robust algorithms • RANSAC maximizes number of inliers • LMed. S minimizes median error inlier percentile 100% 50% residual (pixels) 1. 25 • Not recommended: case deletion, iterative least-squares, etc.

Non-linear refinment

Geometric distance Gold standard Symmetric epipolar distance

Gold standard Maximum Likelihood Estimation (= least-squares for Gaussian noise) Initialize: normalized 8 -point, (P, P‘) from F, reconstruct Xi Parameterize: (overparametrized) Minimize cost using Levenberg-Marquardt (preferably sparse LM, e. g. see H&Z)

Gold standard Alternative, minimal parametrization (with a=1) (note (x, y, 1) and (x‘, y‘, 1) are epipoles) problems: • a=0 pick largest of a, b, c, d to fix to 1 • epipole at infinity pick largest of x, y, w and of x’, y’, w’ 4 x 3 x 3=36 parametrizations! reparametrize at every iteration, to be sure

Symmetric epipolar error

Some experiments:

Some experiments:

Some experiments:

Some experiments: Residual error: (for all points!)

Recommendations: 1. Do not use unnormalized algorithms 2. Quick and easy to implement: 8 -point normalized 3. Better: enforce rank-2 constraint during minimization 4. Best: Maximum Likelihood Estimation (minimal parameterization, sparse implementation)

Automatic computation of F Step 1. Extract features Step 2. Compute a set of potential matches Step 3. do Step 3. 1 select minimal sample (i. e. 7 matches) Step 3. 2 compute solution(s) for F Step 3. 3 determine inliers (verify hypothesis) (generate hypothesis) until (#inliers, #samples)<95% Step 4. Compute F based on all inliers Step 5. Look for additional matches Step 6. Refine F based on all correct matches #inliers 90% 80% 70% 60% 50% #samples 5 13 35 106 382

Abort verification early OOOOOIOOIOOOO I I I I O O I I I O I O O I I… OOOOOOOIOOOO Assume percentage of inliers p, what is the probability P to pick n or more wrong samples out of m? stop if P<0. 05 (sample most probably contains outlier) (P=cum. binomial distr. funct. (m-n, m, p )) To avoid problems this requires to also verify at random! (but we already have a random sampler anyway) (inspired from Chum and Matas BMVC 2002)

Finding more matches restrict search range to neighborhood of epipolar line (e. g. 1. 5 pixels) relax disparity restriction (along epipolar line)

Degenerate cases: • Degenerate cases • Planar scene • Pure rotation • No unique solution • Remaining DOF filled by noise • Use simpler model (e. g. homography) • Solution 1: Model selection (Torr et al. , ICCV´ 98, Kanatani, Akaike) • Compare H and F according to expected residual error (compensate for model complexity) • Solution 2: RANSAC • Compare H and F according to inlier count (see next slide)

RANSAC for quasi-degenerate cases • Full model (8 pts, 1 D solution) (accept inliers to solution F) • Planar model (6 pts, 3 D solution) (accept inliers to solution F 1, F 2&F 3) Accept if large number of remaining inliers • Plane+parallax model (plane+2 pts) closest rank-6 of Anx 9 for all plane inliers Sample for out of plane points among outliers

More problems: • Absence of sufficient features (no texture) • Repeated structure ambiguity • Robust matcher also finds support for wrong hypothesis • solution: detect repetition (Schaffalitzky and Zisserman, BMVC‘ 98)

RANSAC for ambiguous matching • Include multiple candidate matches in set of potential matches • Select according to matching probability (~ matching score) • Helps for repeated structures or scenes with similar features as it avoids an early commitment (Tordoff and Murray ECCV 02)

two-view geometry geometric relations between two views is fully described by recovered 3 x 3 matrix F

Triangulation L 2 C 1 x 1 X L 1 Triangulation x 2 C 2 - calibration - correspondences

Triangulation • Backprojection L 2 X C 1 x 1 L 1 x 2 C 2 • Triangulation Iterative least-squares • Maximum Likelihood Triangulation

Optimal 3 D point in epipolar plane • Given an epipolar plane, find best 3 D point for (m 1, m 2) l 1 m 1 ´ m 2 ´ l 2 l 1 m 2 m 2 l 2 Select closest points (m 1´, m 2´) on epipolar lines Obtain 3 D point through exact triangulation Guarantees minimal reprojection error (given this epipolar plane)

Non-iterative optimal solution • Reconstruct matches in projective frame by minimizing the reprojection error 3 DOF • Non-iterative method (Hartley and Sturm, CVIU´ 97) Determine the epipolar plane for reconstruction (polynomial of degree 6) l 1(a) m 1 l 2(a) m 2 1 DOF Reconstruct optimal point from selected epipolar plane Note: only works for two views