Some problems Lens distortion Uncalibrated structure and motion
Some problems. . .
Lens distortion è Uncalibrated structure and motion recovery assumes pinhole cameras è Real cameras have real lenses è How can we correct distortion, when original calibration is inaccessible?
1. 2. 3. 4. 5. Even small amounts of lens distortion can upset uncalibrated structure from motion A single distortion parameter is enough for mapping and SFX accuracy Including the parameter k in the multiview relations changes the 8 -point algorithm from You can solve such “Polynomial Eigenvalue Problems” This is as stable as computation of the Fundamental matrix, so you can use it all the time.
E ven small amounts of lens distortion can upset uncalibrated structure from motion—
A map-building problem (a) (b) (a) Input movie – relatively low distortion (b) Plan view: red is structure, blue is motion
Effects of Distortion (a) (c) (a) Input movie – relatively low distortion (b) Recovered plan view, uncorrected distortion
Does distortion do that? Distortion of image plane is conflated with focal length when the camera rotates [From: Tordoff & Murray, ICPR 2000]
Distortion correction in man-made scenes
Distortion correction in natural scenes [Farid and Popescu, ICCV 2001] è In natural images, distortion introduces correlations in frequency domain è Choose distortion parameters to minimize correlations in bispectrum è Less effective on manmade scenes. .
Distortion correction in multiple images Multiple views, static scene • Use motion and scene rigidity [Zhang, Stein, Sawhney, Mc. Lauchlan, . . . ] Advantages: • Applies to man-made or natural scenes Disadvantages: • Iterative solutions|require initial estimates
A single distortion parameter is accurate enough for map-building and cinema post production—
Modelling lens distortion p x x: xeroxed noxious experimental artifax Known p x p: perfect pinhole perspective pure Unknown
Single-parameter models
Single-parameter modelling power Single-parameter model Radial term only Assumes distortion centre is at centre of image A one-parameter model suffices
A direct solution for k
Look at division model again
A quick matlab session >> help polyeig POLYEIG Polynomial eigenvalue problem. [X, E] = POLYEIG(A 0, A 1, . . , Ap) solves the polynomial eigenvalue problem of degree p: (A 0 + lambda*A 1 +. . . + lambda^p*Ap)*x = 0. The input is [etc etc. . . ] >>
Algorithm
T his is as stable as computation of the fundamental matrix, so you can use it all the time—
Performance: Synthetic data • Stable – small errorbars • Biased – not centred on true value Computed l 0 -0. 1 -0. 2 -0. 3 -0. 4 0 0. 2 0. 4 0. 6 Noise s (pixels) 0. 8 1
Analogy: Linear ellipse fitting True Fitted: 10 trials Best-fit line Data
Performance: Synthetic data
Performance: Real sequences
50 40 30 20 10 • • 250 pairs 0 -0. 6 -0. 5 -0. 4 -0. 3 -0. 2 -0. 1 0 Low distortion Linear estimate used to initialize nonlinear Number of inliers changes by [-25. . 49] 0. 1 0. 2 0. 3
Conclusions
Environment matting In: magnifying glass moving over background Out: same magnifying glass, new background
Environment matting: why? • Learn – light-transport properties of complex optical elements • Previously – Ray tracing geometric models – Calibrated acquisition • Here – Acquisition in situ
Image formation model • Purely 2 D-2 D – Optical element performs weighted sum of (image of) background at each pixel – suffices for many interesting objects – separate receptive field for each output pixel – Environment matte is collection of all receptive fields—yes, it’s huge.
Image formation model
Step 1: Computing background Input: Point tracks: Mosaic: Clean plate:
Input: Step 2: Computing w. . .
Computing w(x, y, u, v) at a single (x, y)
Assume wi independent
Composite over new background
A more subtle example Input: Two images Moving camera Planar background - Need priors
Window example
Discussion • Works well for non-translucent elements – need to develop for diffuse • Combination assumes independence – ok for large movements: “an edge crosses the pixel” • Need to develop for general backgrounds
A Clustering Problem • Watch a movie, recover the cast list – Run face detector on every frame – Cluster faces • Problems – Face detector unreliable – Large lighting changes – Changes in expression – Clustering is difficult
A sample sequence
Detected faces
Face positions
Lighting correction
Clustering: pairwise distances Raw distance
Clustering: pairwise distances Transform-invariant distance
Clusters: “tangent distance”
Clusters: Bayesian tangent distance
Conclusions • Extend to feature selection, texton clustering etc • Remove face detector
- Slides: 53