Computer vision models learning and inference Chapter 14

Structure • • Pinhole camera model Three geometric problems Homogeneous coordinates Solving the problems

Motivation Sparse stereo reconstruction Compute the depth at a set of sparse matching points

Pinhole camera Real camera image is inverted Instead model impossible but more convenient virtual

Pinhole camera terminology Computer vision: models, learning and inference. © 2011 Simon J. D.

Normalized Camera By similar triangles: Computer vision: models, learning and inference. © 2011 Simon

Focal length parameters Computer vision: models, learning and inference. © 2011 Simon J. D.

Focal length parameters Can model both • the effect of the distance to the

Offset parameters • Current model assumes that pixel (0, 0) is where the principal

Skew parameter • Finally, add skew parameter • Accounts for image plane being not

Position and orientation of camera • Position w=(u, v, w)T of point in the

Complete pinhole camera model • Intrinsic parameters (stored as intrinsic matrix) • Extrinsic parameters

Complete pinhole camera model For short: Add noise – uncertainty in localizing feature in

Radial distortion Computer vision: models, learning and inference. © 2011 Simon J. D. Prince

Problem 1: Learning extrinsic parameters (exterior orientation) Use maximum likelihood: Computer vision: models, learning

Problem 2 – Learning intrinsic parameters (calibration) Use maximum likelihood: Computer vision: models, learning

Calibration • Use 3 D target with known 3 D points Computer vision: models,

Problem 3 – Inferring 3 D points (triangulation / reconstruction) Use maximum likelihood: Computer

Solving the problems • None of these problems can be solved in closed form

Homogeneous coordinates Convert 2 D coordinate to 3 D To convert back Computer vision:

Geometric interpretation of homogeneous coordinates Computer vision: models, learning and inference. © 2011 Simon

Pinhole camera in homogeneous coordinates Camera model: In homogeneous coordinates: (linear!) Computer vision: models,

Pinhole camera in homogeneous coordinates Writing out these three equations Eliminate l to retrieve

Adding in extrinsic parameters Or for short: Or even shorter: Computer vision: models, learning

Exterior orientation Start with camera equation in homogeneous coordinates Pre-multiply both sides by inverse

Exterior orientation The third equation gives us an expression for l Substitute back into

Exterior orientation Linear equation – two equations per point – form system of equations

Exterior orientation Minimum direction problem of the form Find minimum of subject to To

Exterior orientation Now we extract the values of and from . Problem: the scale

Calibration One approach is to alternately • Optimize extrinsic parameters for fixed intrinsic •

Intrinsic parameters Maximum likelihood approach This is a least squares problem. Computer vision: models,

Intrinsic parameters The function is linear w. r. t. intrinsic parameters. Can be written

Reconstruction Write jth pinhole camera in homogeneous coordinates: Pre-multiply with inverse of intrinsic matrix

Reconstruction Last equations gives Substitute back into first two equations Re-arranging get two linear

Reconstruction • The method requires correspondences between points in different images – SIFT descriptors

Depth from structured light Computer vision: models, learning and inference. © 2011 Simon J.

Depth from structured light • Camera and projectors are calibrated – The projector illuminates

Shape from silhouette • Adding cameras constraints the shape to the 3 D intersection

Shape from silhouette (augmented reality) • The actor is captured from 15 cameras in

Shape from silhouette (novel view generation) • In the new view, compute the ray

Conclusion • Pinhole camera model is a non-linear function who takes points in 3

Slides: 52

Download presentation

Computer vision: models, learning and inference Chapter 14 The pinhole camera Please send errata to s. prince@cs. ucl. ac. uk

Structure • • Pinhole camera model Three geometric problems Homogeneous coordinates Solving the problems – Exterior orientation problem – Camera calibration – 3 D reconstruction • Applications Computer vision: models, learning and inference. © 2011 Simon J. D. Prince 2

Focal length parameters Can model both • the effect of the distance to the focal plane • the density of the receptors with a single focal length parameter f In practice, the receptors may not be square: So use different focal length parameter for x and y dirns Computer vision: models, learning and inference. © 2011 Simon J. D. Prince 8

Offset parameters • Current model assumes that pixel (0, 0) is where the principal ray strikes the image plane (i. e. the center) • Model offset to center Computer vision: models, learning and inference. © 2011 Simon J. D. Prince 9

Skew parameter • Finally, add skew parameter • Accounts for image plane being not exactly perpendicular to the principal ray Computer vision: models, learning and inference. © 2011 Simon J. D. Prince 10

Position and orientation of camera • Position w=(u, v, w)T of point in the world is generally not expressed in the frame of reference of the camera. • Transform using 3 D transformation or Point in frame of reference of camera Point in frame of reference of world Computer vision: models, learning and inference. © 2011 Simon J. D. Prince 11

Solving the problems • None of these problems can be solved in closed form • Can apply non-linear optimization to find best solution but slow and prone to local minima • Solution – convert to a new representation (homogeoneous coordinates) where we can solve in closed form. • Caution! We are not solving the true problem – finding global minimum of wrong problem. But can use as starting point for non-linear optimization of true problem Computer vision: models, learning and inference. © 2011 Simon J. D. Prince 20

Exterior orientation Start with camera equation in homogeneous coordinates Pre-multiply both sides by inverse of camera calibration matrix Computer vision: models, learning and inference. © 2011 Simon J. D. Prince 29

Exterior orientation Minimum direction problem of the form Find minimum of subject to To solve, compute the SVD set to the last column of . . , and then Computer vision: models, learning and inference. © 2011 Simon J. D. Prince 32

Exterior orientation Now we extract the values of and from . Problem: the scale is arbitrary and the rows and columns of the rotation matrix may not be orthogonal. Solution: compute SVD choose. and then Use the ratio between the rotation matrix before and after to rescale Use these estimates for start of non-linear optimisation. Computer vision: models, learning and inference. © 2011 Simon J. D. Prince 33

Calibration One approach is to alternately • Optimize extrinsic parameters for fixed intrinsic • Optimize intrinsic parameters for fixed extrinsic Then use non-linear optimization. Not very efficient w. r. t estimating simultaneously all the parameters Computer vision: models, learning and inference. © 2011 Simon J. D. Prince 36

Intrinsic parameters The function is linear w. r. t. intrinsic parameters. Can be written in form Now solve least squares problem Computer vision: models, learning and inference. © 2011 Simon J. D. Prince 38

Reconstruction Last equations gives Substitute back into first two equations Re-arranging get two linear equations for [u, v, w] Solve using >1 cameras and then use non-linear optimization Computer vision: models, learning and inference. © 2011 Simon J. D. Prince 42

Reconstruction • The method requires correspondences between points in different images – SIFT descriptors • The method requires the extrinsic and intrinsic camera parameters – Camera calibration – Uncalibrated cameras reconstruct the scene up to a projective transformation – If the same camera acquired all the images • We can estimate its intrinsic parameters • Reconstruction up to a scaling factor Computer vision: models, learning and inference. © 2011 Simon J. D. Prince 43

Depth from structured light • Camera and projectors are calibrated – The projector illuminates the scene one pixel at a time – The corresponding pixel is found in the image provided by the camera – Easy to reconstruct but too slow • Structured light algorithm – A series of horizontal and vertical stripe patterns are projected onto the scene – Top half light and bottom half dark – Capture the image and its inverse – If the difference is positive the pixels belong to the top half – Refine by using a second pattern of four horizontal stripes – Determine if the pixel is at the top or bottom half of the region determined by the first pattern etc. – More sophisticated coding schemes may be used Computer vision: models, learning and inference. © 2011 Simon J. D. Prince 46

Shape from silhouette • Adding cameras constraints the shape to the 3 D intersection of the ray bundles • The shape estimate is refined Computer vision: models, learning and inference. © 2011 Simon J. D. Prince 49

Shape from silhouette (augmented reality) • The actor is captured from 15 cameras in a green screen studio • A novel view of the actor is generated and superimposed onto the scene • The view is carefully selected to match the direction of the camera to give the impression that the actor is stable part of the scene Computer vision: models, learning and inference. © 2011 Simon J. D. Prince 50

Shape from silhouette (novel view generation) • In the new view, compute the ray passing through the pixel • Search along the ray for a foreground point – It should be a foreground point in all the real images • The point on the visual hull is now known • Project the point onto the closest real image and copy its color Computer vision: models, learning and inference. © 2011 Simon J. D. Prince 51

Conclusion • Pinhole camera model is a non-linear function who takes points in 3 D world and finds where they map to in image • Parameterized by intrinsic and extrinsic matrices • Difficult to estimate intrinsic/extrinsic/depth because non-linear • Use homogeneous coordinates where we can get closed from solutions (initial solns only) Computer vision: models, learning and inference. © 2011 Simon J. D. Prince 52

Related presentations