Computer Vision Calibration Marc Pollefeys COMP 256 Read

Computer Vision Calibration Marc Pollefeys COMP 256 Read F&P Chapter 2 Some slides/illustrations from Ponce, Hartley & Zisserman

Tentative class schedule Computer Vision Jan 16/18 - Introduction Jan 23/25 Cameras Radiometry Sources & Shadows Color Feb 6/8 Linear filters & edges Texture Feb 13/15 Multi-View Geometry Stereo Feb 20/22 Optical flow Project proposals Affine Sf. M Projective Sf. M Camera Calibration Segmentation Mar 13/15 Springbreak Mar 20/22 Fitting Prob. Segmentation Mar 27/29 Silhouettes and Photoconsistency Linear tracking Apr 3/5 Project Update Non-linear Tracking Apr 10/12 Object Recognition Apr 17/19 Range data Final project Jan 30/Feb 1 Feb 27/Mar 1 Mar 6/8 2 Apr 24/26

Computer Vision Previously Hierarchy of 3 D transformations Projective 15 dof Affine 12 dof Similarity 7 dof Euclidean 6 dof 3 Intersection and tangency Parallellism of planes, Volume ratios, centroids, The plane at infinity π∞ Angles, ratios of length The absolute conic Ω∞ Volume

Computer Vision Camera calibration Compute relation between pixels and rays in space ? 4

Computer Vision 5 Pinhole camera

Computer Vision Pinhole camera model non-homogeneous coordinates 6 linear projection in homogeneous coordinates!

Computer Vision 7 Pinhole camera model

Computer Vision Principal point offset principal point 8

Computer Vision Principal point offset calibration matrix 9

Computer Vision 11 Object motion

Computer Vision 12 Camera motion

Computer Vision 13 CCD camera

Computer Vision General projective camera 11 dof (5+3+3) non-singular intrinsic camera parameters extrinsic camera parameters 14

Computer Vision Camera matrix decomposition Finding the camera center (use SVD to find null-space) (for all X and λ C must be camera center) Finding the camera orientation and internal parameters (use RQ decomposition ~QR) (if only QR, invert) =( 15 Q ) = R -1 Q -1

Computer Vision 16 Affine cameras

Computer Vision Radial distortion • Due to spherical lenses (cheap) • Model: R R barrel dist. 17 straight lines are not straight anymore pincushion dist. http: //foto. hut. fi/opetus/260/luennot/11/atkinson_6 -11_radial_distortion_zoom_lenses. jpg

Computer Vision 18 Radial distortion example

Computer Vision Camera model Relation between pixels and rays in space ? 19

Computer Vision Projector model Relation between pixels and rays in space (dual of camera) ? (main geometric difference is vertical principal point offset to reduce keystone effect) 20

Computer Vision 21 Meydenbauer camera vertical lens shift to allow direct ortho-photographs

Computer Vision Action of projective camera on points and lines projection of point forward projection of line back-projection of line 22

Computer Vision Action of projective camera on conics and quadrics back-projection to cone projection of quadric 23

Computer Vision 24 Resectioning

Computer Vision Direct Linear Transform (DLT) rank-2 matrix 25

Computer Vision Direct Linear Transform (DLT) Minimal solution P has 11 dof, 2 independent eq. /points 5½ correspondences needed (say 6) Over-determined solution n 6 points minimize subject to constraint use SVD 26

Computer Vision Singular Value Decomposition Homogeneous least-squares 27

Computer Vision Degenerate configurations (i) Points lie on plane or single line passing through projection center (ii) Camera and points on a twisted cubic 28

Computer Vision Data normalization Scale data to values of order 1 1. 2. 29 move center of mass to origin scale to yield order 1 values

Computer Vision Line correspondences Extend DLT to lines (back-project line) (2 independent eq. ) 30

Computer Vision 31 Geometric error

Computer Vision Gold Standard algorithm Objective Given n≥ 6 2 D to 2 D point correspondences {Xi↔xi’}, determine the Maximum Likelyhood Estimation of P Algorithm (i) Linear solution: (a) Normalization: (b) DLT (ii) Minimization of geometric error: using the linear estimate as a starting point minimize the geometric error: ~ ~~ (iii) Denormalization: 32

Computer Vision Calibration example (i) Canny edge detection (ii) Straight line fitting to the detected edges (iii) Intersecting the lines to obtain the images corners (iv) typically precision <1/10 (v) (H&Z rule of thumb: 5 n constraints for n unknowns) 33

Computer Vision Errors in the image (standard case) Errors in the world Errors in the image and in the world 34

Computer Vision Restricted camera estimation Find best fit that satisfies • skew s is zero • pixels are square • principal point is known • complete camera matrix K is known Minimize geometric error impose constraint through parametrization Minimize algebraic error assume map from param q P=K[R|-RC], i. e. p=g(q) minimize ||Ag(q)|| 35

Computer Vision Restricted camera estimation Initialization • Use general DLT • Clamp values to desired values, e. g. s=0, x= y Note: can sometimes cause big jump in error Alternative initialization • Use general DLT • Impose soft constraints • 36 gradually increase weights

Computer Vision 37

Computer Vision 38 Image of absolute conic

Computer Vision A simple calibration device (i) (ii) (iii) (iv) 39 compute H for each square (corners (0, 0), (1, 0), (0, 1), (1, 1)) compute the imaged circular points H(1, ±i, 0)T fit a conic to 6 circular points compute K from w through cholesky factorization (≈ Zhang’s calibration method)

Computer Vision Some typical calibration algorithms Tsai calibration Zhangs calibration http: //research. microsoft. com/~zhang/calib/ Z. Zhang. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11): 1330 -1334, 2000. Z. Zhang. Flexible Camera Calibration By Viewing a Plane From Unknown Orientations. International Conference on Computer Vision (ICCV'99), Corfu, Greece, pages 666 -673, September 1999. 40 http: //www. vision. caltech. edu/bouguetj/calib_doc/

Computer Vision Sequential Sf. M • Initialize motion from two images • Initialize structure • For each additional view – Determine pose of camera – Refine and extend structure • Refine structure and motion 41

Computer Vision Initial projective camera motion • Choose P and P´compatible with F (reference plane; arbitrary) Reconstruction up to projective ambiguity Same for more views? different projective basis 42 • Initialize motion • Initialize structure • For each additional view • Determine pose of camera • Refine and extend structure • Refine structure and motion

Computer Vision Initializing projective structure • Reconstruct matches in projective frame by minimizing the reprojection error Non-iterative optimal solution 43 • Initialize motion • Initialize structure • For each additional view • Determine pose of camera • Refine and extend structure • Refine structure and motion

Computer Vision Projective pose estimation • Infere 2 D-3 D matches from 2 D-2 D matches • Compute pose from (RANSAC, 6 pts) X F x Inliers: 44 • Initialize motion • Initialize structure • For each additional view • Determine pose of camera • Refine and extend structure • Refine structure and motion

Computer Vision Refining and extending structure • Refining structure (Iterative linear) • Extending structure 2 -view triangulation • Initialize motion • Initialize structure • For each additional view 45 • Determine pose of camera • Refine and extend structure • Refine structure and motion

Computer Vision Refining structure and motion • use bundle adjustment Also model radial distortion to avoid bias! 46

Computer Vision Metric structure and motion use self-calibration (see next class) Note that a fundamental problem of the uncalibrated approach is that it fails if a purely planar scene is observed (in one or more views) (solution possible based on model selection) 47

Computer Vision 48 Dealing with dominant planes

Computer Vision PPPgric HHgric 49

Computer Vision Farmhouse 3 D models (note: reconstruction much larger than camera field-of-view) 50

Computer Vision 51 Application: video augmentation

Computer Vision 52 Next class: Segmentation Reading: Chapter 14