3 D Computer Vision CSc 83020 Camera Calibration

3 -D Computer Vision CSc 83020 Camera Calibration CSc 83020 3 -D Computer Vision – Ioannis Stamos

Camera Calibration n Problem: Estimate camera’s extrinsic & intrinsic parameters. Method: Use image(s) of known scene. Tools: Geometric camera models. n SVD and constrained least-squares. n Line extraction methods. n CSc 83020 3 -D Computer Vision – Ioannis Stamos

Camera Calibration Perspective Equations Feature Extraction From Sebastian Thrun and Jana Kosecka CSc 83020 3 -D Computer Vision – Ioannis Stamos

Perspective Projection, Remember? O X Z x f From Sebastian Thrun and Jana Kosecka CSc 83020 3 -D Computer Vision – Ioannis Stamos

Intrinsic Camera Parameters n n Determine the intrinsic parameters of a camera (with lens) What are Intrinsic Parameters? (can you name 7? ) From Sebastian Thrun and Jana Kosecka CSc 83020 3 -D Computer Vision – Ioannis Stamos

Intrinsic Parameters O X Z Image center(ox, oy) f From Sebastian Thrun and Jana Kosecka CSc 83020 3 -D Computer Vision – Ioannis Stamos

Intrinsic Camera Parameters n Intrinsic Parameters: n Focal Length f n Pixel size sx , sy n Image center ox , oy n n (Nonlinear radial distortion coefficients k 1 , k 2…) Calibration = Determine the intrinsic parameters of a camera From Sebastian Thrun and Jana Kosecka CSc 83020 3 -D Computer Vision – Ioannis Stamos

Coordinate Frames Camera Coordinate Frame Zc Pixel Coordinates Yc Xc Extrinsic Parameters Zw Intrinsic Parameters Image Coordinate Frame Yw World Coordinate Frame Xw CSc 83020 3 -D Computer Vision – Ioannis Stamos

Why Calibrate? Image point Scene ray Image Calibration: relates points in the image to rays in the scene CSc 83020 3 -D Computer Vision – Ioannis Stamos

Why Calibrate? Image point Scene ray x y z Image x y Calibration: relates points in the image to rays in the scene CSc 83020 3 -D Computer Vision – Ioannis Stamos

Example Calibration Pattern From Sebastian Thrun and Jana Kosecka Calibration Pattern: Object with features of known size/geometry CSc 83020 3 -D Computer Vision – Ioannis Stamos

Harris Corner Detector From Sebastian Thrun and Jana Kosecka CSc 83020 3 -D Computer Vision – Ioannis Stamos

Perspective Camera r r’=(X, Y, Z) r’ (X, Y, Z) Center of Projection r =(x, y, z) x=f * X/Z y=f * Y/Z f: effective focal length: z=f CSc 83020 3 -D Computer Vision – Ioannis Stamos distance of image plane from O. r/f=r’/Z

Extrinsic Parameters T Pc=R(Pw-T) CSc 83020 3 -D followed Computer Visionby – Ioannis Stamos Translation rotation

Extrinsic Parameters (2 nd formulation) R same as before Pc=R Pw +T T different CSc 83020 followed 3 -D Computer Vision – Ioannis Stamos Rotation by translation

The Rotation Matrix R * RT= RT* R = I => R-1= RT Orthonormal Matrix Degrees of freedom? I= 100 010 001 CSc 83020 3 -D Computer Vision – Ioannis Stamos

Perspective Camera Model n Step 1: Transform into camera coordinates n Step 2: Transform into image coordinates From Sebastian Thrun and Jana Kosecka CSc 83020 3 -D Computer Vision – Ioannis Stamos

Intrinsic Parameters CSc 83020 3 -D Computer Vision – Ioannis Stamos

Image and Camera Frames Zcamera Ycamera (xim, yim) Xcamera (ox, oy) Yimage CSc 83020 3 -D Computer Vision – Ioannis Stamos Ximage

Geometric Model x= y= Xc. T Zc. T Yc. T Zc. T 3 D Point in Camera Coordinate Frame • Transformation from Image • Transformation from to Camera Frame. World to Camera Frame. (ox, oy, sx, sy) • Perspective projection • No distortion! (f, R, T) Point in Camera Frame CSc 83020 3 -D Computer Vision – Ioannis Stamos

Camera Calibration: Issues n Which parameters need to be estimated. Focal length, image center, aspect ratio n Radial distortions n n What kind of accuracy is needed. n n Application dependent What kind of calibration object is used. One plane, many planes n Complicated three dimensional object n CSc 83020 3 -D Computer Vision – Ioannis Stamos

Camera Calibration object Extracted features CSc 83020 3 -D Computer Vision – Ioannis Stamos

Camera Calibration Extract centers of circles CSc 83020 3 -D Computer Vision – Ioannis Stamos

Basic Equations CSc 83020 3 -D Computer Vision – Ioannis Stamos

Basic Equations In the remaining slides x means xim and y means yim CSc 83020 3 -D Computer Vision – Ioannis Stamos

Basic Equations CSc 83020 3 -D Computer Vision – Ioannis Stamos

Basic Equations Extrinsic Parameters 1) Rotation matrix R (3 x 3) 2) Translation vector T (3 x 1) Intrinsic Parameters 1) fx=f/sx, length in effective horizontal pixel size units. 2) α=sy/sx, aspect ratio. 3) (ox, oy), image center coordinates. 4) Radial distortion coefficients. Total number of parameters (excluding distortion): ? CSc 83020 3 -D Computer Vision – Ioannis Stamos

Basic Equations 1) Assume that image center is known. 2) Solve for the remaining parameters. 3) Use N image points and their corresponding N world points CSc 83020 3 -D Computer Vision – Ioannis Stamos

Basic Equations (1) 1) Assume that image center is known. 2) Solve for the remaining parameters. 3) Use N image points and their corresponding N world points Here x means xim - ox and y means yim - oy CSc 83020 3 -D Computer Vision – Ioannis Stamos

Basic Equations (2) 1) Assume that image center is known. 2) Solve for the remaining parameters. 3) Use N image points and their corresponding N world points CSc 83020 3 -D Computer Vision – Ioannis Stamos

Basic Equations (3) CSc 83020 3 -D Computer Vision – Ioannis Stamos

Basic Equations (3) How would we solve this system? CSc 83020 3 -D Computer Vision – Ioannis Stamos

Basic Equations (3) How would we solve this system? Rank of matrix A? Solution up to a scale factor. CSc 83020 3 -D Computer Vision – Ioannis Stamos

Singular Value Decomposition Appendix A. 6 (Trucco) A: m x n U: m x m, columns orthogonal unit vectors. V: n x n , -//D: m x n , diagonal. The diagonal elements σi are the singular values σ1>= σ2>= … >= σn >= 0 CSc 83020 3 -D Computer Vision – Ioannis Stamos

Singular Value Decomposition Appendix A. 6 (Trucco) 1. 2. 3. 4. 5. 6. Square A non-singular iff σi != 0 For square A C=σ1/σN is the condition number For rectangular A # of non-zero σi is the rank For square non-singular A: For square A, pseudoinverse: Singular values of A = square roots of eigenvalues of and 7. Columns of U, V Eigenvectors of 8. Frobenius norm of a matrix CSc 83020 3 -D Computer Vision – Ioannis Stamos

Singular Value Decomposition Appendix A. 6 (Trucco) If rank(A)=n-1 (7 in our case) then the solution is the eigenvector which corresponds to the ONLY zero eigenvalue. Solution up to a scale factor. CSc 83020 3 -D Computer Vision – Ioannis Stamos

Solving for v (3) How would we solve this system: SVD. Solution: Uknown scale factor γ=? Aspect ratio α=? CSc 83020 3 -D Computer Vision – Ioannis Stamos

Solving for Tz and fx? CSc 83020 3 -D Computer Vision – Ioannis Stamos

Solving for Tz and fx? How would we solve this system? CSc 83020 3 -D Computer Vision – Ioannis Stamos

Solving for Tz and fx? How would we solve this system? Solution in the least squares sense. CSc 83020 3 -D Computer Vision – Ioannis Stamos

Camera Center CSc 83020 3 -D Computer Vision – Ioannis Stamos

Camera Models (linear versions) x= 3 D Point in World Coordinate Frame y= • Transformation from Image • Transformation from World to Camera Frame. • Perspective projection (ox, oy, sx, sy) (f, R, T) • No distortion! Point in Camera Frame CSc 83020 3 -D Computer Vision – Ioannis Stamos

Camera Models (linear versions) Elegant decomposition. No distortion! Homogeneous Coordinates Measured Pixel World Point (xim, CScyim) (Xw, Yw, Zw) 83020 3 -D Computer Vision – Ioannis Stamos ? CSc 83020 3 -D Computer Vision – Ioannis Stamos

Camera Calibration – Other method Extracted features Step 1: Estimate P Step 2: Decompose P into internal and external parameters R, T, C CSc 83020 3 -D Computer Vision – Ioannis Stamos

Camera Calibration: Step 1 Extracted features Each point (x, y) gives us two equations CSc 83020 3 -D Computer Vision – Ioannis Stamos

Camera Calibration: Step 1 Extracted features Each corner (x, y) gives us two equations CSc 83020 3 -D Computer Vision – Ioannis Stamos

Camera Calibration: Step 1 2 n Extracted features n points gives us 2 n equations CSc 83020 3 -D Computer Vision – Ioannis Stamos

Camera Calibration: Step 1 2 n Extracted features We need to solve In the presence of noise we need to solve The solution is given by the eigenvector with the smallest eigenvalue of CSc 83020 3 -D Computer Vision – Ioannis Stamos

Camera Calibration: Step 1 The result can be improved through non-linear minimization. Extracted features CSc 83020 3 -D Computer Vision – Ioannis Stamos

Camera Calibration: Step 1 The result can be improved through non-linear minimization. Extracted features Minimize the distance between the predicted and detected features. CSc 83020 3 -D Computer Vision – Ioannis Stamos