Vision Video 3 D Vision and Virtual Reality

Vision, Video 3 D Vision and Virtual Reality CSC 59866 CD Fall 2004 Lecture 12 Camera Models Zhigang Zhu, NAC 8/203 A http: //www-cs. engr. ccny. cuny. edu/~zhu/ Capstone 2004/Capstone_Sequence 2004. html

Vision, Video and Virtual Reality n Closely Related Disciplines l l l n Image processing – image to mage Pattern recognition – image to classes Photogrammetry – obtaining accurate measurements from images What is 3 -D ( three dimensional) Vision? l l l n 3 D Vision Motivation: making computers see (the 3 D world as humans do) Computer Vision: 2 D images to 3 D structure Applications : robotics / VR /Image-based rendering/ 3 D video Lectures on 3 -D Vision Fundamentals l l Camera Geometric Model (1 lecture – this class) Camera Calibration (1 lecture) Stereo (2 lectures) Motion (2 lectures)

Vision, Video and Virtual Reality n Geometric Projection of a Camera l l l n Pinhole camera model Perspective projection Weak-Perspective Projection Camera Parameters l l Intrinsic Parameters: define mapping from 3 D to 2 D Extrinsic parameters: define viewpoint and viewing direction n n Basic Vector and Matrix Operations, Rotation Camera Models Revisited l Linear Version of the Projection Transformation Equation n n Lecture Outline Perspective Camera Model Weak-Perspective Camera Model Affine Camera Model for Planes Summary

Vision, Video and Virtual Reality n Camera Geometric Models l l l n Lecture Assumptions Knowledge about 2 D and 3 D geometric transformations Linear algebra (vector, matrix) This lecture is only about geometry Goal Build up relation between 2 D images and 3 D scenes -3 D Graphics (rendering): from 3 D to 2 D -3 D Vision (stereo and motion): from 2 D to 3 D -Calibration: Determing the parameters for mapping

Vision, Video and Virtual Reality Image Formation

Vision, Video Image Formation and Virtual Reality Light (Energy) Source Surface Imaging Plane Camera: Spec & Pose 3 D Scene Pinhole Lens World Optics Sensor Signal 2 D Image

Vision, Video and Virtual Reality n n n Pinhole Camera Model Pin-hole is the basis for most graphics and vision l Derived from physical construction of early cameras l Mathematics is very straightforward 3 D World projected to 2 D Image l Image inverted, size reduced l Image is a 2 D plane: No direct depth information Perspective projection l f called the focal length of the lens l given image size, change f will change FOV and figure sizes

Vision, Video Focal Length, FOV and Virtual Reality n Consider case with object on the optical axis: Image plane f z viewpoint n n n Optical axis: the direction of imaging Image plane: a plane perpendicular to the optical axis Center of Projection (pinhole), focal point, viewpoint, nodal point Focal length: distance from focal point to the image plane FOV : Field of View – viewing angles in horizontal and vertical directions

Vision, Video Focal Length, FOV and Virtual Reality n Consider case with object on the optical axis: Image plane z f Out of view n n n Optical axis: the direction of imaging Image plane: a plane perpendicular to the optical axis Center of Projection (pinhole), focal point, viewpoint, , nodal point Focal length: distance from focal point to the image plane FOV : Field of View – viewing angles in horizontal and vertical directions Increasing f will enlarge figures, but decrease FOV

Vision, Video Equivalent Geometry and Virtual Reality n Consider case with object on the optical axis: f z n More convenient with upright image: z f Projection plane z = f n Equivalent mathematically

Vision, Video Perspective Projection and Virtual Reality n Compute the image coordinates of p in terms of the world (camera) coordinates of P. y Y p(x, y) x P(X, Y, Z) X 0 Z Z=f n n n Origin of camera at center of projection Z axis along optical axis Image Plane at Z = f; x // X and y//Y

Vision, Video and Virtual Reality n Reverse Projection Given a center of projection and image coordinates of a point, it is not possible to recover the 3 D depth of the point from a single image. In general, at least two images of the same point taken from two different locations are required to recover depth.

Vision, Video and Virtual Reality Pinhole camera image Amsterdam : what do you see in this picture? lstraight line lsize lparallelism/angle lshape of planes ldepth Photo by Robert Kosara, robert@kosara. net http: //www. kosara. net/gallery/pinholeamsterdam/pic 01. html

Vision, Video and Virtual Reality Pinhole camera image Amsterdam üstraight line lsize lparallelism/angle lshape of planes ldepth Photo by Robert Kosara, robert@kosara. net http: //www. kosara. net/gallery/pinholeamsterdam/pic 01. html

Vision, Video and Virtual Reality Pinhole camera image Amsterdam üstraight line ´size lparallelism/angle lshape of planes ldepth Photo by Robert Kosara, robert@kosara. net http: //www. kosara. net/gallery/pinholeamsterdam/pic 01. html

Vision, Video and Virtual Reality Pinhole camera image Amsterdam üstraight line ´size ´parallelism/angle lshape of planes ldepth Photo by Robert Kosara, robert@kosara. net http: //www. kosara. net/gallery/pinholeamsterdam/pic 01. html

Vision, Video and Virtual Reality Pinhole camera image Amsterdam üstraight line ´size ´parallelism/angle ´shape lshape of planes ldepth Photo by Robert Kosara, robert@kosara. net http: //www. kosara. net/gallery/pinholeamsterdam/pic 01. html

Vision, Video and Virtual Reality Pinhole camera image Amsterdam üstraight line ´size ´parallelism/angle ´shape lshape ü of planes parallel to image ldepth Photo by Robert Kosara, robert@kosara. net http: //www. kosara. net/gallery/pinholeamsterdam/pic 01. html

Vision, Video and Virtual Reality Pinhole camera image Amsterdam: what do you see? üstraight line ´size ´parallelism/angle ´shape lshape ü of planes parallel to image l. Depth ? lstereo lmotion lsize lstructure … - We see spatial shapes rather than individual pixels - Knowledge: top-down vision belongs to human - Stereo &Motion most successful in 3 D CV & application - You can see it but you don't know how…

Vision, Video and Virtual Reality Yet other pinhole camera images Rabbit or Man? Markus Raetz, Metamorphose II, 1991 -92, cast iron, 15 1/4 x 12 inches Fine Art Center University Gallery, Sep 15 – Oct 26

Vision, Video and Virtual Reality Yet other pinhole camera images 2 D projections are not the “same” as the real object as we usually see everyday! Markus Raetz, Metamorphose II, 1991 -92, cast iron, 15 1/4 x 12 inches Fine Art Center University Gallery, Sep 15 – Oct 26

Vision, Video and Virtual Reality It’s real!

Vision, Video Weak Perspective Projection and Virtual Reality n Average depth Z is much larger than the relative distance between any two scene points measured along the optical axis y Y p(x, y) x P(X, Y, Z) X 0 Z Z=f n A sequence of two transformations l l n Orthographic projection : parallel rays Isotropic scaling : f/Z Linear Model l Preserve angles and shapes

Vision, Video Camera Parameters and Virtual Reality xim Image frame (xim, yim) yim n Coordinate Systems l l n Frame Grabber Frame coordinates (xim, yim) pixels Image coordinates (x, y) in mm Camera coordinates (X, Y, Z) World coordinates (Xw, Yw, Zw) Camera Parameters l l Y Pose / Camera X O y x Z p Object / World Zw P Pw Xw Yw Intrinsic Parameters (of the camera and the frame grabber): link the frame coordinates of an image point with its corresponding camera coordinates Extrinsic parameters: define the location and orientation of the camera coordinate system with respect to the world coordinate system

Vision, Video Intrinsic Parameters (I) and Virtual Reality y x Y X p (x, y, f) O n Size: (sx, sy) (0, 0) oy yim From frame to image l l l n Z Image center Directions of axes Pixel size Intrinsic Parameters l l l (ox , oy) : image center (in pixels) (sx , sy) : effective size of the pixel (in mm) f: focal length ox xim Pixel (xim, yim)

Vision, Video Intrinsic Parameters (II) and Virtual Reality (x, y) n n k 1 , k 2 (xd, yd) Lens Distortions Modeled as simple radial distortions l l r 2 = xd 2+yd 2 (xd , yd) distorted points k 1 , k 2: distortion coefficients A model with k 2 =0 is still accurate for a CCD sensor of 500 x 500 with ~5 pixels distortion on the outer boundary

Vision, Video Extrinsic Parameters and Virtual Reality xim (xim, yim) yim n Y X O y x Z p From World to Camera Zw P n Extrinsic Parameters l l Pw Xw T Yw A 3 -D translation vector, T, describing the relative locations of the origins of the two coordinate systems (what’s it? ) A 3 x 3 rotation matrix, R, an orthogonal matrix that brings the corresponding axes of the two systems onto each other

Vision, Video and Virtual Reality n A point as a 2 D/ 3 D vector l l l n Linear Algebra: Vector and Matrix T: Transpose Image point: 2 D vector Scene point: 3 D vector Translation: 3 D vector Vector Operations l Addition: n l Dot product ( a scalar): n l Translation of a 3 D vector a. b = |a||b|cosq Cross product (a vector) n Generates a new vector that is orthogonal to both of them a x b = (a 2 b 3 - a 3 b 2)i + (a 3 b 1 - a 1 b 3)j + (a 1 b 2 a 2 b 1)k

Vision, Video and Virtual Reality n Rotation: 3 x 3 matrix l Orthogonal : n l n Linear Algebra: Vector and Matrix 9 elements => 3+3 constraints (orthogonal) => 2+2 constraints (unit vectors) => 3 DOF ? (degrees of freedom) How to generate R from three angles? (next few slides) Matrix Operations l R Pw +T= ? - Points in the World are projected on three new axes (of the camera system) and translated to a new origin

Vision, Video and Virtual Reality n Rotation: from Angles to Matrix Rotation around the Axes l Result of three consecutive rotations around the coordinate axes Y O X Z g Zw Xw n Notes: l l l Only three rotations Every time around one axis Bring corresponding axes to each other n l Xw = X, Yw = Y, Zw = Z First step (e. g. ) Bring Xw to X b Yw a

Vision, Video and Virtual Reality Rotation: from Angles to Matrix g Zw Y X O n l l n Yw Rotation g around the Zw Axis Rotate in Xw. OYw plane Goal: Bring Xw to X But X is not in Xw. OYw Xw Yw X X in Xw. OZw ( Yw Xw. OZw) Yw in YOZ ( X YOZ) Next time rotation around Yw Z

Vision, Video and Virtual Reality Rotation: from Angles to Matrix g Zw Y Xw n Rotation g around the Zw Axis l l n Rotate in Xw. OYw plane so that Yw X X in Xw. OZw ( Yw Xw. OZw) Yw in YOZ ( X YOZ) Zw does not change X O Yw Z

Vision, Video and Virtual Reality Rotation: from Angles to Matrix Zw Y X Xw n Rotation b around the Yw Axis l l n O b Rotate in Xw. OZw plane so that Xw = X Zw in YOZ (& Yw in YOZ) Yw does not change Yw Z

Vision, Video and Virtual Reality Rotation: from Angles to Matrix Y X O n Rotation b around the Yw Axis l l n b Rotate in Xw. OZw plane so that Xw = X Zw in YOZ (& Yw in YOZ) Yw does not change Yw Xw Zw Z

Vision, Video and Virtual Reality Rotation: from Angles to Matrix Y a X O n Rotation a around the Xw(X) Axis l l n Rotate in Yw. OZw plane so that Yw = Y, Zw = Z (& Xw = X) Xw does not change Yw Xw Zw Z

Vision, Video and Virtual Reality Rotation: from Angles to Matrix Yw Y a X O Xw Zw Z n Rotation a around the Xw(X) Axis l l n Rotate in Yw. OZw plane so that Yw = Y, Zw = Z (& Xw = X) Xw does not change

Vision, Video and Virtual Reality Rotation: from Angles to Matrix Appendix A. 9 of the textbook n n Rotation around the Axes l Result of three consecutive rotations around the coordinate axes Y O X Z g Zw Notes: l l l Rotation directions The order of multiplications matters: g, b, a Same R, 6 different sets of a, b, g R Non-linear function of a, b, g R is orthogonal It’s easy to compute angles from R Xw b Yw a

Vision, Video and Virtual Reality Rotation- Axis and Angle Appendix A. 9 of the textbook n n According to Euler’s Theorem, any 3 D rotation can be described by a rotating angle, q, around an axis defined by an unit vector n = [n 1, n 2, n 3]T. Three degrees of freedom – why?

Vision, Video Linear and Virtual Reality. Version n World to Camera l l l n Camera to Image l l l n Camera: P = (X, Y, Z)T Image: p = (x, y)T Not linear equations Image to Frame l l n Camera: P = (X, Y, Z)T World: Pw = (Xw, Yw, Zw)T Transform: R, T Neglecting distortion Frame (xim, yim)T World to Frame l l (Xw, Yw, Zw)T -> (xim, yim)T Effective focal lengths n n fx = f/sx, fy=f/sy Three are not independent of Perspective Projection

Vision, Video and Virtual Reality n Projective Space l Add fourth coordinate n l l Only extrinsic parameters World to camera 3 x 3 Matrix Mint l l n x 1/x 3 =xim, x 2/x 3 =yim 3 x 4 Matrix Mext l n Pw = (Xw, Yw, Zw, 1)T Define (x 1, x 2, x 3)T such that n n Linear Matrix Equation of perspective projection Only intrinsic parameters Camera to frame Simple Matrix Product! Projective Matrix M= Mint. Mext l l l (Xw, Yw, Zw)T -> (xim, yim)T Linear Transform from projective space to projective plane M defined up to a scale factor – 11 independent entries

Vision, Video and Virtual Reality n Perspective Camera Model l Making some assumptions n n l n Known center: Ox = Oy = 0 Square pixel: Sx = Sy = 1 11 independent entries <-> 7 parameters Weak-Perspective Camera Model l Average Distance Z >> Range d. Z Define centroid vector Pw l 8 independent entries l n Three Camera Models Affine Camera Model l Mathematical Generalization of Weak-Pers Doesn’t correspond to physical camera But simple equation and appealing geometry n l Doesn’t preserve angle BUT parallelism 8 independent entries

Vision, Video and Virtual Reality n Planes are very common in the Man-Made World l n l l Zw=0 Pw =(Xw, Yw, 0, 1)T 3 D point -> 2 D point Projective Model of a Plane l n One more constraint for all points: Zw is a function of Xw and Yw Special case: Ground Plane l n Camera Models for a Plane 8 independent entries General Form ? l 8 independent entries

Vision, Video and Virtual Reality n A Plane in the World l n l l Zw=0 Pw =(Xw, Yw, 0, 1)T 3 D point -> 2 D point Projective Model of Zw=0 l n One more constraint for all points: Zw is a function of Xw and Yw Special case: Ground Plane l n Camera Models for a Plane 8 independent entries General Form ? l 8 independent entries

Vision, Video and Virtual Reality n A Plane in the World l n l l Zw=0 Pw =(Xw, Yw, 0, 1)T 3 D point -> 2 D point Projective Model of Zw=0 l n One more constraint for all points: Zw is a function of Xw and Yw Special case: Ground Plane l n Camera Models for a Plane 8 independent entries General Form ? l nz = 1 l 8 independent entries n 2 D (xim, yim) -> 3 D (Xw, Yw, Zw) ?

Vision, Video and Virtual Reality n Graphics /Rendering l From 3 D world to 2 D image n n l n Changing viewpoints and directions Changing focal length Fast rendering algorithms Vision / Reconstruction l From 2 D image to 3 D model n n l l n Applications and Issues Inverse problem Much harder / unsolved Robust algorithms for matching and parameter estimation Need to estimate camera parameters first Calibration l l Find intrinsic & extrinsic parameters Given image-world point pairs Probably a partially solved problem ? 11 independent entries n <-> 10 parameters: fx, fy, ox, oy, a, b, g, Tx, Ty, Tz

Vision, Video and Virtual Reality 3 D Reconstruction from Images (1) Panoramic texture map Flower Garden Sequence Vision: l. Camera l. Motion l 3 D Calibration Estimation reconstruction (2)panoramic depth map

Vision, Video Image-based and Virtual Reality 3 Dmodel of the FG sequence

Vision, Video and Virtual Reality Graphics: l. Virtual Camera l. Synthetic l. From motion 3 D to 2 D image Rendering from 3 D Model

Vision, Video and Virtual Reality n Geometric Projection of a Camera l l l n Pinhole camera model Perspective projection Weak-Perspective Projection Camera Parameters (10 or 11) l l n Camera Model Summary Intrinsic Parameters: f, ox, oy, sx, sy, k 1: 4 or 5 independent parameters Extrinsic parameters: R, T – 6 DOF (degrees of freedom) Linear Equations of Camera Models (without distortion) l l l General Projection Transformation Equation : 11 parameters Perspective Camera Model: 11 parameters Weak-Perspective Camera Model: 8 parameters Affine Camera Model: generalization of weak-perspective: 8 Projective transformation of planes: 8 parameters

Vision, Video Next and Virtual Reality n Determining the value of the extrinsic and intrinsic parameters of a camera Calibration (Ch. 6)