Single View Metrology A Criminisi I Reid and

Single View Metrology A. Criminisi, I. Reid, and A. Zisserman University of Oxford IJCV Nov 2000 Presentation by Kenton Anderson CMPT 820 March 24, 2005 Single View Metrology

Overview n n n Introduction Geometry Algebraic Representation Uncertainty Analysis Applications Conclusions March 24, 2005 Single View Metrology 2

Problem n Is it possible to extract 3 D geometric information from single images? YES n n How? Why? March 24, 2005 Single View Metrology 3

Background 3 D 2 D Optical centre Photograph Flat Drawing Painting image Projec. AC P triavcam e hinite G ere ca r, to, , metr De Lsin Lce aria w y prstipvoefr. G s. Opepeotcicm tsiveetry March 24, 2005 Single View Metrology Real AReconstructed mental Real or imaginary object model 3 D object model 4

Introduction n 3 D affine measurements may be measured from a single perspective image March 24, 2005 Single View Metrology 5

Introduction 1. 2. 3. n Measurements of the distance between any of the planes Measurements on these planes Determine the camera’s position Results are sufficient for a partial or complete 3 D reconstruction of the observed scene March 24, 2005 Single View Metrology 6

Sample La Flagellazione di Cristo March 24, 2005 Single View Metrology 7

Geometry n n Overview Measurements between parallel lines Measurements on parallel planes Determining the camera position March 24, 2005 Single View Metrology 8

Geometry Overview n n n Possible to obtain geometric interpretations for key features in a scene Derive how 3 D affine measurements may be extracted from the image Use results to analyze and/or model the scene March 24, 2005 Single View Metrology 9

Assumptions n n Assume that images are obtained by perspective projection Assume that, from the image, a: • vanishing line of a reference plane • vanishing point of another reference direction may be determined from the image March 24, 2005 Single View Metrology 10

Geometric Cues n Vanishing Line ℓ • Projection of the line at infinity of the reference plane into the image March 24, 2005 Single View Metrology 11

Geometric Cues n Vanishing Point(s) v • A point at infinity in the reference direction • Reference direction is NOT parallel to reference plane • Also known as the vertical vanishing point March 24, 2005 Single View Metrology 12

Geometric Cues Vertical vanishing point (at infinity) Vanishing line Vanishing point March 24, 2005 Vanishing point Single View Metrology 13

Generic Algorithm 1) 2) Edge detection and straight line fitting to obtain the set of straight edge segments SA Repeat a) Randomly select two segments s 1, s 2 € SA and intersect them to give point p b) The support set Sp is the set of straight edges in SA going through point p 3) 4) Set the dominant vanishing point as the point p with the largest support Sp Remove all edges in Sp from SA and repeat step 2 March 24, 2005 Single View Metrology 14

Automatic estimation of vanishing points and lines RANSAC algorithm March 24, 2005 15 Candidate vanishing point Single View Metrology

Automatic estimation of vanishing points and lines March 24, 2005 Single View Metrology 16

Measurements between Parallel Lines n Wish to measure the distance between two parallel planes, in the reference direction • The aim is to compute the height of an object relative to a reference March 24, 2005 Single View Metrology 17

Cross Ratio n n n Point b on plane ∏ correspond to point t on plane ∏’ Aligned to vanishing point v Point i is the intersection with the vanishing line March 24, 2005 Single View Metrology 18

Cross Ratio n The cross ratio is between the points provides an affine length ratio • The value of the cross ratio determines a ratio of distances between planes in the world n Thus, if we know the length for an object in the scene, we can use it as a reference to calculate the length of other objects March 24, 2005 Single View Metrology 19

Estimating Height • The distance || tr – br || is known • Used to estimate the height of the man in the scene March 24, 2005 Single View Metrology 20

Measurements on Parallel Planes n If the reference plane is affine calibrated, then from the image measurements the following can be computed: i. Ratios of lengths of parallel line segments on the plane ii. Ratios of areas on the plane March 24, 2005 Single View Metrology 21

Parallel Line Segments • Basis points are manually selected and measured in the real world • Using ratios of lengths, the size of the windows are calculated March 24, 2005 Single View Metrology 22

Planar Homology n n Using the same principals, affine measurements can be made on two separate planes, so long as the planes are parallel to each other A map in the world between parallel planes induces a map between images of points on the two planes March 24, 2005 Single View Metrology 23

Homology Mapping between Parallel Planes n A point X on plane ∏ is mapped into the point X’ on ∏’ by a parallel projection March 24, 2005 Single View Metrology 24

Planar Homology n Points in one plane are mapped into the corresponding points in the other plane as follows: X’ = HX where (in homogeneous coordinates): • X is an image point • X’ is its corresponding point • H is the 3 x 3 matrix representing the homography transformation March 24, 2005 Single View Metrology 25

Measurements on Parallel Planes n This means that we can compare measurements made on two separate planes by mapping between the planes in the reference direction via the homology March 24, 2005 Single View Metrology 26

Parallel Line Segments lying on two Parallel Planes March 24, 2005 Single View Metrology 27

Camera Position n Using the techniques we developed in the previous sections, we can: • Determine the distance of the camera from the scene • Determine the height of the camera relative to the reference plane March 24, 2005 Single View Metrology 28

Camera Distance from Scene n n In Measurements between Parallel Lines, distances between planes are computed as a ratio relative to the camera’s distance from the reference plane Thus we can compute the camera’s distance from a particular frame knowing a single reference distance March 24, 2005 Single View Metrology 29

Camera Position Relative to Reference Plane n The location of the camera relative to the reference plane is the back-projection of the vanishing point onto the reference plane March 24, 2005 Single View Metrology 30

Algebraic Representation n n Overview Measurements between parallel lines Measurements on parallel planes Determining the camera position March 24, 2005 Single View Metrology 31

Overview n Algebraic approach offers many advantages (over direct geometry): 1. Avoid potential problems with ordering for the cross ratio 2. Minimal and over-constrained configurations can be dealt with uniformly 3. Unifies the different types of measurements 4. Are able to develop an uncertainty analysis March 24, 2005 Single View Metrology 32

Coordinate Systems n Define an affine coordinate system XYZ in space • Origin lies on reference plane • X, Y axes span the reference plane • Z axis is the reference direction n Define image coordinate system xy • y in the vertical direction • x in the horizontal direction March 24, 2005 Single View Metrology 33

Coordinate Systems March 24, 2005 Single View Metrology 34

Projection Matrix n n If X’ is a point in world space, it is projected to an image point x’ in image space via a 3 x 4 projection matrix P x’ = PX’ = [ p 1 p 2 p 3 p 4 ]X’ where x’ and X’ are homogeneous vectors: x’ = (x, y , w) and X’ = (X, Y, Z, W) March 24, 2005 Single View Metrology 35

Vanishing Points n n Denote the vanishing points for the X, Y and Z directions as v. X, v. Y, and v By inspection, the first 3 columns of matrix P are the vanishing points: • p 1 = v X • p 2 = v Y • p 3 = v n Origin of the world coordinate system is p 4 March 24, 2005 Single View Metrology 36

Vanishing Line n Furthermore, v. X and v. Y are on the vanishing line l • Choosing these points fixes the X and Y affine coordinate axes • Denote them as l 1 , l 2 where li · l = 0 T T T n Note: • Columns 1, 2 and 4 make up the reference plane to image homography matrix H March 24, 2005 Single View Metrology 37

Projection Matrix Redux n o = p 4 = l/|| l || = l^ • o is the Origin of the coordinate system Thus, the parametrization of P is: T n T P = [ l 1 l 2 αv l^ ] α is the affine scale factor March 24, 2005 Single View Metrology 38

Measurements between Parallel Lines n n The aim is to compute the height of an object relative to a reference Height is measured in the Z direction March 24, 2005 Single View Metrology 39

Measurements between Parallel Lines n n Base point B on the reference plane Top point T in the scene b = n(Xp 1 + Yp 2 + p 4) t = m(Xp 1 + Yp 2 + Zp 3 + p 4) n and m are unknown scale factors March 24, 2005 Single View Metrology 40

Affine Scale Factor n n α is known, then we can obtain Z If Z is known, we can compute α, If removing affine ambiguity March 24, 2005 Single View Metrology 41

Representation March 24, 2005 Single View Metrology 42

Measurements on Parallel Planes n n Projection matrix P from the world to the image is defined with respect to a coordinate frame on the reference plane The translation from the reference plane to another plane along the reference direction can be parametrized into a new projection matrix P’ March 24, 2005 Single View Metrology 43

Plane to Image Homographies P = [ l 1 l 2 αv l^ ] T T P’ = [ l 1 l 2 αv αZv + l^ ] where Z is the distance between the planes March 24, 2005 Single View Metrology 44

Plane to Image Homographies n Homographies can be extracted: H = [ p 1 p 2 αZv + l^ ] H’ = [ p 1 p 2 l^ ] n Then H” = H’H-1 maps points from the reference plane to the second plane, and so defines the homology March 24, 2005 Single View Metrology 45

Generic Algorithm 1. 2. Given an image of a planar surface estimate the image-to-world homography matrix H Repeat a) Select two points x 1 and x 2 on the image plane b) Back-project each image point into the world plane using H to obtain the two world points X 1 and X 2 c) Compute the Euclidean distance dist(X 1, X 2) i. March 24, 2005 dist(A, B) = || A – B || Single View Metrology 46

Camera Position n Camera position C = (Xc, Yc, Zc, Wc) PC = 0 Implies: n Using Cramer’s Rule: n n March 24, 2005 Single View Metrology 47

Camera In Scene March 24, 2005 Single View Metrology 48

Uncertainty Analysis n Errors arise from the finite accuracy of the feature detection and extraction • ie- edge detection, point specifications n Uncertainty analysis attempts to quantify this error March 24, 2005 Single View Metrology 49

Uncertainty Analysis n Uncertainty in • Projection matrix P • Top point t • Base point b • Location of vanishing line l • Affine scale factor n α As the number of reference distances increases, so the uncertainty decreases March 24, 2005 Single View Metrology 50

Illustration n Ellipses are user specified t and b are then aligned to the vertical vanishing point Alignment constraint v · (t X b) = 0 March 24, 2005 Single View Metrology 51

Applications n Forensic Science • Height of suspect n Virtual Modeling • 3 D reconstruction of a scene n Art History • Modeling paintings March 24, 2005 Single View Metrology 52

Forensic Science March 24, 2005 Single View Metrology 53

Virtual Modeling March 24, 2005 Single View Metrology 54

Art History March 24, 2005 Single View Metrology 55

Conclusions n n n Affine structure of 3 D space may be partially recovered from perspective images Measurements between and on parallel planes can be determined Practical applications can be derived March 24, 2005 Single View Metrology 56