Using Vanishing Points to Correct Camera Rotation Andrew

Using Vanishing Points to Correct Camera Rotation Andrew C. Gallagher Eastman Kodak Company andrew. gallagher@kodak. com May 9, 2005 Andrew C. Gallagher CRV 2005 1

Problem • An unintentionally tilted camera can negatively affect image appearance. • Caused by lightweight cameras that are difficult to hold level. • People prefer images where the horizon is level. • Human can see as little at 1 o tilt. May 9, 2005 Andrew C. Gallagher CRV 2005 2

Solution • Vanishing point location can be used to detect and correct image tilt resulting from camera rotation. • A vanishing point is the image of a world line at infinity. • Vanishing point location is useful for: – computing focal length [Kanatani] – finding principal point [Caprile et al. ] – determining camera parameters and rotation matrix[Cipolla et al. ] May 9, 2005 Andrew C. Gallagher CRV 2005 3

Vanishing Points • Parallel scene lines meet at a vanishing point in the image. Vertical Line Vanishing Point Horizontal Line Vanishing Point May 9, 2005 Andrew C. Gallagher CRV 2005 4

The Camera Model • The camera model describes the projection of 3 D world to 2 D camera plane. • K is a 3 x 3 matrix of the internal camera parameters. world • R is a 3 x 3 matrix describing coordinate frame the rotation from the world to the camera frame. camera coordinate • T is a 3 x 1 matrix describing frame translation between the world and camera coordinate frame. • Assume no skew, square pixels. The vanishing points of world directions are: May 9, 2005 Andrew C. Gallagher CRV 2005 5

The Rotation Matrix • R is any matrix in the special orthogonal group SO(3). • In practice the camera positions used by typical consumers follow a fairly predictable nonlinear distribution. • This distribution is then used to find where vanishing points will occur. May 9, 2005 Andrew C. Gallagher CRV 2005 6

Camera Position Analysis World rotation by q about the Y-axis Default Position World rotation by f about the X-axis May 9, 2005 World rotation about the Z-axis TILTED IMAGE Andrew C. Gallagher CRV 2005 7

Camera Position Analysis Rotation about both X- and Yaxes • This position model encompasses all “preferred” camera positions. • The vanishing point associated with vertical world direction (Y-axis) is constrained to fall on the image y-axis. • The horizon is parallel to image x-axis. Location of Vy Location of Vx or Vz May 9, 2005 Andrew C. Gallagher CRV 2005 8

Camera Position Analysis Additional rotation about the Z-axis • The original rotation matrix is multiplied by a rotation about the Z-axis. • The new vanishing points are simply rotated by the same amount! • In essence, the rotation of the camera from the ideal position is equivalent to the rotation of the Location vanishing points. of Vy Location of Vx or Vz May 9, 2005 Andrew C. Gallagher CRV 2005 9

Ground Truth Analysis • 357 vanishing points were manually labeled to compare with expected distribution. • 160 vertical (Vy) vanishing points 197 horizontal (Vx or Vz) vanishing points. • The match is visually good. Location of Vy Location of Vx or Vz EXPECTED DISTRIBUTION May 9, 2005 MEASURED DISTRIBUTION Andrew C. Gallagher CRV 2005 10

Vanishing Point Classification • The vertical and horizontal vanishing point distributions are well-separated. • A classifier can be used to identify vertical vanishing points. • The camera rotation is found from the vertical vanishing point. Vertical vanishing point classifier • On ground truth, only 2 vanishing points (0. 6%) are misclassified. May 9, 2005 Andrew C. Gallagher CRV 2005 11

The Tilt Correction Algorithm • Find vanishing points • Identify vertical vanishing points ^ • Compute camera rotation angle b from the vertical vanishing point • Compute correction angle bc according to table: • Rotate image • The rotated image can be shown to be equivalent to capturing with a camera having no component of rotation about the Z-axis. May 9, 2005 Andrew C. Gallagher CRV 2005 12

Vanishing Point Detection • Initial work by Barnard 1983. • Line Segment Detection – Lines are found by calculating local gradients, then clustering, or by using Hough transform. • Line Intersection Computation – Intersections of the lines are found. Line intersections are possible locations of a vanishing point. • Maximum Detection – A detected vanishing point is hypothesized to be at a location of many line intersections. May 9, 2005 Andrew C. Gallagher CRV 2005 13

Vanishing Point Detection Lines associated with 1 st VP. Original Image Plot of all line segment Intersections (Higher probabilities are red). Detected Vanishing Points Detected Line Segments May 9, 2005 Lines associated with 2 nd VP. Andrew C. Gallagher CRV 2005 14

Original May 9, 2005 Algorithm Results Lines associated with vertical VP Andrew C. Gallagher CRV 2005 Corrected 15

Original May 9, 2005 Algorithm Results Lines associated with vertical VP Andrew C. Gallagher CRV 2005 Corrected 16

Conclusions • Rotation of the camera about the principal axis moves the vertical vanishing point from the image y-axis. • This novel algorithm corrects a tilted image by detecting the vertical vanishing point, and determining the magnitude of camera rotation. May 9, 2005 Andrew C. Gallagher CRV 2005 17