Wide Angle Camera Calibration and Realtime Distortion Correction

Wide Angle Camera Calibration and Real-time Distortion Correction Vitaliy Orekhov Imaging, Robotics, & Intelligent Systems Laboratory The University of Tennessee April 28, 2006 Slide

Outline • Task 1 – Math 472 – Modern Transforms • Task 2 – ECE 571 – Pattern Classifications • Task 3 – ECE 574 –Test, Evaluate, and Transfer Camera Calibration Code to C++ • Real time distortion correction • GUI to perform real time distortion correction on an image sequence • Task 4 – ECE 671 –Survey on Wide Angle Camera Calibration and Image Correction • Survey paper outline • Compare a recent journal article to C. Broaddus thesis 9/14/2021 Slide 2/23

Wide Angle Camera Calibration • Easier to map local information for visual search, navigation, or detection. • It is a necessary step in 3 D computer vision to extract metric information from 2 D images. • Before the camera can be used for precise computer vision applications, the camera needs to be characterized. – How does a point in 3 D world coordinates get projected onto the camera imaging NIKON 14 MM f/2. 8 D ED Ultra Wide Angle plane? AF Nikkor Lens www. amazon. com 9/14/2021 Slide 3/23

Survey on Wide Angle Camera Calibration and Image Correction • Challenging to go into details and compare the results. – Lack of standardized notation – Various approaches to evaluate accuracy – Distortion models [Salvi 02] J. Salvi, X. Armangue, and J. Batlle, “A comparative review of camera calibrating methods with accuracy evaluation, ” Patter Recogntion, pp. 1617 -1635, July 2002. • Many ways to classify wide angle camera calibration. • Using geometric objects / Test range calibration • Non-metric calibration / Using geometric properties • Point correspondence • Straight lines • Object of revolution • Self-calibration • 3 D reference objects • 2 D reference objects • 1 D line based calibration • Self-Calibration 9/14/2021 Slide 4/23 • Linear calibration / Closed form solution / Non-iterative • Non-linear calibration / Iterative • Two step methods • Manual methods • Semi-Automatic methods • Automatic methods • Offline calibration • Using 2 D and 3 D object • Online calibration • Plumb line method • Rigidity assumption

Survey Outline I. Introduction a. b. c. d. II. Motivation Background Problem definition Paper outline Wide Angle Camera Models/ Omnidirectional Systems a. b. Modeling lenses Modeling distortion i. ii. Distortion models Finding distortion center III. Classification Methods IV. Calibration Methods with Distortion Consideration a. b. c. Test range calibration Non-metric calibration Self-calibration V. Radial Distortion Calibration a. b. c. Estimating the center of distortion Using lines Point correspondence VII. Concluding Remarks 9/14/2021 Slide 5/23

[Hartley 05] R. I. Hartley and S. B. Kang, “Parameter-free Radial Distortion Correction with Centre of Distortion Estimation, ” Computer Vision, 2005. ICCV 2005. Tenth IEEE International Conference on , vol. 2, no. pp. 1834 - 1841, 17 -20 Oct. 2005 [Broaddus 05] C. Broaddus, “Universal Geometrci Camera Calibration with Statistical Model Selection, ” Masters Thesis, Department of Electrical Engineering, The Unverstity of Tennessee, Knoxville, TN 2005. • Method to simultaneously calibrate the radial distortion function of a camera Radial and other internal calibration parameters. distortion • Method relies on the use of a planar calibration pattern. • Radial distortion is found in a parameter. Radial position free way, not relying on any particular Graph generated from 19 images with distortion model. resolution of 640 x 480. [Hartley 05] • Method relies on accurate estimation of center of distortion. 9/14/2021 Slide 6/23

Distortion Center • • Center of distortion is found by computing the fundamental matrix relating the known coordinates of points on a planer calibration pattern and the extracted points from the image. Radial distortion forces ideal points to be expanded towards or away from the center of distortion. -distorted image point -undistorted image point - known coordinates on planar calibration pattern Epipole in one view is the camera center of the other view. Center of radial expansion is the epipole. 9/14/2021 Slide 7/23

Image Homographies • Estimate homography which maps calibration pattern to the captured image. • Coordinates of image are transformed so that the center of radial distortion is at the origin. • 2 assumptions: – Distortion is radially symmetric – Radial distance of distorted points from distortion center is a monotonic function of the distance of undistorted points. 9/14/2021 Slide 8/23

Image Homographies Radial distortion Radial position Graph generated from 19 images with resolution of 640 x 480. [Hartley 05] • Monotonic smooth curve. • Linear Least-Squares Problem for finding the total squared variation of function. • With some simplifications image homographies can be uniquely estimated by a linear algorithm. • Can also use assumption of local linearity. 9/14/2021 Slide 9/23

Distortion Estimation • Curve can be used to correct distortion in any other image captured with the same camera. • Approximate the curve by a distortion function. Radial distortion Radial position Graph generated from 19 images with resolution of 640 x 480. [Hartley 05] 9/14/2021 Slide 10/23

Complete Calibration • With homographies between the calibration pattern and images known, true undistorted projection of each calibration point can be made onto the image. • Given the homographies for at least three images, Zhang’s calibration method for a projective camera is used to find internal camera parameters. • Non-iterative rapid method. Could be used to initialize a bundle-adjustment algorithm or as a stand alone algorithm for camera calibration not requiring extreme precision. • Could be improved with iterative refinement at different steps of the process. 9/14/2021 Slide 11/23

Comparison [Broaddus 05] [Hartley 05] • Center of distortion is assumed to be at the principle point. Where the optical axis from the camera center intersects the image plane. • Decentering distortion is estimated together with radial distortion. • Center of distortion is estimated using calibration pattern separate from the rest of the calibration process. • Only radial distortion is included. • Homography matrix is estimated by using normalized DLT algorithm to find initial values and then refined with non-linear optimization method and bundle adjustment. • With the center of distortion known, only the last row of the homography matrix needs to be estimated. Linear or non-linear methods can be used to find the values. • Iterative with computing time dependent on stop criteria. • Estimates are optimized using bundle adjustment and Levenberg-Marquardt. • Non-iterative and fast method to find distortion curve and image homographies. • Iterative refinement could improve results at different stages of the procedure. • Results can be used to initialize bundleadjustment algorithm or as a stand alone algorithm for applications not requiring extreme precision. • Statistical criteria method used to automatically select complexity of distortion model. • Distortion curve needs to be approximated by a distortion function. • Method can be expanded to not use calibration patterns but then it is very sensitive to noise. 9/14/2021 Slide 12/23

Correcting Distortion on an image sequence in real-time IQeye 3 Network Camera • 1. 8 M pixels/sec JPEG • 1/2" 1288 x 968 progressive scan CMOS digital imager • 50 fps (frames per second) at 160 by 120 • 19 fps at 320 by 240 • 5. 4 fps at 640 by 480 • 3. 6 fps at 800 by 600 • 1. 3 fps at 1288 by 968 www. oksolar. com YV 2. 2 x 1. 4 A-SA 2 Lens • Fish-eye vari-focal lens with the horizontal field angle of 185 -94 degrees (when used on 1/3 cameras) www. fujinon. de 9/14/2021 Slide 13/23

Image Sequence Correction Diagram 9/14/2021 Slide 14/23

Camera Calibration • Capturing sequence of images with calibration pattern. • Extracting coordinates using Harris corner detections. • Use MATLAB calibration code written by Chris Broaddus. – Extract camera matrix – Distortion parameters • Radial and tangential distortion – Single polynomial approximation used to model the lens projections. 9/14/2021 Slide 15/23

Image Correction • Code written in C++ • Creates a look up table based on the camera matrix and distortion coefficients. • Code works with most commonly used distortion models: • Perspective • Stereographic • Equidistant • Orthogonal • Logarithmic • Radial polynomial • Look up table is used to correct each image captured from the camera in real-time. 9/14/2021 Slide 16/23 Image size: 620 x 620 185 deg. fov Image size: 620 x 620 80 deg. fov

GUI 9/14/2021 Slide 17/23

Wide Angle Distortion Correction • • Image size 312 x 312 Frame rate from camera: 14 fps. Frame rate with distortion correction: 10 fps. Smallest angle-of-view setting (about 80 deg. ) 9/14/2021 Slide 18/23

Fish-eye Distortion Correction • • Image size 312 x 312 Frame rate from camera: 14 fps. Frame rate with distortion correction: 10 fps. Widest angle-of-view setting (185 deg. ) 9/14/2021 Slide 19/23

Wide Angle Distortion Correction • • Original Image size 312 x 312 Restored Image size 428 x 428 Frame rate: 14 fps. Smallest angle-of-view setting (about 80 deg. ) 9/14/2021 Slide 20/23

Processing Time Comparison Clock speed Time (ms) 312 x 312 pixels Time (ms) 620 x 620 pixels 1. 4 GHz 72 (13. 8 fps) 200 (5 fps) 1. 4 GHz 100 (10 fps) 333 (3 fps) Implementation Pentium 4 with lookup table (displaying results without saving *) Pentium 4 with lookup table (saving results to computer *) Clock speed Time (ms) 600 x 600 pixels Pentium 4 with lookup table 1. 7 GHz 760 (1. 3 fps) Pentium 4 with real time calculation 1. 7 GHz 780 (1. 3 fps) Various FPGA implementations 10 MHz – 34 MHz 11 -360 (91 -2. 8 fps) Implementation 1* 1* D. Eadie, F. Shevlin, A, Nisbet “Correction of Geometric Image Distortion Using FPGA’s” SPIE Proceeding Opto-Ireland, Vol. 4877, pp. 28 -37, March 2003. 9/14/2021 Slide 21/23

Summary Task 3: Test, Evaluate, and Transfer Camera Calibration Code to C++ Accomplished: • Calibrated IQEye 3 camera for different field of views and image sizes. • Generated real time distortion correction of an image sequence. • Have a working GUI to control and display the real time distortion correction of an image sequence. Future Objectives: • Work on optimization step in camera calibration for C++ code. • Create a GUI for complete camera calibration in C++. Task 4: Survey on Wide Angle Camera Calibration and Image Correction Accomplished: • Have been exploring calibration methods with considering radial distortion. 9/14/2021 Slide 22/23 Future Objectives: • Still have more articles to examine. • Finalize survey paper. • Reexamine and revise content previously written.

Thank you Suggestions/Comments/Questions 9/14/2021 Slide 23/23