Telephoto and Zoom Lens Calibration Vitaliy Orekhov Imaging

Telephoto and Zoom Lens Calibration Vitaliy Orekhov Imaging, Robotics, & Intelligent Systems Laboratory The University of Tennessee December 1, 2006 Slide

Outline • IRIS Demos • Tasks – Telephoto Lens Calibration – Lens Parameter Optimization in C++ – Thesis/Publication • Telephoto and Zoom Lens Calibration Results – – – Previous Observations 50 mm Lens 18 -70 mm Lens 70 -210 mm Lens Overview of Literature on Zoom Lens Calibration • Conclusions and Future Work 12/5/2020 Slide 2/28

IRIS Demos • Homecoming Demo • URPR Demo for DOE • Engineer’s Day Demo URPR Demo 12/5/2020 Slide 3/28 Engineer’s Day Demo

IRIS Tasks • Telephoto Lens Calibration • Lens Parameter Optimization in C++ • Thesis/Publication – Telephoto and zoom lens calibration – Capture images with Rodenstock 50 mm lens – Put together a portable image recording setup 12/5/2020 Slide 4/28

Rodenstock 50 mm Lens Sony DXC-930 CCD Color Video Camera CCD Video Camera Module 12/5/2020 Slide 5/28 Sony Exware. HAD SSC-DC 393 CCD Color Video Camera IQEye 3 Camera

Rodenstock 50 mm Lens 1/3 Type Sensor CCD Board 12/5/2020 Slide 6/28

IRIS Tasks • Telephoto Lens Calibration • Lens Parameter Optimization in C++ • Thesis/Publication – Final Step of Complete Camera Calibration – Levenberg-Marquardt nonlinear least squares algorithm • Numerical Recipes Implementation – Code is completely written – Still need to finish debugging and run test to compare performance to MATLAB 12/5/2020 Slide 7/28

IRIS Tasks • Telephoto Lens Calibration • Lens Parameter Optimization in C++ • Thesis/Publication 1. 2. Introduction Survey and Background Theory – – 3. Camera Models and Calibration Wide Angle and Omnidirectinal Camera Calibration Telephoto and Zoom Lens Calibration Complete Camera Calibration Experimental Results and Comparison – – Automatic Model Complexity Selection Validation Telephoto and Zoom Lens Calibration Synthetic Data Real Data 4. 5. Improved Complete Camera Calibration Method Practical Implementations – – 6. 12/5/2020 Slide 8/28 Real Time Video Distortion Correction Calibration GUI Summary and Conclusion

Outline • IRIS Demos • Tasks – Telephoto Lens Calibration – Lens Parameter Optimization in C++ – Thesis/Publication • Telephoto and Zoom Lens Calibration Results – – – Previous Observations 50 mm Lens 18 -70 mm Lens 70 -210 mm Lens Overview of Literature on Zoom Lens Calibration • Conclusions and Future Work 12/5/2020 Slide 9/28

Telephoto and Zoom Lens Calibration Results • Motivation: Normal to Telephoto Lens Calibration • Previous results were inconclusive • Go back to the basics (50 mm Lens) (1 -28 mm, 2 -100 mm, 3 -200 mm, 4 -300 mm) 12/5/2020 Slide 10/28

Nikon 50 mm Lens • www. nikon. com Lens Specifications – Nikon Nikkor AF/MF 50 mm F/1: 1. 4 -16 – Manual aperture f 2. 8 – Varied focus settings (focus settings: 1 -0. 7 m, 2 -1. 0 m, 3 -1. 2 m, 4 -1. 5 m, 5 -2. 1 m, 6 -3. 0 m, x-image center) 12/5/2020 Slide 11/28

• Nikon 18 -70 mm Lens Specifications – Nikon Nikkor AF-S 18 -70 mm 1: 3. 5 -4. 5 – Manual aperture – Fixed focus settings – Varied focal length www. nikon. com Focal Length Estimation with IRIS Method Actual Focal Length 18 mm 50 mm 70 mm Measured Focal Length [mm] 16. 7 46. 5 60. 0 * % Error 7% 7% 14% 12/5/2020 Slide 12/28 (1 -18 mm, 2 -50 mm, 3 -70 mm, x-image center)

• Nikon 18 -70 mm Lens Specifications – Nikon Nikkor AF-S 18 -70 mm 1: 3. 5 -4. 5 – Manual aperture – Fixed focus settings – Varied focal length www. nikon. com Focal Length Estimation with Open. CV (1 -18 mm, 2 -50 mm, 3 -70 mm, x-image center) Actual Focal Length 18 mm 50 mm 70 mm Measured Focal Length [mm] 17. 2 46. 0 61. 0 * % Error 4% 8% 13% 12/5/2020 Slide 13/28

Nikon 70 -210 mm Lens • Lens Specifications – Nikon Nikkor AF/MF 70 -210 mm F/4 -5. 6 – Focal length 70 -210 mm – Lens construction: 12 elements in 9 groups – Manual aperture – Varied focus settings (1. 5 m, 2 m, 3 m) – Varied focal length (70 mm, 90 mm, 110 mm, 135 mm, 210 mm) www. amazon. com 12/5/2020 Slide 14/28

Varifocal Error Measurements Error (IRIS) Error (Open. CV) 3 m Focus MSE [pixels] STDY [pixels] STDX [pixels] 210 mm 0. 9072 0. 6543 0. 5830 2. 12 2. 02 135 mm 0. 2731 0. 3464 0. 3389 6662 7525 110 mm 0. 1222 0. 2447 0. 2137 0. 28 90 mm 0. 0994 0. 2277 0. 1719 754. 5 669. 6 70 mm 0. 0450 0. 1320 0. 1470 0. 17 0. 20 210 mm - - - 135 mm 0. 7690 0. 5324 0. 6048 3. 15 3. 18 110 mm 0. 3957 0. 3716 0. 4472 2. 24 2. 09 90 mm 0. 1943 0. 2722 0. 3018 47. 30 9. 26 70 mm 0. 1418 0. 2524 0. 2541 1. 17 1. 18 210 mm - - - 135 mm - - - 110 mm 0. 9611 0. 5934 0. 6853 0. 67 0. 60 90 mm 0. 3709 0. 3798 0. 4183 0. 52 0. 45 70 mm 0. 2031 0. 3203 0. 2735 0. 43 0. 38 2 m Focus 1. 5 m Focus 12/5/2020 Slide 15/28

Focal Length Measurements Focal Length (IRIS) Focal Length (Open. CV) Fx mm Fy mm 210 mm 203. 6 203. 8 635. 0 570 135 mm 136. 6 1315 1383 110 mm 107. 7 107. 1 112. 3 112. 7 90 mm 88. 3 88. 1 123. 3 126. 6 70 mm 83. 0 82. 7 76. 5 76. 3 210 mm - - 135 mm 138. 4 138. 5 523. 3 475. 1 110 mm 106. 0 343. 6 294. 4 90 mm 91. 8 234. 8 225. 7 70 mm 81. 5 81. 4 284. 9 484. 2 210 mm - - 135 mm - - 110 mm 119. 3 119. 8 128. 1 90 mm 97. 5 97. 4 102. 0 70 mm 81. 6 81. 5 mm (pixels) 3 m Focus 2 m Focus 1. 5 m Focus 12/5/2020 Slide 16/28

Principal Point Location Principal Point (IRIS) Principal Point (GML) Distortion Center X Y X Y 1504 1000 210 mm 229 1119 600 1163 1767 636 135 mm 747 1212 -14669 25367 1454 1366 110 mm 1557 724 1534 1579 1428 1120 90 mm 1501 876 37700 -26103 1884 1177 70 mm 1324 901 1355 1144 1184 564 210 mm - - - 135 mm 1319 992 2279 571 1364 934 110 mm 1460 915 -1007 1068 1310 1018 90 mm 1441 994 -20 810 1249 972 70 mm 1512 898 1757 -11334 1152 1081 210 mm - - - 135 mm - - - 110 mm 1140 1022 1481 1367 1705 769 90 mm 1366 1000 1374 1283 1450 898 70 mm 1331 969 1353 1227 1573 936 Image Center 3 m Focus 2 m Focus 1. 5 m Focus 12/5/2020 Slide 17/28

Nikon 70 -210 mm Lens Principal Point Location f=210 mm (3 m) 12/5/2020 Slide 18/28 f=70 mm (1. 5 m)

Nikon 70 -210 mm Lens Principal Point Location 12/5/2020 Slide 19/28

Distortion Center [Hartley 05] R. I. Hartley and S. B. Kang, “Parameter-free Radial Distortion Correction with Centre of Distortion Estimation, ” Computer Vision, 2005. ICCV 2005. Tenth IEEE International Conference on , vol. 2, no. pp. 18341841, 17 -20 Oct. 2005 • • Center of distortion is found by computing the fundamental matrix relating the known coordinates of points on a planer calibration pattern and the extracted points from the image. Radial distortion forces ideal points to be expanded towards or away from the center of distortion. X Y 1504 1000 210 mm 1767 636 135 mm 1454 1366 110 mm 1428 1120 90 mm 1884 1177 70 mm 1184 564 210 mm - - 135 mm 1364 934 110 mm 1310 1018 90 mm 1249 972 70 mm 1152 1081 210 mm - - 135 mm - - 110 mm 1705 769 90 mm 1450 898 70 mm 1573 936 Image Center 3 m Focus 2 m Focus 1. 5 m Focus Epipole in one view is the camera center of the other view. Center of radial expansion is the epipole. 12/5/2020 Slide 20/28

Distortion Center X Y 1504 1000 210 mm 1767 636 135 mm 1454 1366 110 mm 1428 1120 90 mm 1884 1177 70 mm 1184 564 210 mm - - 135 mm 1364 934 110 mm 1310 1018 90 mm 1249 972 70 mm 1152 1081 210 mm - - 135 mm - - 110 mm 1705 769 90 mm 1450 898 70 mm 1573 936 Image Center 3 m Focus 2 m Focus 1. 5 m Focus 12/5/2020 Slide 21/28

Distortion Center f=210 mm at 3 meters from target 12/5/2020 Slide 22/28

Survey of Zoom Lens Calibration Methods Willson, R. G. , Modeling and Calibration of Automated Zoom Lenses, Ph. D. Thesis, Carnegie Mellon University, CMU-RI-TR-94 -03, 1994 Mengxiang Li; Lavest, J. -M. , "Some aspects of zoom lens camera calibration, " Pattern Analysis and Machine Intelligence, IEEE Transactions on , vol. 18, no. 11 pp. 1105 -1110, Nov 1996 Collins, R. T. ; Tsin, Y. , "Calibration of an outdoor active camera system, " Computer Vision and Pattern Recognition, 1999. IEEE Computer Society Conference on. , vol. 1, no. pp. -534 Vol. 1, 1999 Yong-Sheng Chen; Shen-Wen Shih; Yi-Ping Hung; Chiou-Shann Fuh, "Camera calibration with a motorized zoom lens, " Pattern Recognition, 2000. Proceedings. 15 th International Conference on , vol. 4, no. pp. 495 -498 vol. 4, 2000 Ahmed, M. T. ; Farag, A. A. , "A neural optimization framework for zoom lens camera calibration , " Computer Vision and Pattern Recognition, 2000. Proceedings. IEEE Conference on , vol. 1, no. pp. 403 -409 vol. 1, 2000 Thirthala, S. R. ; Sinha, S. N. ; Pollefeys, M. , "Calibration of pan-tilt-zoom (PTZ) cameras and omni-directional cameras, " Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on , vol. 2, no. pp. 1198 vol. 2 -, 20 -25 June 2005 12/5/2020 Slide 23/28

Mengxiang Li; Lavest, J. -M. , "Some aspects of zoom lens camera calibration, " Pattern Analysis and Machine Intelligence, IEEE Transactions on , vol. 18, no. 11 pp. 1105 -1110, Nov 1996 Parameters are calibrated at discrete lens positions (focus and zoom) based on Tsai Camera Calibration method • Determine principal point by zooming – – • Center of expansion moves about 0. 5 of a pixel on average Fixed principal point Focal length changed as a smooth function – – Focus effect is not so important for short distances but increases greatly for long distances Polynomials of order five were used to model the surface www. cas. kth. se/~hic/active-vision. pdf KTH Head-Eye System Focus: 0. 8 m – inf. Lens: f=12. 5 mm to 75 mm Residual errors, i. e. , fitness between model and data for different zoom and focus settings. Least squares technique is applied to minimize measurement error in images. 12/5/2020 Slide 24/28 Calibrated results of focal length (fx) for the zoom lens mounted on the KTH head-eye system.

Collins, R. T. ; Tsin, Y. , "Calibration of an outdoor active camera system, " Computer Vision and Pattern Recognition, 1999. IEEE Computer Society Conference on. , vol. 1, no. pp. -534 Vol. 1, 1999 Parameters are calibrated at discrete lens positions using Tsai Camera Model • First the center of zoom expansion is recovered – – • • 205 images for each camera Image center can vary by as much as 40 pixels Later calibration by pure rotation determines the remaining intrinsic parameters (f, s, k) Extrinsic calibration (R) is performed last SONY / EVI 370 www. broadcaststore. com Color Camera- 1/3 inch NTSC HAD 12 x optical zoom, auto focus, auto iris lens High resolution: 768 x 494 effective pixels, 460 TV lines Lens: f=5. 4 mm to 64. 8 mm, F 1. 8 to 2. 7 Figure 2. Calibration by zooming. Left: image center vs. zoom. Right: Detail of image center for one camera. Image center (384, 247). 12/5/2020 Slide 25/28

Ahmed, M. T. ; Farag, A. A. , "A neural optimization framework for zoom lens camera calibration , " Computer Vision and Pattern Recognition, 2000. Proceedings. IEEE Conference on , vol. 1, no. pp. 403 -409 vol. 1, 2000 • • Since optimal polynomial fails to follow complex variations in the model parameters they use neural network approach Parameters are calibrated at discrete lens positions based on Tsai Camera Calibration method except they decouple distortion parameter estimation Use chessboard calibration pattern Focal length range from 11 mm to 110 mm (10 x) Focal Length Yong-Sheng Chen; Shen-Wen Shih; Yi-Ping Hung; Chiou-Shann Fuh, "Camera calibration with a motorized zoom lens, " Pattern Recognition, 2000. Proceedings. 15 th International Conference on , vol. 4, no. pp. 495 -498 vol. 4, 2000 • • • 12/5/2020 Slide 26/28 Parameters are calibrated at discrete lens positions based on Weng (92) Camera Calibration method Use calibration pattern with circular control points Different calibration patterns are used depending on the field of view of a particular lens setting Fujinon 12. 5 -75 mm manual lens Focus range from 1 m to ∞ 3 servo motors used to control lens settings

Future Work • Telephoto and Zoom Lens Calibration – Perform experiments on 70 -300 mm lens – Calculate the center of zoom expansion • Lens Parameter Optimization in C++ – Debug and compare optimization results of parameters – Continue work on user friendly calibration GUI • Thesis/Publication – Finish adding content of telephoto and zoom lens calibration 12/5/2020 Slide 27/28

Thank you Suggestions/Comments/Questions 12/5/2020 Slide 28/28