Relationship between stereo imaging system design and depth

Relationship between stereo imaging system design and depth resolution Andreas F. Koschan, Ph. D Mongi A. Abidi, Ph. D Imaging, Robotics, & Intelligent Systems Department of Electrical Engineering and Computer Science The University of Tennessee

2 Motivation • A camera’s sensor area, not only its pixel count, determines the quality of an image in digital photography. • Cameras with larger sensors typically produce better image fidelity than cameras with smaller sensors even when the pixel count is the same [Grotta 2012]. • The goal of a stereo imaging system is to compute threedimensional depth from two-dimensional images. • It is also true for stereo imaging systems that the camera’s sensor area and not only its pixel count determines the quality/precision of a 3 D measurement. However, bigger is not always better. • The following study investigates to what degree the sensor size, the pixel size, and the stereo system setup impact the accuracy of 3 D measurements. [Grotta 2012] S. Wiener Grotta and D. Grotta, “Pixels: Size Matters, ” IEEE Spectrum, May 2012.

3 Geometry of a stereo system q q P x. L x. R f f XR XL OL z. L +b 2 Z +b 2 OR z. R X • 2 cameras with identical focal length f • b is the base distance between their optical centers OL and OR • the angle between the two optical axes is 2 q • the coordinate system XYZ is defined such that the Z-axis exactly halves the angle between the z. L-axis and the z. R -axis.

4 Geometry of a stereo system Coordinate system XYZ can be transformed into coordinate system through a clockwise rotation by the angle q about the Y-axis and a translation by b/2 to the left: (1) Analogously the coordinate system XYZ can be transformed into the coordinate system : (2)

5 Geometry of a stereo system A point P = (X, Y, Z) in 3 D space is assumed to be represented in the image planes of both cameras at points and. Assuming central projection, the relation between the points is defined by (3) The Euclidian distance between the corresponding pixels p. L=(x. L, y. L) in the left image and p. R=(x. R, y. R) in the right image is indicated as disparity.

6 Standard stereo geometry A special case exists for the camera geometry if the cameras are shifted parallel to each other and the focal planes are coplanar. In this case, and the calculations of the depth value can be further simplified. For two corresponding points (x. L, y. L) and (x. R, y. R) in the left and right image respectively, the depth value Z is determined by (4)

7 Analysis of depth resolutions in stereo systems The absolute error in stereo depth calculations has to be kept as small as possible to attain useful 3 D descriptions of a scene. The maximum resolution of the depth values [Ver, Tor 86] can be estimated by (5) where e. D is the error in depth estimate, ZL is the distance between the camera and the object, f is the focal length, b is the base distance between the cameras, and Dx is the camera resolution (size of a sensor element). [Ver, Tor 86] A. Verri and V. Torre, “Absolute depth estimate in stereopsis, ” J. Opt. Soc. Am. A, Vol. 3, No. 3, pp. 297 -299, 1986.

8 Analysis of depth resolutions in stereo systems (example). • Two identical cameras with 16. 2 Megapixels • Sensor size is 23. 6 mm x 15. 6 mm (resolution of 4928 x 3264 pixels) • A single sensor element (sel) is 4. 7 mm x 4. 7 mm. • f = 50 mm and b = 20 cm • Consider 3 objects being Z 1 = 1. 0 m, Z 2 = 0. 75 m, and Z 3 = 0. 5 m away from the cameras. The depth resolution at position 1 can be estimated as

9 Analysis of depth resolutions in stereo systems Recall equation (5) The depth resolution of a stereo system depends a) b) c) d) on the distance between the camera and the object, the effective focal length of the lens, the base distance between the cameras, and the size of a sel.

10 Analysis of depth resolutions in stereo systems The depth resolution of a stereo system can be improved by the following procedures: • Reducing of the distance between object and camera, • Usage of a lens with a larger focal length, • Expansion of the distance between the optical centers of the cameras (expansion of the base distance), • Usage of a sensor with higher resolution, • Sampling of the image function with subpixel resolution (superresolution).

11 Analysis of depth resolutions in stereo systems We study the following 5 types of cameras in a stereo system: • Camera 1: Security camera (704 x 480 pixels NTSC, sensor 11. 96 x 8. 16 mm, sel 17 x 17 mm) • Camera 2: Consumer SLR camera (16. 2 MP, 4928 x 3264 pixels, sensor 23. 6 x 15. 6 mm, sel 4. 7 x 4. 7 mm) • Camera 3: Point-and-shoot camera (16 MP, 4608 x 3456 pixels, sensor 6. 17 x 4. 55 mm, sel 1. 3 x 1. 3 mm) • Camera 4: HD-SLR camera (36. 3 MP, 7360 x 4912 pixels, sensor 36 x 23. 9 mm, sel 4. 8 x 4. 8 mm) • Camera 5: Special HR camera (80 MP, 10328 x 7760 pixels, sensor 53. 7 x 40. 4 mm, sel 5. 2 x 5. 2 mm)

12 Analysis of depth resolutions in stereo systems The maximum possible distance between two cameras can be calculated if Equation (4) is solved for b (6) The maximum possible disparity value for in a stereo image pair results from the maximum number of pixels in the row of the image and the size of the sensor elements.

13 Analysis of depth resolutions in stereo systems max b distance between camera and object 0. 5 m 0. 75 m 1. 0 m 1. 5 m 5 m 107. 6 mm 167. 4 mm 227. 2 mm 346. 8 mm 1188. 0 mm 2380. 0 mm 212. 4 mm 330. 4 mm 448. 4 mm 684. 4 mm 2336. 4 mm 4696. 4 mm 55. 5 mm 86. 3 mm 117. 2 mm 178. 9 mm 610. 8 mm 1227. 8 mm HD-SLR 324. 0 mm 504. 0 mm 684. 0 mm 1044. 0 mm 3564. 0 mm 7164. 0 mm Special HR 483. 3 mm 751. 8 mm 1020. 3 mm 1557. 3 mm 5316. 3 mm 10686. 3 mm Security camera Consumer SLR Point-and shoot Tab. 1: Maximum possible distance between two cameras of the same camera type when they are equipped with a 50 mm lens.

14 Analysis of depth resolutions in stereo systems distance between camera and object 0. 5 m 0. 75 m 1. 0 m 1. 5 m 5 m 10 m 1. 58 mm 2. 28 mm 2. 99 mm 4. 41 mm 14. 37 mm 28. 59 mm 0. 22 mm 0. 32 mm 0. 42 mm 0. 62 mm 2. 01 mm 4. 00 mm 0. 23 mm 0. 34 mm 0. 44 mm 0. 65 mm 2. 12 mm 4. 23 mm HD-SLR 0. 15 mm 0. 21 mm 0. 28 mm 0. 41 mm 1. 35 mm 2. 68 mm Special HR 0. 11 mm 0. 15 mm 0. 20 mm 0. 30 mm 0. 98 mm 1. 94 mm Security camera Consumer SLR Point-and shoot Tab. 2: Maximum depth resolution obtained when positioning two cameras of the same type with f = 50 mm at a baseline corresponding to half of the maximum possible baseline.