Design and Calibration of a MultiView TOF Sensor

Design and Calibration of a Multi-View TOF Sensor Fusion System Young Min Kim, Derek Chan, Christian Theobalt, Sebastian Thrun Stanford University

Outline § § § Motivation System Architecture Depth Sensor Characteristics System Calibration Results and Conclusion Young Min Kim 2 TOF-CV Workshop

Outline § § § Motivation System Architecture Depth Sensor Characteristics System Calibration Results and Conclusion Young Min Kim 3 TOF-CV Workshop

Motivation § Goal: Reconstruct geometry and texture of entire scene from minimum sensor data § State-of-the-art [Waschbuesch et al. 2006, 2007] – Stereo [Laurentini et al. 94, Kanade et al. 97, Matusik et al. 00, Matsuyama et al. 02, Wuermlin 02, Carranza et al. 03, Cheung et al. 03, Zitnick et al. 04, Bajcsy et al. 04, …] Young Min Kim 4 TOF-CV Workshop

Motivation § Limitation to stereo – Correspondence problem – Dependency on texture – Densely spaced cameras § Our idea: Build a system that can combine – TOF sensor – Video cameras Young Min Kim 5 TOF-CV Workshop

Outline § § § Introduction System Architecture Depth Sensor Characteristics System Calibration Results and Conclusion Young Min Kim 6 TOF-CV Workshop

Multi-view Sensor Fusion System Young Min Kim 7 TOF-CV Workshop

Multi-view Sensor Fusion System Point Grey Flea 2 • 1024 x 768 pixels • 30 Hz • High resolution • No depth Young Min Kim 8 TOF-CV Workshop

Multi-view Sensor Fusion System Swissranger SR 3000 Point Grey Flea 2 • Flash Ladar • 3 D geometry at 60 Hz • Resolution: 176 x 144 • No visual interference • Noisy low-resolution data • 1024 x 768 pixels • 30 Hz • High resolution • No depth Young Min Kim 9 TOF-CV Workshop

Multi-view Sensor Fusion System Main Contribution System Architecture And Calibration Swissranger SR 3000 Point Grey Flea 2 • Flash Ladar • 3 D geometry at 60 Hz • Resolution: 176 x 144 • No visual interference • Noisy low-resolution data • 1024 x 768 pixels • 30 Hz • High resolution • No depth Young Min Kim 10 TOF-CV Workshop

System Architecture 19 MHz 20 MHz 21 MHz Fire. Wire B … Fire. Wire A – Synchronization bus Young Min Kim 11 TOF-CV Workshop

System Architecture 19 MHz 20 MHz 21 MHz Fire. Wire B … Fire. Wire A – Synchronization bus – Synchronization § Hardware synch for video cameras § Initiate software for Swissrangers – Modulation frequency § Scales up to 4 Swissrangers Young Min Kim 12 TOF-CV Workshop

Example Data Young Min Kim 13 TOF-CV Workshop

Outline § § § Introduction System Architecture Depth Sensor Characteristics System Calibration Results and Conclusion Young Min Kim 14 TOF-CV Workshop

Related Work: TOF-Camera § Random noise characteristics of earlier models [Anderson et al. 05, Herbert et al. 92, …] § Systematic depth errors – PMD (photomic mixer device) – Lookup table for Swissranger [Lindner et al. 06, Lindner et al. 07] [Kahlmann et al. 92] § Detailed analysis on noise characteristics of single TOF sensors [Rapp 07] § Our work – practical model for systematic bias – calibration for multiple cameras Young Min Kim 15 TOF-CV Workshop

Depth Sensor Characteristics § Independent for each pixel (u, v) Image plane dm(u, v) Center of Projection § Measurement model dm(u, v)=dg(u, v)+dr(u, v)+ds(u, v) Measured Measurement uncertainty Ground-truth dr(u, v): random noise ds(u, v): systematic bias Young Min Kim 16 TOF-CV Workshop

Systematic Bias § Two components of systematic bias – Distance misalignment + rigid, directional – Influence of orientation, reflectance, and amplitude Young Min Kim 17 TOF-CV Workshop

Systematic Bias § Influence of orientation, reflectance, and amplitude ratio of normalized amplitude r=0. 3 ds(u, v) = f(r) d’s(u, v) ds(u, v) ≈ d’s(u, v) for r > 0. 3 Young Min Kim 18 TOF-CV Workshop

Outline § § § Introduction System Architecture Depth Sensor Characteristics System Calibration Results and Conclusion Young Min Kim 19 TOF-CV Workshop

Calibration § Video Cameras: – Standard calibration toolboxes – Intrinsic parameters + extrinsic parameters from checkerboard § Depth Cameras – No off-the-shelf solution – Camera provides XYZ / intensity – Use procedure based on optical method practicability Young Min Kim 20 TOF-CV Workshop

Depth Camera Calibration § Use calibration procedure for video cameras § Space 1: optical camera model – viewpoint/projection § Camera XYZ § Space 2: 3 D point cloud in sensor coordinates § Space 1 and Space 2 don’t match -> compensate Young Min Kim 21 TOF-CV Workshop

Depth Camera – Compensate Systematic Bias § Compensation: deform Space 2 to match Space 1 … N checkerboard positions spanning view frustum Intrinsics Young Min Kim ground truth point clouds in space 1 and measured point clouds in space 2 22 TOF-CV Workshop

Depth Camera – Compensate Systematic Bias § Compensation: deform Space 2 to match Space 1 … N checkerboard positions spanning view frustum ground truth point clouds in space 1 and measured point clouds in space 2 Intrinsics K, A § Step 1: Rigid alignment Space 1 Space 2 P Young Min Kim Rrigid, trigid 23 P 1 TOF-CV Workshop

Depth Camera – Compensate Systematic Bias § Step 2: Warp of ray direction Φ(i, j), Ω(i, j) Space 1 Space 2 interpolate P 2 P 1 Young Min Kim 24 TOF-CV Workshop

Depth Camera – Compensate Systematic Bias § Step 2: Warp of ray direction Φ(i, j), Ω(i, j) Space 1 Space 2 interpolate P 2 P 1 § Step 3: Constant per-pixel length bias along ray D(i, j) P 2 Young Min Kim P 3 25 TOF-CV Workshop

Outline § § § Introduction System Architecture Depth Sensor Characteristics System Calibration Results and Conclusion Young Min Kim 26 TOF-CV Workshop

Result Young Min Kim 27 TOF-CV Workshop

Result Young Min Kim 28 TOF-CV Workshop

Result Combination of two depth maps Projectively textured from three video cameras Young Min Kim 29 TOF-CV Workshop

Result Combination of three depth maps Before bias correction Young Min Kim After bias correction 30 TOF-CV Workshop

Result § Mean error of 4. 94 cm reduced into 1. 36 cm Young Min Kim 31 TOF-CV Workshop

Conclusion § Design of a multi-view TOF fusion recording system § Detailed analysis of depth measurement inaccuracy § Calibration of depth and video data into a common frame § Starting point for improved dynamic shape and texture reconstruction § Acknowledgement: Max Planck Center for Visual Computing and Communication Young Min Kim 32 TOF-CV Workshop

Thank you http: //www. stanford. edu/~jinhae