Motion Deblurring Using Hybrid Imaging Moshe BenEzra and

Motion Deblurring Using Hybrid Imaging Moshe Ben-Ezra and Shree K. Nayar Columbia University IEEE CVPR Conference June 2003, Madison, USA

Image Recording Requires Time Niépce 1827 8 hours exposure Daguerre 1829 1/2 hour exposure

Motion Blur is Everywhere Object Motion Camera Motion

Stabilized Lenses 1/250 second (< -1 stop) 1/15 second (< -5 stops) n Stabilization drifts with time n Rotation only Canon Stabilized lens 400 mm

Blind Image Deconvolution n Accurate Point Spread Function (PSF) Needed.

Energy ~ 1/ speed Motion Point Spread Function (PSF) H Motion PSF is a Function of: 1. Motion path 2. Motion speed Y X Spatial spread

PSF Detector? PSF Detector Camera Can the PSF detector be a small and simple imaging device ?

Fundamental Limits of Imaging Photon flux Detector Electron wells Pixel’s Signal Noise Detector’s noise level

Temporal resolution (fps) Fundamental Resolution Tradeoff Hybrid imaging system 3 Hi-resolution camera 30 Conventional video camera 130 Low-resolution camera 3 M 2048 x 1536 330 K 720 x 480 75 K 320 x 240 Spatial resolution (pixels) A Hybrid camera enjoys both worlds

Overview of Approach Low-Res. camera y Motion Analysis x Same time period PSF Estimation Hi-Res. camera Deconvolution

Global Motion From Low Resolution Detector Objective function (Optical flow constraint) Rotation Lucas Kanade Translation

Simulations: Motion Accuracy from Low- Res. Images Average Motion Error in Pixels =3 =9 = 27 = 81 640 x 640 (1: 1) 0. 01 0. 02 0. 04 320 x 320 (1: 4) 0. 03 0. 04 0. 05 0. 1 160 x 160 (1: 16) 0. 03 0. 04 0. 07 0. 4 80 x 80 (1: 64) 0. 13 0. 21 0. 39 2. 6 Noise Resolution

Constraints on Continuous PSF n Energy conservation constraint: n Constant flux assumption: n Smoothness constraint: Path is continuous and twice differentiable

PSF Estimation from Computed Motion y f 1 f 2 f 3 y f 6 f 4 f 5 f 1 f 2 f 3 f 4 f 5 … Frame 2 f 6 Frame 5 x x h 4 h 2 h h 3 h 4 h 5 h 2 h y Frame 2 … Frame 5 h 3 h 5 y Frame 2 … Frame 5 x

Deconvolution of High Resolution Image n Standard iterative ratio-based algorithm* Image estimate PSF Error n Guaranties non-negative pixel result * Richardson [72] Lucy [74]

Designs for Hybrid Imaging A rig of two cameras Using a beam splitter Using a special chip

Our Prototype: Rig of Two Cameras Primary detector (2048 x 1536) Secondary detector (360 x 240) Resolution ratio of 1 : 36

Example 1 - Blurred Hi-Res Image f = 633 mm, Exp. Time 1 Sec (> -9 stops)

PSF Estimation from Motion Estimated PSF Low resolution sequence. 0. 06 Y (Pixels) 90 10 10 X (Pixels) f = 633 mm, Exp. Time 1 Sec 130 0. 001

Deblurred Image f = 633 mm, Exp. Time 1 Sec

Example 1 - Comparison Tripod image (Ground Truth) Blurred image Deblurred image f = 633 mm, Exp. Time 1 Sec

Example 2 - Blurred Night Image f = 884 mm, Exp. Time 4 Sec (> -11 stops)

PSF Estimation from Motion Low resolution sequence. 0. 003 Y (Pixels) 30 10 0. 001 10 X (Pixels) f = 884 mm, Exp. Time 4 Sec 60

Deblurred Night Image f = 884 mm, Exp. Time 4 Sec

Example 3 - Comparison Tripod image (Ground Truth) Blurred image Deblurred image f = 884 mm, Exp. Time 4 Sec

Object Deblurring Problem Moving objects blend into the background

Hybrid Imaging Solution (simulated) Requires clear high-resolution background image

Quantifying The Affect of Motion Blur n Empirical tests: RMS error. n Volume of Solutions (Linear Model): Input Images High-Resolution Image Blur Uncertainty Decimation (Quantization) Volume of Solutions 1/det(A)