Jitter Camera High Resolution Video from a Low

Jitter Camera: High Resolution Video from a Low Resolution Detector Moshe Ben-Ezra, Assaf Zomet and Shree K. Nayar IEEE CVPR Conference June 2004, Washington DC, USA

Video Resolution Plasma Display Resolution 1366 x 768 Mini. DV Camera. Resolution: 720 x 480 Digital Camera. Resolution: 2592 x 1944

Temporal resolution (fps) Fundamental Resolution Tradeoff 3 Hi-resolution still Camera Conventional video camera 30 130 3 M 2048 x 1536 330 K 720 x 480 Spatial resolution (pixels)

Super-Resolution Sequence taken by a moving camera Shechtman, Caspi, and Irani ECCV 2002 Zomet and S. Peleg. ICIP 2000 Baker and Kanade. CVPR 2000 Chiang and Boult, IVC 2000 High-Resolution computed image Capeland, Zisserman ICPR 2000 Elad and Feuer IP 1997 Irani and Peleg GMIP 1996

Super Resolution y = (D G)x + z All Sampled Images Blurring Op. Decimation Hi Res. Image Noise

Motion Blur Hurts Us Again!

Capture Images without Motion Blur

Effect of Motion Blur on Super-Resolution Input: No Motion Blur Super-Resolution Result Input : With Motion Blur (known) Super-Resolution Result

Quantifying The Affect of Motion Blur n Empirical tests: RMS error. n Volume of Solutions (Linear Model): Input Images High-Resolution Image Blur & Noise Decimation (Quantization) Baker and Kanade Volume of Solutions 1/det(A)

How Bad is Motion Blur for Super-Resolution? RMS Error After Super-Resolution Space of Super-Resolution Solutions 18 0 1 2 3 Motion blur in pixels 4 5 0 1 2 3 4 Motion blur in pixels 5

Avoid Motion Blur using Jitter Sampling Conventional Sampling Ti Spatial Jitter Sampling Ti me Space

The Jitter Camera Lens Detector Micro-actuator

The Jitter Camera Lens Detector Micro-actuator Detector is a light weight device! Jitter is instantaneous and synchronous

Computer Controlled Y Micro-Actuator Computer Controlled X Micro-Actuator Lens Board Camera

Y Pixels Jitter Mechanism Accuracy 1μm X Pixels Desired locations. Actual locations.

Result: Resolution Chart Four Images from the Jitter Camera Super-Resolution Image

Result: Color De. Mosaicing and Super-Resolution 1 (out of 4) Jitter camera Image Super-Resolution

Jitter Video (Stabilized) How can we handle dynamic scenes?

Adaptive Super-Resolution for Dynamic Scenes Static blocks: 4 frames used. Occlusions: 1 frame used. Moving object: 2 - 4 frames used

Adaptive Super-Resolution Algorithm I-3 I-2 I-1 I I+1 I+2 I+3 Estimate the aliasing error ‘ ’ (stdv) for each block Ik in I. Compute robust block matching between all pairs {I}{I 1, 2, 3}. Use ‘ ’ as a scale factor for an M-Estimator error function. 3. For each block Ik try to find 3 matching blocks {I x}k, s. t. : 1. 2. a) b) 4. SSD(Ik, {I x}k)-0. 5 < 3 {I x}k are temporally closest to Ik (smallest x) Apply super-resolution to the selected blocks. The algorithm degrades gradually from 4 -frames superresolution to a single frame interpolation and deblurring.

Scale Estimate Mean 6. 4, Stdv 14 Mean 7. 5, Stdv 16 Mean 8. 6, Stdv 17 Mean 10. 5, Stdv 27 Mean 15. 2, Stdv 30 Mean 17. 7, Stdv 33 Low Res - Hi-Res Aliasing Error (Simulated) Low Res 2 nd derivative (Simulated)