Mitsubishi Electric Research Labs MERL SuperRes from Single

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Agrawal & Raskar Resolving Objects at Higher Resolution from a Single Motion-Blurred Image Amit Agrawal and Ramesh Raskar Mitsubishi Electric Research Labs (MERL) Cambridge, MA, USA

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Agrawal & Raskar Can we recover more detail by letting objects blur? Text Object Moves Low Res Camera: Static Object Text: 10 pixels wide Low Res Camera: Blurred Image Blurred Text: 30 pixels wide 10 pixels Traditional Motion Deblurring: Recovers 10 pixels 30 pixels Our approach: Resolution enhanced deblurring Resolution enhancement factor = 3

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Agrawal & Raskar Applications Aerial Imaging Time = T Time = 0 Electronic Toll Booths Goal: Automatic license plate number recognition

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Overview • Exploiting motion blur for super-res – Linear system – No hallucination, image priors or training data • Motion deblurring is ill-posed – Practical Solution: Flutter Shutter Camera • Resolution enhanced deblurring – Using Flutter Shutter Camera Agrawal & Raskar

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Agrawal & Raskar Sensor point spread function (PSF) 20 pixel object 10 Ground Truth Static Scene (High Resolution Camera) Static Scene (Low Resolution Camera) … 00011000 … Decimation Matrix D Decimation Matrix: High Resolution -> Low Resolution

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Agrawal & Raskar Exploiting motion blur for super-res 20 pixel object Ground Truth Static Scene (High Resolution Camera) 10 pixels Static Scene (Low Resolution Camera) Traditional Motion Deblurring Object Moves 40 20 pixels Blurred Image (High Resolution Camera) Blurred Image (Low Resolution Camera)

Mitsubishi Electric Research Labs (MERL) 10 Super-Res from Single Motion Blur Photo 20 Agrawal & Raskar Blurred Image (Low Resolution Camera) Static Scene (Low Resolution Camera) 20 Blurring Matrix A Sharp Image 10 ? x Blurred Image Traditional Motion Deblurring b

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Agrawal & Raskar 10 20 Traditional Motion Deblurring Blurring Matrix A Sharp Image x 00011000 20 … 20 ? Blurred Image Low Resolution … Decimation Matrix D Blurring Matrix A High Resolution Sharp Image x Resolution Enhanced Deblurring Blurred Image Low Resolution

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Agrawal & Raskar Deblurring is ill-posed K = D * A: Typical Case: Combined deblurring + resolution enhancement matrix Condition number of K ~ 1018 Estimated K Blurred Image + Noise Ground Truth

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Overview • Exploiting motion blur for super-res – Linear system – No hallucination, image priors or training data • Motion deblurring is ill-posed – Practical Solution: Flutter Shutter Camera [Raskar, Agrawal, Tumblin, SIGGRAPH 2006] • Resolution enhancement deblurring – Using Flutter Shutter Camera Agrawal & Raskar

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Blurring == Convolution Agrawal & Raskar Frequency Spectrum Magnitude Sinc Function ω Traditional Camera: Shutter is OPEN: Box Filter

Mitsubishi Electric Research Labs (MERL) Traditional Super-Res from Single Motion Blur Photo Coded Exposure Deblurred Image Matlab Lucy Image of Static Object Agrawal & Raskar

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Agrawal & Raskar Implementation LCD Shutter in front of lens becomes opaque and transparent

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Agrawal & Raskar License Plate Retrieval

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Agrawal & Raskar Related Work • Super-Resolution & Motion Deblurring – – – – Sondhi, Proc. IEEE 1972 Irani & Peleg, CVGIP’ 91, CVPR’ 92 Baker & Kanade, PAMI’ 02 Ben-Ezra et al. PAMI 2005 Bascle et al. ECCV’ 96 Tappen et al. SCTV’ 03 Shechtman et al. ECCV’ 02 • Coding & Modulation in Cameras/Projectors – – HDR, Nayar & Mitsunaga, CVPR’ 00 Multiplexed Illumination, Schechner et al. ICCV’ 03 Flutter shutter camera, Raskar et al. SIGGRAPH’ 06 Coded aperture, Hiura & Matsuyama, CVPR’ 98 – Coded aperture with lens, light field modulation, SIGGRAPH’ 07 • With Jack Tumblin, Ankit Mohan and Ashok Veeraraghavan (UMD)

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Overview • Exploiting motion blur for super-res – Linear system – No hallucination, image priors or training data • Motion deblurring is ill-posed – Practical Solution: Flutter Shutter Camera – Raskar et al. SIGGRAPH 2006 • Resolution enhancement deblurring – Using Flutter Shutter Camera Agrawal & Raskar

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Agrawal & Raskar Super-resolution with motion blur Jitter Camera Z. Lin & H. Y Shum, ‘Fundamental Limits of Reconstruction-Based Superresolution Algorithms under Local Translation’, PAMI 2004 M. Ezra, A. Zomet & S. K. Nayar, ‘Video Super-Resolution Using Controlled Subpixel Detector Shifts’, PAMI 2005

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Experimental Setup Resolution Chart is moved horizontally along the tracks Agrawal & Raskar

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Preserves High Spatial Frequencies Flutter Shutter: Shutter is OPEN and CLOSED Agrawal & Raskar

Super-Res from Single Motion Blur Photo Mitsubishi Electric Research Labs (MERL) Input Blurred Photo (cropped) Deblurred Image Recovered at triple resolution Blurred Width Recovered at normal resolution Unblurred Width Agrawal & Raskar Deblurred Image Upsampled Super-Resolution Deblurring Result

Mitsubishi Electric Research Labs (MERL) Input Image Patch Deblurred Patch Our approach Super-Res from Single Motion Blur Photo Agrawal & Raskar

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Agrawal & Raskar Future Work • Exploiting Motion Blur – 2 D Motion Blur: Camera shake – Super-resolution with RGB Bayer mosaic – Coded exposure video with Pointgrey Dragonfly 2 camera • Focus blur – Coded Aperture – Handling motion and focus blur simultaneously • Fluttering Codes – Binary vs Continuous – Suitable for other applications

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Acknowledgments • Jack Tumblin, Northwestern University • Hideaki Nii, University of Tokyo • MERL – Fatih Porikli, Jay Thornton – Joseph Katz, Keisuke Kojima • Haruhisa Okuda, MELCO Agrawal & Raskar

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Agrawal & Raskar Super-res from single motion blurred photo • Exploiting single photo motion blur – Letting objects blur in 1 D – Solving simple linear system – More blur = more resolution but with increased noise • Practical implementation – Coded Exposure (Flutter Shutter) Camera – Makes deblurring well-posed • Results on 1 D resolution enhancement – Using Flutter Shutter Camera

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Agrawal & Raskar Coded Exposure (Flutter Shutter) Camera Motion Deblurring + Resolution Enhancement Google: ‘Flutter Shutter’

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Extra Slides Agrawal & Raskar

Super-Res from Single Motion Blur Photo Are all codes “good”? Mitsubishi Electric Research Labs (MERL) All ones Alternate Random Our Code Agrawal & Raskar

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Need to consider zero padded codes!!! Agrawal & Raskar

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Agrawal & Raskar

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Agrawal & Raskar Input Image

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Rectified Crop Deblurred Result Agrawal & Raskar

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Shutter open closed Agrawal & Raskar Shutter 11111111111 Exposure Time open closed 110110101001100011 Continuous blur Blurred image: Traditional camera code Exposure Time Coded blur Blurred image: Coded exposure camera Code: 110110101001100011

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Comparison Agrawal & Raskar

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Agrawal & Raskar Inverse Filter stable Inverse Filter Unstable

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur Photo Agrawal & Raskar Traditional Motion Deblurring Captured Blurred Image Deblurring is at the same resolution