Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke

Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman 3 Wolfgang Heidrich 1 1 University of British Columbia 2 Saarland University 3 MIT Media Lab VI. Multiplexing Time Eurographics 2011 – State of the Art Report EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 1

History – Eadweard Muybridge 1830 -1904 http: //en. wikipedia. org/wiki/Eadweard_Muybridge EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 2

History – Étienne-Jules Marey 1830 -1904 www. wikipedia. org EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 3

VI. I Time Lapse Photography EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 4

BBC Time Lapse – Look It! • Long exposures to avoid temporal aliasing ND 3. 0 filter, f 22, 1 minute exposure EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 5

VI. II High-Speed Imaging EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 6

High-Speed Cameras Vision Research Phantom Flex (CMOS) 2570 fps at HD resolution Shimadzu Hyper. Vision HPV-2 (CCD) one million fps at 312 x 260 pixels Photron FASTCAM SA 5 (CMOS) 7500 fps at megapixel resolution one million fps at 64 x 64 pixels Casio Exilim Series (consumer cam) 1000 fps at reduced resolution EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 7

Assorted Pixels [Narasimhan & Nayar 05] EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 8

Temporal Mosaic with DMD aligned with CCD in microscope [Bub et al. 10] EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 9

Non-Destructive Sensor Readout & Pixim www. pixim. com Cypress Semiconductor LUPA 3000 3 megapixels, 485 fps EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 10

Coded Rolling Shutter EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging [Gu et al. 10] 11

Reinterpretable Imager • Moving pinhole over time in aperture • Capture with light field camera EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging [Agrawal et al. 10] 12

Bullet Time Effect from ‘The Matrix’ EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 13

Stanford Multi-Camera Array EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging [Wilburn et al. 04] 14

Coded Temporal Sampling EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging [Agrawal et al. 10] 15

High-Speed Illumination – Electronic Strobes Harold ‘Doc’ Edgerton 1903 -1990 EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 16

Temporal Dithering with DLP Illumination EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging [Narasimhan et al. 08] 17

Coded Strobing Photography EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging [Reddy et al. 11] 18

Streak Cameras C 5680 $200 K www. hamamatsu. com EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 19

VI. I Motion Deblurring EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 20

Motion Deblurring Overview • Motion blur is velocity-dependent • Can be described as convolution, where – Kernel shape is motion trajectory – Trajectory is modulated by exposure function http: //en. wikipedia. org/wiki/Motion_blur EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 21

Deconvolution is Still Hard • Again – problems: – Camera noise – Spatially varying kernel (velocity-dependent) – Unknown motion trajectory – Ill-posed problem, kernel of box integration function is not invertible (optical cancellation of image frequencies) EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 22

Approaches to Improve Motion Deblurring • Make PSF invertible coded exposure • Make PSF velocity-invariant shiftinvariant deconvolution • Automatize PSF estimation EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 23

Flutter Shutter [Raskar et al. 06] EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 24

Optimal Motion PSFs • Optimality criteria PSF invertibility & estimation [Agrawal & Xu 07] EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 25

Motion Invariant Photography • Engineer PSF to be motion invariant • Only for 1 D motion EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging [Levin 08] 26

Hybrid Cameras • Combined high-speed low-quality & lowspeed high-quality camera [Ben-Ezra & Nayar 04] Input images Deblurred Result EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging Computed PSF Ground Truth 27

Motion Blur in Video • Coded exposure & super-resolution in successive video frames [Agrawal et al. 09] EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 28

Next: Further Light Properties EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 29

Flexible Voxels • Flexible space-time resolution as postprocessing EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging [Gupta et al. 10] 30

Synthetic Shutter Speed Imaging [Telleen 07] • Combine multiple short exposures to reduce noise • Align with optical flow EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 31

Hybrid Cameras [Tai et al. 08] • Motion deblurring & super-resolution EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 32

Hybrid Cameras • Motion deblurring & depth from two lowresolution high-speed camers [Li et al. 08] Input images Deblurred result EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging Recovered Depth 33

Analysis • Analysis of optimal coded, single image deblurring • MIP becomes worse when velocities exceed expectations Coded Exposure Motion Invariant Photography EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging [Agrawal & Raskar 09] 34