MultiAperture Photography Paul Green MIT CSAIL Wenyang Sun

Multi-Aperture Photography Paul Green – MIT CSAIL Wenyang Sun – MERL Wojciech Matusik – MERL Frédo Durand – MIT CSAIL

2 Motivation Post-Exposure Depth of Field Depth. Control of Field Portrait Landscape Large Aperture Shallow Depth of Field Small Aperture Large Field Depth of http: //photographertips. net

3 Depth and Defocus Blur sensor lens plane of focus circle of confusion defocus blur depends on distance from plane of focus

Defocus Blur & Aperture sensor plane of focus lens aperture circle of confusion defocus blur depends on aperture size http: //photographertips. net 4

Goals Aperture size is a critical parameter for photographers ■ post-exposure DOF control ■ extrapolate shallow DOF beyond physical aperture 5

Outline Multi-Aperture Camera ■ New camera design ■ Capture multiple aperture settings simultaneously Applications ■ Depth of field control ■ Depth of field extrapolation ■ Refocusing 6

7 Related Work Computational Cameras ■ Plenoptic Cameras ■ Adelson and Wang ‘ 92 ■ Ng et al ‘ 05 ■ Georgiev et al ‘ 06 ■ Split-Aperture Camera Georgiev et al‘ 06 Adelson and Wang ‘ 92 ■ Aggarwal and Ahuja ‘ 04 ■ Optical Splitting Trees ■ Mc. Guire et al ‘ 07 ■ Coded Aperture ■ Levin et al ’ 07 ■ Veeraraghavan et al ’ 07 Mc. Guire et al ‘ 07 Aggarwal and Ahuja ‘ 04 ■ Wavefront Coding ■ Dowski and Cathey ‘ 95 Depth from Defocus ■ Pentland ‘ 87 Levin et al ’ 07 Veeraraghavan et al ’ 07

8 Plenoptic Cameras Capture 4 D Light. Field ■ 2 D Spatial (x, y) ■ 2 D Angular (u, v Aperture) v Main Idea: Trade resolution for flexibility after capture Lenslet Array ■ Refocusing ■ DOF control ■ Improved Noise x, y Characteristics Sensor Aperture u u, v Lens Subject

1 D vs 2 D Aperture Sampling 9 Aperture v u 2 D Grid Sampling http: //photographertips. net

1 D vs 2 D Aperture Sampling Aperture v 45 Samples 10 Aperture 4 Samples u 2 D Grid Sampling 1 D “Ring” Sampling http: //photographertips. net

11 Optical Splitting Trees General framework for sampling imaging parameters ■ Beamsplitters ■ Multiple imagers Incoming light Beamsplitter Small Aperture Large Aperture Mc. Guire et al ‘ 07

Goals ■ ■ ■ post-exposure DOF control extrapolate shallow DOF 1 d sampling no beamsplitters single sensor removable 12

Outline Multi-Aperture Camera ■ New camera design ■ Capture multiple aperture settings simultaneously Applications ■ Depth of field control ■ Depth of field extrapolation ■ Refocusing 13

14 Optical Design Principles 3 D sampling ■ 2 D spatial ■ 1 D aperture size ■ 1 image for each “ring” Aperture http: //photographertips. net Sensor

Aperture Splitting Goal: Split aperture into 4 separate optical paths ■ concentric tilted mirrors ■ at aperture plane Tilted Mirrors 15

16 Aperture Splitting Focusing lenses Mirrors Sensor Incoming light Tilted Mirrors

17 Aperture Splitting Ideally at aperture plane , but not physically possible! Solution: Relay Optics to create virtual aperture plane Photographic Relay. Lens system Aperture splitting optics X Aperture Plane New Aperture Plane

18 Optical Prototype mirrors main lens relay optics Mirror Close-up lenses tilted mirrors SLR Camera

Sample Data Raw data from our camera 19

20 PSF Occlusion Analysis inner ring 1 ring 2 Ideally would be rings Gaps are from occlusion outer combined

Outline Multi-Aperture Camera ■ New camera design ■ Capture multiple aperture settings simultaneously Applications ■ Depth of field control ■ Depth of field extrapolation ■ Refocusing 21

DOF Interpolation 22

23 DOF Extrapolation? Approximate defocus blur as convolution Depends on depth and aperture size - Circular aperture blurring kernel ? What is at each pixel in ?

24 DOF Extrapolation Roadmap estimate blur σ extrapolate blur fit model Largest physical aperture IE Blur size capture I 0 I 1 I 2 I 3 D Aperture Diameter

25 Defocus Gradient σ IE Blur size Defocus blur Largest physical aperture I 0 Defocus Gradient I 3 I 2 I 1 Aperture Diameter D G is slope of this line Defocus Gradient Map

26 Optimization solve for discrete defocus gradient values G at each pixel Data term Graph Cuts with spatial regularization term Smallest Aperture Image Defocus Gradient Map

DOF Extrapolation video 27

Synthetic Refocusing 28 Need to focus on nearest object gradient map “refocused” map extrapolated f/1. 8 “refocused” synthetic f/1. 8

Synthetic Refocusing Video 29

Depth Guided Deconvolution 30

Limitations Optical Design ■ Occlusion ■ Difficult alignment process DOF extrapolation and refocusing ■ DOF is dependent on smallest aperture (but our deconvolution helps) 31

Conclusions ■ ■ ■ Post-Exposure DOF control DOF extrapolation 1 D sampling of aperture No beamsplitters Removable 32

Thanks Funding People ■ NSF CAREER award 0447561 ■ John Barnwell ■ Ford Foundation predoctoral Fellowship ■ Jonathan Westhues ■ Se. Baek Oh ■ Microsoft Research New Faculty Fellowship ■ Daniel Vlasic ■ Sloan Fellowship ■ Eugene Hsu ■ Tom Mertens ■ Jane Malcolm 33