Fourier Slice Photography Ren Ng Stanford University Conventional

Light Field Photography Capture the light field inside the camera body

Hand-Held Light Field Camera Medium format digital camera Camera in-use 16 megapixel sensor Microlens

Questions About Digital Refocusing n What is the computational complexity? Are there efficient algorithms?

Overview n Fourier Slice Photography Theorem n Fourier Refocusing Algorithm n Theoretical Limits of

Previous Work Integral photography n Lippmann 1908, Ives 1930 n Lots of variants, especially

Fourier Slice Photography Theorem In the Fourier domain, a photograph is a 2 D

Digital Refocusing by Ray-Tracing x u Lens Sensor

Digital Refocusing by Ray-Tracing x u Imaginary film Lens Sensor

Refocusing as Integral Projection u x x u Imaginary film Lens Sensor

Classical Fourier Slice Theorem Integral Projection 1 D Fourier Transform 2 D Fourier Transform

Classical Fourier Slice Theorem Integral Projection Spatial Domain Fourier Domain Slicing

Fourier Slice Photography Theorem Integral Projection Spatial Domain Fourier Domain Slicing

Fourier Slice Photography Theorem Integral Projection 4 D Fourier Transform Slicing

Fourier Slice Photography Theorem Integral Projection 2 D Fourier Transform 4 D Fourier Transform

Photographic Imaging Equations Spatial-Domain Integral Projection Fourier-Domain Slicing

Theorem Limitations Film parallel to lens n Everyday camera, not view camera Aperture fully

Existing Refocusing Algorithms Existing refocusing algorithms are expensive n O(N 4) where light field

Refocusing in Spatial Domain Integral Projection 2 D Fourier Transform 4 D Fourier Transform

Refocusing in Fourier Domain Integral Projection Inverse 2 D Fourier Transform 4 D Fourier

Asymptotic Performance Fourier-domain slicing algorithm n Pre-process: O(N 4 log N) n Refocusing: O(N

Resampling Filter Choice Triangle filter (quadrilinear) Kaiser-Bessel filter (width 2. 5) Gold standard (spatial

Problem Statement Assume a light field camera with n An f /A lens n

Band-Limited Analysis Band-width of measured light field Light field shot with camera

Results of Band-Limited Analysis Assume a light field camera with n An f /A

Summary of Main Contributions n Formal theorem about relationship between light fields and photographs

Future Work n Apply general signal-processing techniques n Cross-fertilization with medical imaging

Thanks and Acknowledgments Collaborators on camera tech report n Marc Levoy, Mathieu Brédif, Gene

Questions? “Start of the race”, Stanford University Avery Pool, July 2005

Slides: 63

Download presentation

Fourier Slice Photography Ren Ng Stanford University

Conventional Photograph

Light Field Photography Capture the light field inside the camera body

Hand-Held Light Field Camera Medium format digital camera Camera in-use 16 megapixel sensor Microlens array

Light Field in a Single Exposure

Light Field Inside the Camera Body

Digital Refocusing

Questions About Digital Refocusing n What is the computational complexity? Are there efficient algorithms? n What are the limits on refocusing? How far can we move the focal plane?

Overview n Fourier Slice Photography Theorem n Fourier Refocusing Algorithm n Theoretical Limits of Refocusing

Previous Work Integral photography n Lippmann 1908, Ives 1930 n Lots of variants, especially in 3 D TV Okoshi 1976, Javidi & Okano 2002 n Closest variant is plenoptic camera Adelson & Wang 1992 Fourier analysis of light fields n Chai et al. 2000 Refocusing from light fields n Isaksen et al. 2000, Stewart et al. 2003

Fourier Slice Photography Theorem In the Fourier domain, a photograph is a 2 D slice in the 4 D light field. Photographs focused at different depths correspond to 2 D slices at different trajectories.

Digital Refocusing by Ray-Tracing x u Lens Sensor

Digital Refocusing by Ray-Tracing x u Imaginary film Lens Sensor

Refocusing as Integral Projection u x x u Imaginary film Lens Sensor

Classical Fourier Slice Theorem Integral Projection 1 D Fourier Transform 2 D Fourier Transform Slicing

Classical Fourier Slice Theorem Integral Projection Spatial Domain Fourier Domain Slicing

Fourier Slice Photography Theorem Integral Projection Spatial Domain Fourier Domain Slicing

Fourier Slice Photography Theorem Integral Projection 4 D Fourier Transform Slicing

Fourier Slice Photography Theorem Integral Projection 2 D Fourier Transform 4 D Fourier Transform Slicing

Photographic Imaging Equations Spatial-Domain Integral Projection Fourier-Domain Slicing

Theorem Limitations Film parallel to lens n Everyday camera, not view camera Aperture fully open n Closing aperture requires spatial mask

Overview n Fourier Slice Photography Theorem n Fourier Refocusing Algorithm n Theoretical Limits of Refocusing

Existing Refocusing Algorithms Existing refocusing algorithms are expensive n O(N 4) where light field has N samples in each dimension n All are variants on integral projection Isaksen et al. n Vaish et al. n Levoy et al. n Ng et al. 2000 n 2004 2005

Refocusing in Spatial Domain Integral Projection 2 D Fourier Transform 4 D Fourier Transform Slicing

Refocusing in Fourier Domain Integral Projection Inverse 2 D Fourier Transform 4 D Fourier Transform Slicing

Asymptotic Performance Fourier-domain slicing algorithm n Pre-process: O(N 4 log N) n Refocusing: O(N 2 log N) Spatial-domain integration algorithm n Refocusing: O(N 4)

Resampling Filter Choice Triangle filter (quadrilinear) Kaiser-Bessel filter (width 2. 5) Gold standard (spatial integration)

Overview n Fourier Slice Photography Theorem n Fourier Refocusing Algorithm n Theoretical Limits of Refocusing

Problem Statement Assume a light field camera with n An f /A lens n N x N pixels under each microlens If we compute refocused photographs from these light fields, over what range can we move the focal plane? Analytical assumption n Assume band-limited light fields

Band-Limited Analysis

Band-Limited Analysis Band-width of measured light field Light field shot with camera

Band-Limited Analysis

Photographic Imaging Equations Spatial-Domain Integral Projection Fourier-Domain Slicing

Results of Band-Limited Analysis Assume a light field camera with n An f /A lens n N x N pixels under each microlens From its light fields we can n Refocus exactly within depth of field of an f /(A N) lens In our prototype camera n Lens is f /4 n 12 x 12 pixels under each microlens Theoretically refocus within depth of field of an f/48 lens

Light Field Photo Gallery

Stanford Quad

Rodin’s Burghers of Calais

Palace of Fine Arts, San Francisco

Waiting to Race

Start of the Race

Summary of Main Contributions n Formal theorem about relationship between light fields and photographs n Computational application gives asymptotically fast refocusing algorithm n Theoretical application gives analytic solution for limits of refocusing

Future Work n Apply general signal-processing techniques n Cross-fertilization with medical imaging

Thanks and Acknowledgments Collaborators on camera tech report n Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz and Pat Hanrahan Readers and listeners n Ravi Ramamoorthi, Kayvon Fatahalian, Brad Osgood, Vaibhav Vaish, Gaurav Garg, Brian Curless, Dwight Nishimura, Mike Cammarano, Billy Chen, Jeff Klingner n Anonymous SIGGRAPH reviewers Funding sources n NSF, Microsoft Research Fellowship, Stanford Birdseed Grant

Questions? “Start of the race”, Stanford University Avery Pool, July 2005