Generalized Mosaics Yoav Y Schechner Shree Nayar Department

Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University ICCV Conference July 2001, Vancouver, Canada

Mosaics Processing

Redundant Measurements Generalized Mosaicing: Yoav Schechner and Shree Nayar

Scanning with Less Redundancy Mantis Shrimp C Takata Different rows = Different optical characteristics

Generalized Mosaicing camera Spatially varying filter Field of View Brightness Dynamic Range Spectrum Polarization Depth of Field Schechner, Nayar, Generalized mosaics

Mosaic + High Dynamic Range 88 - 18, 794

Fusion of Measurements Maximum-Likelihood Solution Schechner, Nayar, Generalized mosaics

Log of the mask log 2 M -2 -4 -6 -8 8 bits (almost) x (pixels) 8 bit camera 1 10 -6 Dynamic range as 16 bits = 1 10 -2 Generalized Mosaicing: Yoav Schechner and Shree Nayar

Mask Self-Calibration Unknown filter (vignetting) M Consistency, Smoothness A average row consistency constraints Schechner, Nayar, Generalized mosaics x

Image Registration Schechner, Nayar, Generalized mosaics

Image Registration Bias towards “no motion” Schechner, Nayar, Generalized mosaics

Image Registration Uncertainty Schechner, Nayar, Generalized mosaics

Registration: Standard “Coarse to Fine” motion of 2 pixels 4 pixels 5 pixels 10 pix 9 pix 18 pix 19 pix However, we need: multiscale UNCERTAINTIES Schechner, Nayar, Generalized mosaics

Maximum-Likelihood Pyramid At each level L , for each pixel estimate -1 2 Schechner, Nayar, Generalized mosaics

Max-Likelihood Pyramid Gaussian Pyramid ML estimation

Mosaic Generalized Mosaic fram Intensity range e log I frame Spatial range x Schechner, Nayar, Generalized mosaics

Variable Spectral Filter spectral y l x Generalized Mosaicing: Yoav Schechner and Shree Nayar

Multi Spectral Mosaic 400 500 600 l 700 Generalized Mosaicing: Yoav Schechner and Shree Nayar

Rendering : Any Illumination sunset fluorescent Generalized Mosaicing: Yoav Schechner and Shree Nayar

Illumination at a Glance Halogen y Fluorescent y l l x x Extra information § Illuminant spectrum

Spatially Varying Polarizer Polarization Mosaicing: Yoav Schechner and Shree Nayar

Spatially Varying Polarizer 1 polarizance 90 o o polarizing 60 angle 0. 8 30 0. 6 0 o 0. 4 0. 2 0 transmittance o -30 o -60 -90 o o Polarization Mosaicing: Yoav Schechner and Shree Nayar

Raw images Polarization Mosaicing: Yoav Schechner and Shree Nayar

Polarization Mosaic reflected structure Schechner Shamir Kiryati JOSA-A 2000 transmitted painting

Spatially Varying Focus Insert a prism behind the lens y = Depth x Generalized Mosaicing: Yoav Schechner and Shree Nayar

All Focused Mosaic … and a Depth Map Generalized Mosaicing: Yoav Schechner and Shree Nayar

What else? n Multi-dimensional Mosaics - Simultaneously Dynamic Range & Spectrum & Polarization etc. Generalized Mosaicing: Yoav Schechner and Shree Nayar

Sampling Criteria Number of samples per scene point l length L bandwidth B resolution Dl aperture D § Signal undergoing LPF § “Band limited” signal § Minimizing aliasing M 1 1/2 1/4 Generalized Mosaicing: Yoav Schechner and Shree Nayar 1011100101 § Nyquist sampling rate # samples 1 1111 2 1111? 10111001? ? 3 1011100? ? ? 9 101110? ?

Mosaic Generalized Wide field of view Spatially varying filter Dynamic range 88 – 18, 794 I Spectrum l Polarization Depth + Focus