Surface Light Fields for 3 D Photography Daniel

Surface Light Fields for 3 D Photography Daniel N. Wood University of Washington SIGGRAPH 2001 Course

Collaborators (and co-authors on SIGGRAPH 2000 paper) Daniel Azuma Wyvern Aldinger Brian Curless Tom Duchamp David Salesin Werner Stuetzle University of Washington

Outline 1. Surface light field representation 2. Compression primer 3. Surface light fields for 3 D photography • • With details of compression And a preliminary look at a new compression algorithm

Surface light fields

Lumisphere-valued “texture” maps Lumisphere

Surface light fields in flatland q s

Surface light fields in flatland q s

Surface light fields in flatland q s

Compression primer • Singular value decomposition (or principal components analysis) – Handling color – Using regions – Reflection parameterization • Vector quantization • Others… (Wavelets, DCT)

Singular value decomposition U SLF VT

SVD in two matrices Eigen Textures SLF Eigen Lumispheres

First eigenvectors Eigen Textures SLF Eigen Lumispheres First eigentexture and corresponding first eigenlumisphere

Truncated SVD Eigen-texture Eigen-lumisphere = ~ Outer product of first eigen-texture and first eigenlumisphere is closest rank 1 (separable) matrix.

Handling color Separate matrices Colors in directions (rows) Colors in surface texture (columns)

Handling color Separate matrices Colors in directions (rows) Colors in surface texture (columns) Chen et al. Wood et al. Nishino et al.

SVD in color Eigen Textures SLF Eigen Lumispheres

Zooming in… Eigen Textures SLF Eigen Lumispheres

Zoom into important vectors . . . SLF . . .

Reconstruction using SVD Original Rank 1 Rank 7 (1: 20)

Surface light field structure Rows are points on surface What are the columns? And, can they be made more coherent?

Surface light field structure Rows are points on surface Increasing column coherence: 1. Break into regions, and / or 2. Use reflection parameterization

Separate SVD for regions Eigen Textures Eigen Lumispheres

Reconstruction using regions Original One region (Rank 5) Two regions (Rank 5)

Reflection reparameterization

Reflection reparameterization

Reflection reparameterization

Reflection in flatland* Un-reflected Reflected *sort of

Reflection doesn’t happen in the plane q Un-reflected q Reflected

Reflection in 3 space Un-reflected Reflected

Reflection in “flatland” Un-reflected Original Rank 5 Reflected Original Rank 5

Other compression strategies • Discrete cosine transform [ Miller et al. 1999 – Eurographics Workshop on Rendering] • Vector quantization [ Wood et al. 2000 - SIGGRAPH ] • Wavelet decomposition [ Magnor and Girod 2000 - SPIE VCIP]

Vector quantization (unreflected) Uncompressed Codebook Vectorquantized

Vector quantization (reflected) Uncompressed Codebook Vectorquantized

Surface light fields for 3 D photography (SIGGRAPH 2000) Goals Rendering and editing Inputs Photographs and geometry Requirements Estimation and compression

Overview Data acquisition Estimation and compression Rendering Editing

Overview Data acquisition Estimation and compression Rendering Editing

Scan and reconstruct geometry Range scans (only a few shown. . . ) Reconstructed geometry

Take photographs Camera positions Photographs

Register photographs to geometry Geometry Photographs

Register photographs to geometry User selected correspondences (rays)

Parameterizing the geometry Map Base mesh Scanned geometry

Sample base mesh faces Base mesh Detailed geometry

Assembling data lumispheres Data lumisphere

Overview Data acquisition Estimation and compression Rendering Editing

Pointwise fairing Data lumisphere Faired lumisphere

Pointwise fairing results Input photograph Pointwise faired (177 MB)

Pointwise fairing Many input data lumispheres Many faired lumispheres

Compression Small set of prototypes

Compression / Estimation Many input data lumispheres Small set of prototypes

Median removal Reflected Median (“diffuse”) + Median-removed (“specular”) +

Median removal Median values Specular Result

Function quantization Input data lumisphere Codebook of lumispheres

Lloyd iteration Input data lumispheres

Lloyd iteration Codeword

Lloyd iteration Perturb codewords to create larger codebook

Lloyd iteration Form clusters around each codeword

Lloyd iteration Optimize codewords based on clusters

Lloyd iteration Create new clusters

Function quantization results Input photograph Function quantized (1010 codewords, 2. 6 MB)

Principal function analysis Input data lumisphere Prototype lumisphere Subspace of lumispheres

Principal function analysis Prototype lumisphere Approximating subspace

Principal function analysis

Principal function analysis

Principal function analysis + Median + + PFA decomposition

Principal function analysis +… + Median PFA decomposition = Final approximation

Principal function analysis results Input photograph PFA compressed (Order 5 - 2. 5 MB)

Compression comparison Pointwise fairing (177 MB) Function quantization (2. 6 MB) Principal function analysis (2. 5 MB)

Rewind… what we didn’t want to do

Instead, a middle ground Regularly sampled directions, but not all there.

New method: Principal components with missing data Hallucinate data SLF with missing data Use either fairing or pair-wise present covariance matrix to fill holes Hole-filled SLF

New method: Principal components with missing data Hallucinate data SLF with missing data Find best low rank approximation Hole-filled SLF Low rank approximation

New method: Principal components with missing data Hallucinate data Improve hallucination SLF with missing data Low rank approximation

Preliminary comparison Principal function analysis Order 3, RMS error 26. 9 SVD with missing data Order 3, RMS error 26. 1

Overview Data acquisition Estimation and compression Rendering Editing

Interactive renderer screen capture

Overview Data acquisition Estimation and compression Rendering Editing

Lumisphere filtering Original surface light field Glossier coat

Lumisphere filtering

Rotating the environment Original surface light field Rotated environment

Deformation Original Deformed

Deformation

Summary 1. Estimation and compression • • Function quantization Principal function analysis 2. Rendering • • From compressed representation With view-dependent level-of-detail 3. Editing • • Lumisphere filtering Geometric deformations and transformations

Future work • Better geometry-to-image registration • Derive geometry from images • More complex surfaces (mirrored, refractive, fuzzy…) under more complex illumination

Acknowledgements • Marc Levoy and Pat Hanrahan – (Thanks for the use of the Stanford Spherical Gantry) • Michael Cohen and Richard Szeliski • National Science Foundation

Nearly the end

For more information http: //graphics. washington. edu/projects/slf Talks, papers, … and raw data.