Mahdi M Bagher Cyril Soler Kartic Subr Laurent

Mahdi M. Bagher / Cyril Soler / Kartic Subr / Laurent Belcour / Nicolas Holzschuch Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction

Shading • Direct reflection under distant illumination with no visibility and global from Wikipedia illumination effects Illumination BRDF • Rendering an image is computationally expensive.

Acquired Materials • • Using measured materials photo-realistic rendering large memory footprint (33 MB for a single isotropic BRDF) Not suitable for real-time rendering Phong shading BRDF measurement Gantry [Matusik 2002] Measured “Color-Changing-Paint-3”

Monte Carlo Sampling • Monte Carlo approximation to the shading integral • Importance sampling • less noisy • Monte Carlo sampling Data driven reflectance • no analytical importance function • pre-computed Importance samples BRDF importance sampling

Sampling • Dealing with two types of sampling • Reconstruction • sampling in image space • to render an image • Integration • sampling in the space of incident directions • to shade each individual pixel Illumination BRDF Cosine

The Problem • Shading is slow… • How to speed-up shading?

Case Study • What difference various materials make in rendering an image? Measured reflectance data from [Matusik 2002]

Diffuse Materials • • • Diffuse materials shading varies slowly across the image (low frequencies) many integration samples for each pixel (wide BRDF lobe) Measured Teflon [Matusik 2002] BRDF Lobe [BRDFLab]

Specular Materials • • • Specular materials shading varies quickly across the image (high frequencies) few integration samples for each pixel (narrow BRDF lobe) Measured color-Changing-Paint 3 [Matusik 2002] BRDF Lobe [BRDFLab]

The Problem • Rendering is slow… • How to speed-up shading? • Can we exploit the coherency between • Reconstruction samples… • Integration samples… • …for adaptive sampling? • How to measure the coherency between samples?

Intuition • Study the frequencies in the image • As a function of what happens to the light • Traveling from the light source to the camera

The Traveling Light • Light when traveling is affected by • Travel through free space • Object’s curvature • Reflectance • • BRDF Texture • Occlusion

Our Contribution • How to speed-up shading? 1. By sparse sampling the image • How to combine the sparse samples? • Multi-resolution rendering and bi-lateral up-sampling 2. By adaptive sampling for integration

Our Contribution (Illustrated) , Sparse reconstruction samples Adaptive Integration samples ∝ the local maximum variations in the image ∝ the average variance of the shading integrand Multi-resolution shading and bi-lateral up-sampling Final shaded image

Assumptions • Direct reflection under distant illumination Normal • • No visibility and global illumination effects Allow dynamic editable geometry Screen-space rendering in the context of deferred-shading Any shading model is possible • Save more if shading cost is higher e. g. measured materials • BRDF importance sampling

Dynamic Geometry Animation

Roadmap • • Related work Our technique • • The goal Sparse sampling for reconstruction Adaptive sampling for integration Validation The rendering algorithm Results and comparisons Extension: MSAA Conclusion / Future work

Related Work GPU Rendering of Complex Materials [Heidrich&Seidel 1999] [Kautz&Mc. Cool 1999] [Kautz 2002] [Mc. Cool 2001] [Latta&Kolb 2002] [Ramamoorthi 2002] [Claustres 2007] [Wang 2009]

Related Work Multi-Resolution Screen-Space Algorithms [Nichols&Wyman 2009] [Shopf 2009] [Nichols&Wyman 2010] [Nichols 2010] [Segovia 2006] [Soler 2010]

Related Work Pre-Computed Transport [Sloan 2002] [Liu 2004] [Ramamoorthi 2009] [Loos 2011] [Loos 2012] [Sun 2007]

Roadmap • • Related work Our technique • • The goal Sparse sampling for reconstruction Adaptive sampling for integration Validation The rendering algorithm Results and comparisons Extension: MSAA Conclusion / Future work

Goal • Sparse sampling for reconstruction • ∝ maximum local variations (local bandwidth) • Adaptive sampling for integration • ∝ variance of the shading integrand • Multi-resolution rendering and bilateral upsampling. • a single coarse-to-fine rendering pass

Roadmap • • Related work Our technique • • The goal Sparse sampling for reconstruction • • Local light-field definition Spectral Operations 2 D local bandwidth Image-space local bandwidth Adaptive sampling for integration Validation …

Local Light-Field • 4 D local light-field illustrated in 2 D Space Central ray Angle

Roadmap • • Related work Our technique • • The goal Sparse sampling for reconstruction • • Local light-field definition Spectral Operations 2 D local bandwidth Image-space local bandwidth Adaptive sampling for integration Validation …

Spectral Operations • Spectral operations [Durand et al. 2005]

Roadmap • • Related work Our technique • • The goal Sparse sampling for reconstruction • • Local light-field definition Spectral Operations 2 D local bandwidth Image-space local bandwidth Adaptive sampling for integration Validation …

2 D Local Bandwidth • 2 D Local Bandwidth • maximum local variations about a light path both in angle and space. 2 D local bandwidth spatial angular

Propagating Local Bandwidth

Propagating Local Bandwidth • One-bounce 2 D bandwidth propagation • Transport (from light to reflector) • Reflection (BRDF & Texture) • Transport (from reflector to pixel) • Matrix operators on 2 D bandwidth where Bandwidth Transport to Reflection Transport to Illumination at pixel p image plane operator the surface bandwidth Scale Curvature Mirror BRDF and texture Curvature Scale

Input Signals • Illumination • environment map Grace cathedral [Debevec 2001] • Reflectance • Acquired isotropic BRDFs • Textures Gold-paint [ Matusik 2003]

Input Signals’ Bandwidth • Illumination • environment map • Purely angular bandwidth Grace cathedral [Debevec 2001] • Reflectance • Acquired isotropic BRDFs • Purely angular bandwidth • Textures • Purely spatial bandwidth Gold-paint [ Matusik 2003]

Computing Local Bandwidth • • Discrete Wavelet Transform to compute bandwidth • environment map and reflectance (BRDF & Texture) • wavelets are well localized both in space and frequency. Discrete Wavelet Transform instead of WFFT Daubechies 4 tap wavelet FFT Space Frequency

Local Bandwidth Examples Grace cathedral [Debevec 2001] Angular bandwidth Gold-paint Angular bandwidth [ Matusik 2003]

Roadmap • • Related work Our technique • • The goal Sparse sampling for reconstruction • • Local light-field definition Spectral Operations 2 D local bandwidth Image-space local bandwidth Adaptive sampling for integration Validation …

Image-Space Bandwidth • • We combine bandwidth from various incident directions to get the final bandwidth at a given pixel. Taking a max is too conservative, we take the weighted average instead. Bandwidth estimate based on max • Bandwidth at each pixel Bandwidth estimate Rendered image

Roadmap • • Related work Our technique • • The goal Sparse sampling for reconstruction Adaptive sampling for integration Validation The rendering algorithm Results and comparisons Extension: MSAA Conclusion / Future work

Variance of the Shading Integrand • # shading samples ∝ sum of bandwidths, weighted by the illumination times reflectance squared. Angular bandwidth of arriving at the surface the reflectance • The sum is a conservative approximation of the actual variance of the shading integrand.

Roadmap • • Related work Our technique • • The goal Sparse sampling for reconstruction Adaptive sampling for integration Validation The rendering algorithm Results and comparisons Extension: MSAA Conclusion / Future work

Validation of the Bandwidth and Variance Estimated bandwidth Measured bandwidth (WFFT) Estimated variance Measured variance

Roadmap • • Related work Our technique • • The goal Sparse sampling for reconstruction Adaptive sampling for integration Validation The rendering algorithm Results and comparisons Extension: MSAA Conclusion / Future work

The Algorithm • Step 1 • Compute g-buffers • Load Illumination and reflectance bandwidth • Step 2 • Estimate bandwidth and variance • Step 3 • Mip-map bandwidth and variance • Step 4 • Shade and up-sample

Video: algorithm

Roadmap • • Related work Our technique • • The goal Sparse sampling for reconstruction Adaptive sampling for integration Validation The rendering algorithm Results and comparisons Extension: MSAA Conclusion / Future work

Result (Final Image)

Result (Bandwidth)

Result (Variance)

Timings • Fast bandwidth/variance estimation • (8 ms) (ms)

Timings • Computation time scales linearly with the total number of shading samples

Comparison with Reference Similar quality (2639 ms) Our algorithm (1015 ms) Similar time (906 ms)

Comparison with Spherical Gaussian Approximation (Wang et al. 2009) Fast but different from ground-truth (notice that color changing effects are missing) Path-traced reference Interactive but more accurate

Roadmap • • Related work Our technique • • The goal Sparse sampling for reconstruction Adaptive sampling for integration Validation The rendering algorithm Results and comparisons Extension: MSAA Conclusion / Future work

Extension Adaptive Multi-Sample Anti-Aliasing

Video: adaptive MSAA

Timings • Sub-linear performance for MSAA

Roadmap • • Related work Our technique • • The goal Sparse sampling for reconstruction Adaptive sampling for integration Validation The rendering algorithm Results and comparisons Extension: MSAA Conclusion / Future work

Conclusion • Interactively shading dynamic geometry with acquired materials • Shade a fraction of pixels using bandwidth • Adaptively sample shading integrals • Combine shaded pixels using up-sampling • Exploiting bandwidth to sub-linearly scale MSAA with deferred shading

Future Work • Local light sources • Better up-sampling algorithm • Depth-of-field • 6 D bandwidth computation for SVBRDFs • Visibility and indirect illumination

Thank you Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction