RealTime Relighting Digital Image Synthesis YungYu Chuang 1102008

Real-Time Relighting Digital Image Synthesis Yung-Yu Chuang 1/10/2008 with slides by Ravi Ramamoorthi, Robin Green and Milos Hasan

Realistic rendering • We have talked about photorealistic rendering for complex materials, complex geometry and complex lighting. They are realistic but slow.

Real-time rendering • Its goal is to achieve interactive rendering with reasonable quality. It’s important in many applications such as games, visualization, computer-aided design, …

Real-Time relighting • Lighting is the process of adjusting lights. It is an important but time-consuming step in animation production pipeline. • Relighting algorithms for two kinds of lights – Distant environment lights – Near-field lights for production

Relighting algorithms for distant environment lights

Natural illumination People perceive materials more easily under natural illumination than simplified illumination. Images courtesy Ron Dror and Ted Adelson

Natural illumination Rendering with natural illumination is more expensive compared to using simplified illumination directional source natural illumination

Reflection maps Blinn and Newell, 1976

Environment maps Miller and Hoffman, 1984

HDR lighting

Examples of complex environment light

Examples of complex environment light

Direct lighting with complex illumination p q

Function approximation • G(x): the function to approximate • B 1(x), B 2(x), … Bn(x): basis functions • We want • Storing a finite number of coefficients ci gives an approximation of G(x)

Function approximation • How to find coefficients ci? – Minimize an error measure • What error measure? – L 2 error • Coefficients

Function approximation • Basis Functions are pieces of signal that can be used to produce approximations to a function

Function approximation • We can then use these coefficients to reconstruct an approximation to the original signal

Function approximation • We can then use these coefficients to reconstruct an approximation to the original signal

Orthogonal basis functions • Orthogonal Basis Functions – These are families of functions with special properties – Intuitively, it’s like functions don’t overlap each other’s footprint • A bit like the way a Fourier transform breaks a functions into component sine waves

Integral of product

Basis functions • Transform data to a space in which we can capture the essence of the data better • Spherical harmonics, similar to Fourier transform in spherical domain, is used in PRT.

Real spherical harmonics • A system of signed, orthogonal functions over the sphere • Represented in spherical coordinates by the function where l is the band m is the index within the band

Real spherical harmonics

Reading SH diagrams This direction – Not this direction +

Reading SH diagrams This direction – Not this direction +

The SH functions

The SH functions

Spherical harmonics

Spherical harmonics m 0 l 1 2 -2 -1 0 1 2

SH projection • First we define a strict order for SH functions • Project a spherical function into a vector of SH coefficients

SH reconstruction • To reconstruct the approximation to a function • We truncate the infinite series of SH functions to give a low frequency approximation

Examples of reconstruction

An example • Take a function comprised of two area light sources – SH project them into 4 bands = 16 coefficients

Low frequency light source • We reconstruct the signal – Using only these coefficients to find a low frequency approximation to the original light source

Harr wavelets • Scaling functions (Vj) • Wavelet functions (Wj) • The set of scaling functions and wavelet functions forms an orthogonal basis

Harr wavelets

Example for wavelet transform • Delta functions, f=(9, 7, 3, 5) in V 2

Wavelet transform • V 1, W 1

Example for wavelet transform • V 0, W 0 , W 1

Example for wavelet transform

Quadratic B–spline scaling and wavelets

2 D Harr wavelets

Example for 2 D Harr wavelets

Applications 19% 5% L 2 3% 10% L 2 1% 15% L 2

Relighting algorithms for animation production

Relighting for production • Lighting is a time-consuming process. • Artists adjust lighting parameters and wait for a couple of hours or days to get feedback. • Local shading with complex scene and many lights • Interactive relighting – – Interative visual eedback Fixed scene and camera Lower quality Scalable with sene complexity and number of lights

Deep framebuffer • Gershbein and Hanrahan, SIGGRAPH 2000

Deep framebuffer

Deep framebuffer

LPICS • Pixar, SIGGRPH 2005. A practical realization for the deep framebuffer approach on GPUs LPICS 0. 1 s Final renderer 2, 000 s video

Lightspeed • ILM, SIGGRAPH 2007 • An even more practical system with automatic shader conversion. (2. 7 s v. s. 57 m)

Direct-to-indirect transfer • Hasan et. al. SIGGRAPH 2006 • Deep framebuffer approaches only support local shading, but not indirect lighting With indirect lighting

Concept • Distribute gather samples on scene surfaces

Concept • Direct illumination on both gather samples and view samples

Concept • Inter-reflections between gather samples

Concept • Final gather on view samples

Inter-reflections between gather samples gather sample

Inter-reflections between gather samples • Assume all gather samples are diffuse

Inter-reflections between gather samples

Inter-reflections between gather samples

Final gathering

Concept Direct on view Transfer matrix Final Direct on gather Indirect on view

Scene: Still Life Precomputation: 1. 6 hours 11. 4 – 18. 7 fps Polygon: 107 k

Scene: Temple Precomputation: 2. 5 hours 8. 5 – 25. 8 fps Polygon: 2 M

Scene: Hair Ball Precomputation: 2. 9 hours 9. 7 – 24. 7 fps Polygon: 320 k

Scene: Sponza Atrium Precomputation: 1. 5 hours 13. 7 – 24. 9 fps Polygon: 66 k

Comparison DTI: 8 -25 fps (2. 5 hr precomputation) Monte Carlo path tracer: 32 hours