Advanced 3 D graphics for movies and games

Advanced 3 D graphics for movies and games (NPGR 010) – Rendering equation and its solution Jiří Vorba, MFF UK/Weta Digital jirka@cgg. mff. cuni. cz Slides of prof. Jaroslav Křivánek, minor edits by Jiří Vorba

Goal: Global illumination (GI) Direct illumination Global = direct + indirect 2

Review: Reflection equation Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Review: Reflection equation n If the shading point x itself is on an emitting surface (i. e. if x is on a light source): Emitted radiance Total outgoing radiance Reflected radiance (previous slide) Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

From local reflection to global light transport n Where does the incoming radiance Lin(x, win) come from, really? q From other places in the scene! Ray casting function Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

From local reflection to global light transport n Plug for Lin into the reflection equation n Incoming radiance Lin drops out n Outgoing radiance Lout at x is described in terms of Lout at other points in the scene Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Rendering equation n Removed the subscript “out” from the outgoing radiance for brevity The RE describes the steady state = energy balance in the scene Rendering = calculate L(x, wout) for all points visible through pixels, such that it fulfils the rendering equation Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Reflection equation vs. Rendering equation Similar form – different meaning n Reflection equation q q Describes local light reflection at a single point Integral that can be used to calculate the outgoing radiance if we know the incoming radiance n Rendering equation q q Condition on the global distribution of light in scene Integral equation – unknown quantity L on both sides Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Rendering Equation – Kajiya 1986 Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Path tracing sketch

Recursive unwinding of the RE n To calculate L(x, wout), we need to calculate L(ray(x, win), - win) for all directions win around x. n At each intersected point, we need to do the same recursively. Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Path tracing, v. 0. 1 estimate. Lin (x, ω): // radiance incident at x from direction ω y = find. Nearest. Intersection(x, ω) if (no intersection) return backgroud. get. Le (–ω) // emitted radiance from envmap else return get. Le (y, –ω) + // emitted radiance (if y is on a light) estimate. Lrefl (y, –ω)// reflected radiance estimate. Lrefl(x, ωout): [ωin , pdf] = gen. Random. Dir(x, ωout); // e. g. BRDF imp. sampling return estimate. Lin(x, ωin) * brdf(x, ωin, ωout) * dot(nx, ωin) / pdf Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

n Let’s now argue more rigorously why the recursive path tracing actually solves the rendering equation… Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

The operator form of the RE

RE is a Fredhom integral equation of the 2 nd kind General form of the Fredholm integral equation of the 2 nd kind unknown functions equation “kernel” Rendering equation: Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Linear operators n Linear operators act on functions q (as matrices act on vectors) n The operator is linear if the “acting” is a linear operation n Examples of linear operators Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Transport operator n Rendering equation Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Solution of the RE in the operator form n Rendering equation n Formal solution n unusable in practice – the inverse cannot be explicitly calculated Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Expansion of the rendering equation n Recursive substitution L n n-fold repetition yields the Neumann series Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Expansion of the rendering equation n Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

A different derivation of the Neumann series Formal solution of the rendering equation n Proposition n Proof n Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Rendering equation n Solution: Neumann series Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Progressive approximation Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Progressive approximation n Each application of T corresponds to one step of reflection & light propagation one-bounce indirect illumination emission direct illumination Open. GL shading Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015 two-bounce indirect illumination

Contractivity of T n Holds for all physically correct models q Follows from the conservation of energy n It means that repetitive application of the operator lower the remaining light energy (makes sense, since reflection/refraction cannot create energy) n Scenes with white or highly specular surfaces q q reflectivity close to 1 to achieve convergence, we need to simulate more bounces of light Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Alright, so what have we achieved? Rendering equation Solution through the Neumann series n n n We have replaced an integral equation by a sum of simple integrals Great, we know how to calculate integrals numerically (the Monte Carlo method), which means that we know how to solve the RE, and that means that we can render images, yay! Recursive application to T corresponds to the recursive ray tracing from the camera Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Paths vs. recursion: Same thing, depends on how we look at it n Paths in a high-dimensional path space n Recursive solution of a series of nested (hemi)spherical integrals: Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Recursive unwinding of the RE n To calculate L(x, wout), we need to calculate L(ray(x, win), - win) for all directions win around x. n At each intersected point, we need to do the same recursively. We’ve seen this already, right? But unlike at the beginning of the lecture, by now we know this actually solves the RE. Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Path tracing, v. 2012, Arnold Renderer © 2012 Columbia Pictures Industries, Inc. All Rights Reserved.

Angular and Area form of the rendering equation

Angular integral form of the RE n Integral over the hemisphere in incoming directions Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Going from angular to area form n Change-of-variables applied to the angular form Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Area integral form of the RE Area form n q Integral over the scene surface geometry term Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015 visibility 1 … y visible from x 0 … otherwise

Angular integral form n Sum up radiance contributions from all directions n For each direction, find the nearest surface by ray tracing n Implementation in stochastic path tracing: q For a given x, generate random direction(s), for each find the nearest intersection, return the outgoing radiance at that intersection and multiply it with the cosine-weighted BRDF. Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Area integral form n Sum up contributions from all other points on the scene surface q n Implementation in stochastic path tracing: q n Contribution added only if visible Generate randomly point y on scene geometry (e. g. on a light source). Test visibility between x and y. If mutually visible, add the outgoing radiance at y modulated by the geometry factor. Typical use: direct illumination calculation for area light sources Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Most rendering algorithms = (approximate) solution of the RE n Local illumination (Open. GL) q q n Only point sources, integral becomes a sum Does not calculate equilibrium radiance, is not really a solution of the RE Finite element methods (radiosity) [Goral, ’ 84] q q q Discretize scene surface (finite elements) Disregard directionality of reflections: everything is assumed to be diffuse Cannot reproduce glossy reflections Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Source: Wikipedia Radiosity Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Most rendering algorithms = (approximate) solution of the RE n Ray tracing [Whitted, ’ 80] q Direct illumination on diffuse and glossy surfaces due to point sources q Indirect illumination only on ideal mirror reflection / refractions q Cannot calculate indirect illumination on diffuse and glossy scenes, soft shadows etc. … n Distributed ray tracing [Cook, ’ 84] q Estimate the local reflection using the MC method q Can calculate soft shadows, glossy reflections, camera defocus blur, etc. Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Most rendering algorithms = (approximate) solution of the RE n Path tracing [Kajiya, ’ 86] q q q True solution of the RE via the Monte Carlo method Tracing of random paths (random walks) from the camera Can calculate indirect illumination of higher order Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

The Umbrella Academy 2 | Copyright: Netflix 2020

From the rendering equation to finite element radiosity (Optional material)

From the RE to radiosity n Start from the area integral form of the RE n The Radiosity method– assumptions q q Only diffuse surfaces (BRDF constant in win and wout) Radiosity (i. e. radiant exitance) is spatially constant (flat) over the individual elements Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

From the RE to radiosity Diffuse surfaces only n q The BRDF is constant in win and wout q Outgoing radiance is independent of wout. It is equal to radiosity B divided by p Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

From the RE to radiosity n Spatially constant (flat) radiosity B of the contributing surface elements Radiosity of the j-th element Geometry factor between surface element j and point x Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

From the RE to radiosity n Spatially constant (flat) radiosity of the receiving surface element i: q Average radiosity over the element … form factor Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Classic radiosity equation n System of linear equations n Form factors n Conclusion: the radiosity method is nothing but a way to solve the RE under a specific set of assumptions Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015

Radiosity method n Classical radiosity 1. 2. n Stochastic radiosity q q n Form facto calculation (Monte Carlo, hemicube, …) Solve the linear system (Gathering, Shooting, …) Avoids explicit calculation of form factors Monte Carlo method Radiosity is not practical, not used anymore q q Scene subdivision -> sensitive to the quality of the geometry model (but in reality, models are always broken) High memory consumption, complex implementation Advanced 3 D Graphics (NPGR 010) - J. Vorba 2020, created by J. Křivánek 2015