Lazy Solid Texture Syntehsis Eurographics Symposium on Rendering

Lazy Solid Texture Syntehsis Eurographics Symposium on Rendering 2008 Yue Dong, Sylvain Lefebvre, Xin Tong, George Drettakis

Abstract � We introduce a new algorithm with the unique ability to restrict synthesis to a subset of the voxels, while enforcing spatial determinism ◦ Only a thick layer around the surface needs to be synthesized � Synthesize a volume from a set of precomputed 3 D candidates ◦ Carefully select in a pre-process only those candidates forming consistent triples

Abstract � Runs efficiently on the GPU ◦ Generates high resolution solid textures on surfaces within seconds ◦ Memory usage and synthesis time only depend on the output textured surface area ◦ Our method rapidly synthesizes new textures for the surfaces appearing when interactively breaking or cutting objects

Introduction � Solid textures define the texture content directly in 3 D ◦ Removes the need of a planar parameterization ◦ Unique feeling that the object has been carved out of a block of matter

Previous Work � Implicit Volume ◦ Color = f(x, y, z) ◦ Procedural texturing � Texturing and Modeling: A Procedural Approach � EBERT D. , MUSGRAVE K. , PEACHEY D. , PERLIN K. , WORLEY � Academic Press, 1994 ◦ Spectral analysis � Spectral analysis for automatic 3 d texture generation � GHAZANFARPOUR D. , DISCHLER J. -M. � Computers & Graphics, 1995 � Generation of 3 d texture using multiple 2 d models analysis � GHAZANFARPOUR D. , DISCHLER J. -M. � Computers & Graphics, 1996 � � Low memory usage Limited range of materials

Previous Work � Explicit Volume ◦ Color = g[x, y, z] ◦ Histogram matching ◦ ◦ � Pyramid-Based texture analysis/synthesis � HEEGER D. J. , BERGEN J. R. � SIGGRAPH, 1995 Stereological technique � Stereological techniques for solid textures � JAGNOW R. , DORSEY J. , RUSHMEIER H. � SIGGRAPH, 2004 Neighborhood matching � Texture synthesis by fixed neighborhood searching � WEI L. -Y. � Ph. D thesis, 2002, Stanford University � Aura 3 d textures � QIN X. , YANG Y. -H. � IEEE Transactions on Visualization and Computer Graphics, 2007 � Solid texture synthesis from 2 d exemplars � KOPF J. , FU C. -W. , COHEN-OR D. , DEUSSEN O. , � LISCHINSKI D. , WONG T. -T. � SIGGRAPH, 2007 � � � Good Quality Can synthesis various materials Take long time to compute

Process Overview � Pre-computation ◦ 3 D candidates from 2 D exemplars � Multi-resolution ◦ Upsample ◦ Jitter ◦ Correction pyramid synthesis

Terminology � Pixel : 2 D / Voxel : 3 D � Triple : a set of three 2 D coordinates � Crossbar : a set of pixels which are crossing in three neighborhoods of size N (N = 5)

3 D candidates selection � We select candidate triples following two important properties ◦ A good triple must have matching colors along the crossbar of the three neighborhood �To provide color consistency ◦ A good triple must have a good coherence across all three exemplars �Which is likely to form coherent patches with other neighboring candidates

Color consistency �A suitable candidate should be consistent across the crossbar ◦ Minimize the color difference of the crossbar ◦ Compute L 2 color difference between each pairs ◦ The sum of difference for the three pairs defines a crossbar error CB

Color consistency � In each pixel of each exemplar ◦ Form triples using the pixel itself and two neighborhoods from the other two exemplars ◦ Select the triples producing the smallest crossbar error � To speed up the process ◦ Extract S most-similar pixel strips from each of the two exemplars, using ANN library ◦ Form S 2 triples then take 100 best triples ◦ S = 65

Triples of coherent candidates � Check whether a candidate may form coherent patches in all directions with candidates from neighboring pixels � For each coordinate within a candidate triple ◦ Verify that at least one candidate from a neighboring pixel has a continuous coordinate

Triples of coherent candidates Eyzx p p+1

Triples of coherent candidates – Candidates for Ex � x. Ck[p] – k-th candidate triple for pixel p in Ex � x. Ck[p] – E coordinate of the triple x. Ck[p] y y � x. C

Triples of coherent candidates � Iterate until having no more than 12 candidates per pixel ◦ Typically requires 2 iterations � If more candidates remain select first 12 with the smallest crossbar error � It is possible to have no candidate at all ◦ Rare in practice

Candidate Slab � Candidates are not only useful for neighborhood matching, but also provide a very good initialization for the synthesis process � For each pixel ◦ One 2 D neighborhood is in the plane of the exemplar ◦ Two others are orthogonal to it and intersect along a line of neighborhood size of N voxels

Candidate Slab � To initialize synthesis we create such a slab using the best (first) candidate at each pixel � Using the slab directly as a 3 D exemplar would be very limiting ◦ This would ignore all other candidates ◦ Uses a slab only for initialization

Parallel Solid Synthesis � Extended ‘Parallel Controllable Texture Synthesis’ [SIGGRAPH 2005] � Same overall structure ◦ Upsample ◦ Jitter ◦ Correction

Parallel Solid Synthesis � Contrary to the original scheme we perturb the result through jitter only once, after initialization ◦ If finer control is desired, jitter could be explicitly added after each upsampling step

Initialization � To reduce synthesis time, multi-resolution synthesis algorithms can start from an intermediate level of the image pyramid �A good initialization is key to achieve highquality synthesis � We simply choose one of the candidate slabs and tile it in the volume ◦ Three levels above the finest (Maximum Level L – 3) ◦ Using the candidate slab from the corresponding level

Candidate Slab Random Initialization Slab Initialization

Jitter � To explicitly introduce variations to generate variety in the result � Perturb the initial result by applying a continuous deformation, similar to a random warp

Jitter �J – Jittered Volume � v – Voxel coordinate � ci – Random point in space � di – Nomalized Random direction � G = 200 � Ai = 0. 1 ~ 0. 3 � σi = 0. 01 ~ 0. 05

Jitter � It is important for Ai to have larger magnitude with smaller σi ◦ Adds stronger perturbation at small scales, while adding subtle distortions to coarser scales ◦ Small scale distortions are corrected by synthesis, introducing variety � The overall magnitude of the jitter is directly controllable by the user

Upsample � Each of the eight child volume cells inherits three coordinates from its parent, one for each direction

Correction � Performed on all synthesized voxels simultaneously, in parallel � We compute a color by averaging the corresponding three colors from the exemplars � We visit each of its direct neighbors, and use the stored coordinate triples to gather the candidate sets

Candidate Set � Px – 3 x 2 matrix transforming a 2 D offset from Ex to volume space

Correction � Search for the best matching candidate by distance between voxel neighborhood and 3 D candidate � Distance is measured by L 2 norm on color differences ◦ Can use PCA projection to speed up the process � Replace the triple with best matching candidate ◦ triples have been pre-computed and optimized to guarantee that the color disparity between the three colors in each voxel is low � Two correction pass for every level ◦ Using sub-pass mechanism of PCTS ◦ 8 pass sub-pass

Correction � We gather 12 candidates from the 33 = 27 direct neighbors, for a total of 324 candidates per voxel ◦ Too many candidates � Search for the best matching 2 D candidates in each of the three directions then gather the 3 D candidates only from these three best matching pixels ◦ Still a lot ◦ In practice we keep 4 2 D and 12 3 D candidates per exemplar pixel at coarse levels � 27× 4 = 108 2 D candidates � 3× 12 = 36 3 D candidates ◦ 2 2 D and 4 3 D candidates at the finest level

Lazy Subset Synthesis � Determine the entire dependency chain throughout the volume pyramid from a requested set of voxels to synthesize the smallest number of voxels ◦ Compute a synthesis mask Shape - dilation of the mask by the shape of the neighborhoods � Masklp⊗Neighborhood

Lazy Subset Synthesis � To compute a single voxel, with N = 5, 2 passes and synthesis of the 3 last levels, our scheme requires a dependency chain of 6778 voxels ◦ The size of the dependency chain grows quadratically with the number of passes

Implementation and Results � Entirely in software and using the GPU to accelerate the actual synthesis � Intel Core 2 6400 (2. 13 GHz) CPU and an NVIDIA Ge. Force 8800 Ultra � We sometimes add feature distance

Candidate pre-computation � Most results in the paper are computed from a single example image repeated three times ◦ Pre-computed candidates may be shared depending on the orientation chosen for the image ◦ Typically 7 seconds for 642 exemplars ◦ 25 to 35 seconds for 1282 exemplars ◦ Includes building the exemplar pyramids, computing the PCA bases and building the candidate sets ◦ 231 KB memory required for a 642 exemplar

GPU implementation � Implemented in fragment shaders, using the Open. GL Shading Language ◦ Unfold volumes in tiled 2 D textures, using three 2 channel 16 bit render targets to store the synthesized triples ◦ Pre-compute and reduce the dimensionality of all candidate 3 -neighborhoods using PCA, keeping between 12 and 8 dimensions �Keep more terms at coarser levels since less variance is captured by the first dimensions ◦ Quantize the neighborhoods to 8 -bits to reduce bandwidth �Stored in RGBA 8 bit textures

GPU implementation � In order to minimize memory consumption, we perform synthesis into a Tile. Tree data structure ◦ LEFEBVRE S. , DACHSBACHER C. In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3 D ◦ Graphics and Games (2007)

GPU implementation � When interactively cutting an object, synthesis occurs only once for the newly appearing surfaces ◦ Tile-Tree cannot be updated interactively �Store the result in a 2 D texture map for display ◦ Our implementation only allows planar cuts �new surfaces are planar and are trivially parameterized onto the 2 D texture synthesized when the cut occurs

Full volume synthesis and comparisons

Full volume synthesis and comparisons

Full volume synthesis and comparisons � 7. 22 seconds for synthesizing the 643 volume from 642 exemplar ◦ 7 seconds for pre-computation and 220 milliseconds for synthesis ◦ Memory requirement during synthesis is 3. 5 MB � 28. 7 seconds for synthesizing the 1283 volume from 1282 exemplar ◦ 27 seconds for pre-computation and 1. 7 seconds for synthesis ◦ ‘Solid texture synthesis from 2 d exemplars’ [SIGGRAPH 2007] takes 10 to 90 minutes

Solid synthesis on surfaces

Solid synthesis on surfaces � 4. 1 seconds (dragon) to 17 seconds (complex structure), excluding pre-computation � Storage of the texture data requires between 17. 1 MB (statue) and 54 MB (complex structure) ◦ The equivalent volume resolution is 10243 which would require 3 GB � Slower than state-of-the-art pure surface texture synthesis approaches ◦ But inherits all the properties of solid texturing

Solid synthesis on surfaces � On demand synthesis when cutting or breaking objects (Fig. 10) ◦ Resolution of 2563 �Initially requires 1. 3 MB ◦ The average time for synthesizing a 2562 texture for a new cut is 8 ms ◦ Synthesizing a 2562 slice of texture content requires 14. 4 MB �Due to the necessary padding to ensure spatial determinism

Comparison with a simple tiling

Comparison with a method using standard 2 D candidates

Comparison with a method using standard 2 D candidates � We also implemented our synthesis algorithm using only standard 2 D candidates ◦ Takes roughly twice the number of iterations to obtain a result of equivalent visual quality ◦ Due to the increased number of iterations, the size of the dependency chain for computing a single voxel grows from 6778 voxels with 3 D candidates to 76812 voxels with 2 D candidates �A factor of 11. 3 in both memory usage and speed

Limitations

Conclusion �A new algorithm for solid synthesis ◦ with the unique ability to restrict synthesis to a subset of the voxels, while enforcing spatial determinism ◦ Synthesize a volume from a set of pre-computed 3 D candidates ◦ GPU implementation is fast enough to provide on demand synthesis when interactively cutting or breaking objects