NPR Today Stylized Rendering Techniques for Scalable RealTime

NPR Today • “Stylized Rendering Techniques for Scalable Real-Time 3 D Animation”, Adam Lake, Carl Marshall, Mark Harris and Marc Blackstein, NPAR 2001 • "Computer-Generated Pen-and-Ink Illustration of Trees", Oliver Deussen and Thomas Strothotte, SIGGRAPH 2000 • "Sketchy Drawings", Marc Nienhaus and Jurgen Dollner, Afri. Graph 2004 03/30/05 © 2005 University of Wisconsin

Stylized Rendering Techniques for Scalable Real. Time 3 D Animation Paper Overview Leo Chao

Paper Information • Published – NPAR 2001 • (Non-Photorealistic Animation and Rendering) – Authors • • Adam Lake Carl Marshall Mark Harris Marc Blackstein • Graphics Algorithms and 3 D Technologies Group – Intel Architecture Labs • UNC – Chapel Hill 03/30/03 © 2005 University of Wisconsin

Paper Information • Prior Work – No novel techniques, adapts from prior works to hardware • Motion Lines – Maic Masuch. “Speedlines: Depicting Motion in Motionless Pictures. ” ACM SIGGRAPH 99 Technical sketch. – Siu Chi Hsu and Irene H. H. Lee. "Drawing and Animation using Skeletal Strokes". In Proceedings of ACM SIGGRAPH 94, pages 109 -118. 1994. • Silhouette Edges – Lee Markosian, Michael Kowalski, Samuel Trychi, Lubomir Bourdev, Daniel Goldstein, and John Hughes. Real-Time Nonphotorealistic Rendering. In Proceedings of ACM SIGGRAPH 97, pages 113 -122. 1997. • Cartoon Shading – Philippe Decaudin. Rendu de scenes 3 D imitant le style dessin anime. Rapport 03/30/03 © 2005 University of Wisconsin de Recherche 2919, Institut National de Recherche en Informatique et en

Contributions • Hardware Implementation of NPR Techniques – Stylistic Shading • Cartoon Shading • Pencil Sketch Shading – Stylistic Inking • Silhouette Edge Detection • Stylized Silhouette Rendering – Motion Lines 03/30/03 © 2005 University of Wisconsin

Stylistic Shading • Fundamentals – Preprocess colour textures – Render as normal – Except: • Modify lighting/shading calculations • May require texture coordinate processing • “Painter” – Modified Shading Step – Cartoon Shading – Pencil Sketch Shading 03/30/03 © 2005 University of Wisconsin

Stylistic Shading • Cartoon Shading – Reducing detail aids comprehension – No Smooth Shading

Stylistic Shading 03/30/03 © 2005 University of Wisconsin

Stylistic Shading ALGORITHM Cartoon Shade Preprocess: 1. Calculate the illuminated diffuse color for each material: 2. Calculate the shadowed diffuse color: 3. For each material, create a texture with two colors - u=1 is coloured Ci - u=0 is coloured Cs Ci Vertex Color Cs Shadow Diffuse Color al Light Ambient dl Light Diffuse am Material Ambient dm Material Diffuse 03/30/03 © 2005 University of Wisconsin ag Coefficient of Global Ambient Lighting

Stylistic Shading ALGORITHM Cartoon Shade Runtime: 1. Calculate the one-dimensional texture coordinate at each vertex 2. Render the model using a standard graphics API - Lighting disabled - Enable texture mapping L normalized light vector 03/30/03 © 2005 University of Wisconsin n unit surface normal

Stylistic Shading • Pencil Sketch Shading – Extends cartoon shading to 2 D textures – Extends cartoon shading to multiple textures – Multiple textures to represent light density – Associate texture coordinates • Project texture from the viewport onto the model 03/30/03 © 2005 University of Wisconsin

Stylistic Inking • Fundamentals – Identify “important” edges • Silhouette • Border • Crease – Modify drawing of these edges • “Inker” – Modified Line Drawing – Visually important edge detection – Stylized edge drawing 03/30/03 © 2005 University of Wisconsin

Stylized Inking • Visually important edge detection – Silhouettes • Front-Face and Back-Face share an edge – Borders • Edge shared by two materials • Unshared polygon edge – Creases • Large dihedral angle between polygons 03/30/03 © 2005 University of Wisconsin

Stylized Inking 03/30/03 © 2005 University of Wisconsin

Stylized Inking ALGORITHM SED Preprocess: 1. Create edge-list structure with Edge Type flags a. Create a unique edge list using a hash table. b. Edge Hash Function: sum the two vertex indices c. Border Flag - Edges with only one neighboring face - Two neighboring faces with different materials d. Set crease flags - Non-deformable geometry - Dihedral angle of two faces > threshold 03/30/03 © 2005 University of Wisconsin

Stylized Inking ALGORITHM SED Runtime: 1. For deformable meshes: a. Calculate face normals b. Detect crease edges from face normals c. Set Crease Edge flags 2. Detect silhouette edges a. 03/30/03 © 2005 University of Wisconsin

Stylized Inking • Stylized “Important” Lines – Emphasis on these lines – Option 1: Straight Line • Fast/Easy – Option 2: Texture Maps • • 03/30/03 “Artistic” Style Curvature driven “strokes” Involves a graph driven search This is a HACK © 2005 University of Wisconsin

Motion Lines • Gives the appearance of motion • For fast moving objects •

Performance • Claim: Maintain Real-Time Speeds – 30 fps, in 2001 • Tests: –

Performance 03/30/03 © 2005 University of Wisconsin

Future Work • Inker – Better “artistic” lines • Painter – Pencil sketch “doesn’t look right” – Follow surface gradient perhaps • Incorporate other NPR techniques – Those used by artists – Doesn’t say which would amenable 03/30/03 © 2005 University of Wisconsin

Questions? • ques·tion (kw s ch n) n. – An expression of inquiry that invites or calls for a reply. • heck·le (h k l) tr. v. – To try to embarrass and annoy (someone speaking or performing in public) by questions, gibes, or objections 03/30/03 © 2005 University of Wisconsin

“Computer-Generated Pen-and-Ink Illustration of Trees” (SIGGRAPH 2000) Oliver Deussen Thomas Strothotte 03/30/03 © 2005

Motivation • The paper presents a method for creating pen-and-ink-like illustrations from detailed geometric models. • Motivated by the need for illustrative representations of plant life in architectural and landscape renderings. • We want to be able to produce images similar to this: 03/30/03 © 2005 University of Wisconsin

Related Work • “Cartoon Tree” – Alvay Ray Smith – Use small disks to represent bunches of leaves • Reeves and Blau – Use structured particle system to represent realistic trees. • Kowalski et al. – Two step process 1. Render shaded primitives that approximate foliage 2. Overlay graftals on rendered image using Salisbury’s line drawing method 03/30/03 © 2005 University of Wisconsin

Related Work Examples • Smith’s “cartoon tree”: • Reeves “particle forest”: • Kowalski’s Graftal

Traditional Illustration of Trees • The Skeleton is drawn up to the second level of branching. • Foliage is broken up into three parts: 1. The directly lit canopy – represented by outlines and scarce inner detail 2. The half shadow – greater detail provides an overall darker grey level 3. The shaded area – achieved by thicker lines or drawing leaves in black • Artists will general use a combination of these to draw the leaves. Rarely do they use all three. 03/30/03 © 2005 University of Wisconsin

Drawing the Skeleton • The tree skeleton is represented by generalized cylinders. • Analytical Silhouette Algorithms – Markosian et al. – Rakar and Cohen – Depth Differences • The skeleton is shaded with crosshatch marks. Marks could be placed using the algorithm outlined by Salisbury et al. – The authors actually use a variant of Floyd-Steinberg diffusion that places strokes when the accumulated grey value is above a threshold. 03/30/03 © 2005 University of Wisconsin

Drawing the Foliage • The xfrog modeling system can generate models with 200, 000 leaves on a tree. We want to pair this down to 10, 000 or so. • Leaves are represented abstractly by particles: a location and an orientation vector. • The goal is to illustrate groups of with a set of strokes. • Idea: Render particles as primitives and use depth buffer information to determine which leaves are spatially close and outline them. 03/30/03 © 2005 University of Wisconsin

Depth Differences • Initially, each particle is rendered with a view-facing primitive, such as a disk. • Basic Algorithm: – For each pixel compute the maximum positive difference between it and its neighbors • If the distance is above the given threshold, mark as an edge. 03/30/03 © 2005 University of Wisconsin

Depth Differences Continued… • The threshold and primitive size can be varied to achieve different effects: Primitive size: 0. 15 Primitive size: 0. 70 Threshold: 1000 Threshold: 2000 • The threshold can be varied using other buffers, achieving greater detail in areas that results in a shadowing effect: 03/30/03 © 2005 University of Wisconsin

Abstract Drawing Primitives • More realistic leaves can be achieved by using the particles’ orientation vectors to interpolate between a set of polygons that represent leaves from different views 03/30/03 © 2005 University of Wisconsin

Level-of-Abstraction • Two Approaches 1. Convert raw z-buffer values from non-linear, projected values to linear values using formula below and vary threshold according to depth: d: depth value between 0 and 1 d 0: minimal depth value (e. g. 0) d 1: maximal depth value (e. g. 65535) z 0: depth value of near clipping plane z 1: depth value of far clipping plane 2. Take advantage of the non-linearity to automatically generate more abstract representations. – – Geometry farther away will be pushed into fewer values in the zbuffer. A tree in the background will represented by fewer strokes than the same tree in the foreground. 03/30/03 © 2005 University of Wisconsin

Level of Abstraction Continued… • The level-of-abstraction can be furthered by scaling the primitives so that the amount of screen space they occupy is independent of their depth. r: scaling factor d: desired screen space s: actual screen space No Scaling 03/30/03 Scaled © 2005 University of Wisconsin

Software Implementation 1. Calculate depth differences and mark pixels above threshold. • In the interactive mode this bitmap is directly used and blended with other geometry. 2. Generate shadows in software. Large number of isolated surfaces precludes hardware implementation. 3. Vectorize bitmap using least squares fitting or primitive identification method. • Draw lines using spline interpolation • Apply line styles 03/30/03 © 2005 University of Wisconsin

Results 03/30/03 © 2005 University of Wisconsin

Future Work • Apply cross-hatch shading to the leaves • Create a continuous level-of-detail algorithm that stays performance while getting rid of artifacts that come with discrete representations. • Apply new styles and color for cartoons. • Questions? 03/30/03 © 2005 University of Wisconsin

Sketchy Drawings Marc Niehaus University of Potsdam 03/30/05 Jürgen Döllner University of Potsdam ©

Sketchy Drawings • Sketchiness shows unfinished state • Useful for architectural and product design

Previous Work • • G-Buffers (Saito & Takahashi 1990) Edge enhancement (Nienhaus and Döllner 2003) Loose and Sketchy filter (Curtis 1998) Artistic Strokes (Northrup & Markosian 2000) 03/30/05 © 2005 University of Wisconsin

Algorithm Overview • Obtain edge and shading data • Perturb this data to get

G-Buffers • • • Need to access in real time Depth/Color/Alpha/Stencil Buffer on graphics card Textures Current hardware allows multiple textures Manipulate with fragment and pixel shaders 03/30/05 © 2005 University of Wisconsin

Edge Map • Need to find edges for most NPR algorithms • Store three types of edges – Silhouette – Crease – Border • See Edge Enhancement (Nienhaus & Döllner 2003 )for more details 03/30/05 © 2005 University of Wisconsin

Shade Map • Render unlit 3 D geometry to capture flat colors • Store

Adding Uncertainty • • • Apply uncertainty function to edge and shade maps Store function in texture Use Perlin function to get frame to frame consistency Multiply uncertainty values with 2 x 2 matrix Use different matrix for edge and shade maps to get a more sketchy feel 03/30/05 © 2005 University of Wisconsin

End Result • Use shaders to combine results into texture • Render texture on

Style Variations • Render edges multiple times • Roughen edge and surface colors 03/30/05

Issues • Depth Information • Shower-Door Effect 03/30/05 © 2005 University of Wisconsin

Updating Depth Info • Depth info is lost when rendering 3 D geometry to 2 D textures • Solution: Render geometry into a depth map • Render perturbed depth sprite – use shaders to update depth buffer • Need to account for both edge and shade map – take min value and use this for depth 03/30/05 © 2005 University of Wisconsin

Shower-Door Effect • • • When model moves, edges tend to swim around Need to account for geometrical data Also want to keep uncertainty past geometry edges Solution: Render geometry info into another G-Buffer Use geometry as a parameter to the noise function 03/30/05 © 2005 University of Wisconsin