Realtime HandPainted Graphics for Mobile Games Case Study

Real-time Hand-Painted Graphics for Mobile Games Case Study: Bo children’s book series Fabian Di Fiore, Tom Schaessens, Robin Marx, Frank Van Reeth, Eddy Flerackers Expertise Centre for Digital Media Hasselt University, Belgium

INTRODUCTION 2

Introduction n Many picture books and graphic novels with dynamic and interesting styles n For commercial reasons, publishers create derivative digital products including animation and games n When digitising these books for animation or games n backgrounds and scenes are just static images, which n appear instantaneously or through fading The entertainment industry is asking for alternatives

Motivation n Commercial opportunity as Case Study: Bo children's book series n Bo = transparent piglet n aimed at ages 4 -6 n teach children how to deal with emotions n multiple traditional painting techniques n Recreate scenes from the book the way that the artist made them n unique recreation of the visual style n to visualise the progression of the story/game (as the player continues, more and more is revealed)

Approach n Tools to allow artists to quickly and intuitively create and animate hand-painted scenes n capturing the way an artist creates his picture books n allowing traditional hand-painted strokes to be generated digitally n Real-time animated recreation of hand-painted scenes running on mobile devices n animated recreation of captured scenes (e. g. , stroke by stroke) n optimisation for mobile devices n Integration in existing (game) engine n Unity 3 D

BASIC CONSIDERATIONS 6

Hand-painted Look n Physical simulation

Hand-painted Look n Non-photorealistic Rendering (NPR)

Hand-painted Look n Raster graphics

Hand-painted Look n Raster graphics

Hand-painted Look n Trade-off n Accuracy n indistinguishable from reality n Performance n speed of recreating captured data n Animatability n n ability to animate stroke by stroke drawing from start to finish n Changeability n n flexibility when designing (e. g. , moving drawn strokes) Artistic freedom

Mobile GPU Performance n Immediate Mode Rendering (IMR) n < PC graphics industry n forward facing polygon -> rendered -> shaded -> textured -> Z-buffer -> frame buffer n (most) android devices n Tile-Based Deferred Rendering (TBDR) n < Power. VR architecture n tile based (parallel) + deferred rendering (hidden surface removal) n all Apple mobile devices

Conclusion n Original plan n Raster graphics and TBDR n Bad benchmarking test: ARM + Power. VR architecture n write-to-texture operations not efficient for pixel-based strokes n Mesh-based Strokes n Combination of NPR and raster graphics n n Creation: strokes are flat meshes approximating the contour of brush strokes Colouring using textures: GPU shader program n Allows for fast rendering of meshes on the GPU

creating a hand-painted look IMPLEMENTATION 14

Introduction to the Visual Style of Bo n Main characteristics n Rubber paint roller for background elements n Paper cut stencils for adding details n Scanned paintings for animals and insects n Blending for the final composition

Creating a Base Stroke n Sampling of input points (< digital tablet) n Pressure determines width n Boundaries determine geometry of brush footprint n Fading out of paint n Use uv-coordinates as a length-parameter to determine start and end of the stroke as a shader has no context

Rubber Paint Roller n General fading out supported by base stroke n Background n consist of transparant, slowly fading strokes n new parameters: fade offset + fade multiplier n Unique strokes n create wide texture using two dollops of paint n for each stroke randomly select narrow band from texture

Rubber Paint Roller n Pattern of white dots when rolling back and forth in a fanning pattern n due to the relief created by the paint itself as thin layers of dried paint on the surface of the roller prevent paint in between from touching the paper n Employ a black and white mask n white representing spots where the paint should adhere, black where it doesn’t n add mask as texture and multiply

Grass Brush n Create effect of individual blades of grass n artist: achieved by ripping of a corner of paper, dipping it in paint and dragging it over the canvas n tool: mask with transparent inside and grass-like outside is stretched over the length of the stroke

Other Brushes n Using masks and additional parameters n Oily brush n Stencil roller brush n Stamp brushes

content creation pipeline toolset IMPLEMENTATION 21

Brush Editor n Management of brushes, preview canvas…

Animation Tool n Painting, timeline, recording functionality…

RESULTS 24

Results n 60 strokes n 4500 vertices n 2400 triangles

Results n 150 strokes n 3600 vertices n 2300 triangles

Discussion n i. Pad 2 n Initially frame rate dropped from 60 fps to 5 fps n due to overdraw of transparent strokes: cover significant amount of the screen and, hence, there is much overlap

Discussion n Render-to-texture (RTT) n render to image first i. o. rendering to screen n drawback: background is flattened n RTT for strokes that have been drawn completely and need not to be animated anymore n n do this for every 10 -20 new strokes Consistent frame rate of 60 fps

Video

i. Tunes App Store: “Bo vindt alles”