1 Converting 3 D Facial Animation with Gouraud

1 Converting 3 D Facial Animation with Gouraud shaded SVG A method to extract 3 D animated files and to convert it in SVG allowing Gouraud shading Michel Gaudry, Thomas Di Giacomo, Nadia Magnenat-Thalmann MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005

2 Outline • Introduction – Context – Purpose • Related work – 3 D on mobile devices – Conversion of 3 D to other media • 3 D to SVG converter – – – Context and requirements Process Structure 2 D data extraction Generation of SVG • Optimization • Gouraud shading • Experiments and evaluation – The pyramid – The talking head • Conclusion MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005

3 Introduction • Context and motivation – Developement of wearable devices – User's growing interest for web graphics and 3 D – Hardware limitations – Increasing availability of SVG for web and mobiles devices – Possibility to convert 3 D animated faces to SVG MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005

4 Introduction • Purpose – Extend the authoring of SVG • Enable new SVG authoring by creating scenes in a 3 D environment and converting it in SVG • Extend SVG rendering with motion captured sequence or highly realistic 3 D rendering • Extends availlability of 3 D graphics with other medias – Generate animated and Gouraud shaded SVG to simulate 3 D scenes • Extend 3 D possibilities on mobile devices • Method to convert generic 3 D to SVG • Specific for large 3 D scenes and focus on face animation and deformation • Simulation of light shading with Gouraud MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005

5 Related work • 3 D on mobile devices – Adaptations • Hybrid approaches • VRML • Simplification of 3 D – Conversion of 3 D to other media • To image and video – Current growing technics • Java 2 Micro Edition and JSR-184 • Open. GL ES • X 3 D • 3 D with SVG – Advantages – Conversion from other languages MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005

6 3 D to SVG Converter • Purpose – Convert 3 D animations – Animation data from external file – Allow animation optimization and Gouraud shading • Main structure – Open a 3 D animation – Identification and extraction of 2 D data – Generation of SVG • Optimized animation • Gouraud shaded image MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005

7 Context and requirements • Formats – VRML chosen as input – FAP for face deformation • Animation inputs – User actions – Predefined • Not additional software or scripts • Preserve – Pixel spatial information – Pixel color information • Gouraud shading – Non animated objects • Real time conversion – Ease authoring of SVG animation with 3 D MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005

8 Structure • 3 main modules – Extraction – Animation – Generation • User interface • 2 optional generation modules 3 D rendering with Open GL GUI manager User menu VRML FAP Extraction Animation & record Generation Optimization Gouraud MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005

9 Process • 3 D to SVG – Extraction from VRML file – Get dynamic data from user or file input • 2 D data – 2 D mesh initialized with a list of path and colors per triangles – Conversion of triangles animation with gradient, position and visibility • SVG generator – Pre-generation of SVG file – Generation MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005

10 2 D data extraction • Algorithm – Initialization of • the 2 D "copy": topology, index, colors • static data: camera parameters – Update and Extraction • • 3 D vertices dynamic positions Culling of 3 D faces Projection of 3 D to viewpoint plane Storage of computed 2 D coordinates MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005

11 2 D data extraction • Gouraud shading – Method – In SVG • Projection • Visibility MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005

12 Generation of SVG • From application cache • Animation <svg> <g transform="translate(X, X) scale(X, X) rotate(X)"> <!-- List of animated colored path --> <!-- Each path is a 3 D face -->. . . <!-- Close path list --> </g> <!– close 2 D viewpoint --> </svg> MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005

13 Generation of SVG • To represent triangles – <polygon> – <path> <!-- Animated path from triangle t --> <path style="fill: rgb(c 1, c 2, c 3)"> <animate attribute. Name="d" begin="abs" dur="ass" fill="freeze" vallues=" M x 00 y 00 L x 10 y 10 L x 20 y 20 z; • Coordinates • Visibility • Filling – Single color – Gradients . . . M x 0 n y 0 n L x 1 n y 1 n L x 2 n y 2 n z "/> <set attribute. Name="visibility" to="hidden" begin="hb 1 s" dur="hs 1 s"/> <!--. . . --> <set attribute. Name="visibility" to="hidden" begin="hbn s" dur="hsn s"/> </path> MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005

14 Generation of SVG • Optimization – Problem • Generated files can be big – Module • Can be integrated in an application or can be used independently to optimize any SVG animation file • Processed before or after generation • Complementary to gzip compression – Methods • Reduce length of static sequences • Remove invisibles sequences MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005

15 Generation of SVG • Optimization methods – Reduce static sequences • Remove repeated coordinates when the polygon is not moving – Remove invisible sequences • Remove the <set> tags and cut main animation in parts – Others possibilities • Use relative coordinates • Simplify animation path MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005

16 Generation of SVG • Gouraud shading – Purpose • Improve realism, from flash-shaded triangles to interpolated colors vertex – Method • Adaptation of the method presented in « Interactive 3 D Viewer Written in SVG » (SVG Open 2003) • In large scale • For a 3 D converted mesh – Gouraud with SVG • Use of gradients and filters • Superposition of triangles Gradient area Alpha value : 100% – Gradients • 3 gradients per triangle, one from each vertex • Blending of the gradients with <transparent> attribute • Filters – <fecomposite> – <fecolormatrix> MIRALab Where Research means Creativity Gradient’s path Triangle area University of Geneva www. miralab. ch Alpha value : 0% SVG Open 2005

17 Generation of SVG fe. Color. Matrix • Gouraud shading T 2 path – Structure of a triangle g Element canvas T 1 T 3 T 1 <svg> path T 4 path T 1 Sub-triangle MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005

18 File structure <defs> • Header – SVG – Viewport • Definitions – Filters – Triangles – Gradients <filter id="color. Add"> <fe. Composite in="Source. Graphic" in 2="Background. Image" operator="arithmetic" k 1="0" k 2="1" k 3="1" k 4="0"/> </filter> <filter id="A"> <fe. Color. Matrix type="matrix" values=" 1 0 0 0 0 0 1 1 0 0 0 1 "/> </filter> <path id="t 0" d="M 263. 245 379. 216 227. 297 173. 235 219. 885 164. 915 z" /> <linear. Gradient id="g 0_0" gradient. Units="user. Space. On. Use" x 1="263. 245" y 1="379. 216" x 2="345. 509" y 2="305. 932"> <stop offset="0" style="stop-color: rgb(47. 098, 30. 779, 23. 575); stopopacity: 1"/> <stop offset="1" style="stop-color: rgb(47. 098, 30. 779, 23. 575); stopopacity: 0"/> </linear. Gradient> <!-- one time for each gradient --> <!-- one time per triangle --> MIRALab Where Research means Creativity </defs> University of Geneva www. miralab. ch SVG Open 2005

19 File structure • Construction of the triangles – Subtriangle – Merge of the 3 gradient's triangles – Adjust opacity <use xlink: href="#t 0" fill="rgb(40. 332, 28. 909, 23. 867)" fill-opacity="1"/> <g filter="url(#A)"> <use xlink: href="#t 0" fill="url(#g 0_0)" /> <use xlink: href="#t 0" fill="url(#g 0_1)" filter="url(#color. Add)"/> <use xlink: href="#t 0" fill="url(#g 0_2)" filter="url(#color. Add)"/> </g> MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005

20 Experiments and evaluation • Files used for test – The pyramid • A simple rotating polygon with user interaction – The face • Simple face with 484 triangles • Mesh deformation using MPEG-4 3 D facial animation as input MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005

21 The pyramid Results – Visual • Reproduction of 3 D scene • Faces visibility – Size • Acceptable for playback with standard SVG viewers on light devices MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005

22 The talking head • Results – Animation • Reproduction of the original animation – Gouraud shading • Results • Drawbacks MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005

23 Results table Pyramid Face with Gouraud VRML size in kb 2 31 31 Number of polygons 8 484 Duration in s 12. 08 8. 28 - Raw size in kb 529 5684 610 Optimized size in kb 91 4798 - Optimized gzipped size 27 in kb 1396 63 MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005

24 Conclusion • Conversion – Improvements • Optimization – Interpolation of animation's path • Gouraud shading MIRALab Where Research means Creativity University of Geneva www. miralab. ch SVG Open 2005