Realistic Performancedriven Facial Animation M Sanchez J Edge

Realistic Performance-driven Facial Animation M. Sanchez, J. Edge and S. Maddock

Realism in Facial Animation n Traditional concept: realism n simulating the biomechanical processes that drive facial motion Not necessarily true… n n n Physically-based approaches have issues of their own: n n The human face is a highly complicated system of which we have rather limited information to build a mechanical simulation We mainly perceive the visual effects of such processes, not the mechanisms underneath Stability is difficult to assess, convergence impossible Most numerical methods scale up worse than linearly wrt the complexity of the model The problem is connaturally stiff (linear elasticity >> resistance to bend and shear), making it even more numerically sensitive Conclusion: it’s questionable whether this concept of realism can lead to more plausible results than mere anatomically-driven geometric systems (eg. Waters)

Capturing Realism n n One sensible alternative: Capturing from actual performances the visual clues that make facial motion perceptually plausible Intrinsic problems: n n n Only a limited domain of information of actual facial motion is accessible through capture (spatially discrete / timewise discrete) The captured data is bound to the physiognomy of the actual subject Geometric deformation techniques can be used to extrapolate spatially discrete motion (eg. markers) to the full extent of facial skin n n Can be consistent with large-scale properties of skin deformation Oversee fine detail aspects of skin mechanics (furrows / creases) n Retargeting of facial motion is possible using purely geometric criteria (without an anatomic model) n Small skin deformation can be modelled as a parallel process to largescale motion

System overview

System overview

System overview

System overview

System overview

System overview

1. Large-scale deformation

1. 3. Animating the skin

1. 3. Animating the skin n Large-scale aspects of skin mechanics to be reproduced: n n Conventional techniques for spatial interpolation of motion do not fully satisfy these criteria n n Motion is propagated across the connectivity of the skin Underlying layers of tissue dampen the extents of deformation Local smoothness is generally preserved Point-based deformations (Williams, Pelachaud, Kshirsagar) Volumetric techniques (Kalra, Escher) RBFs (Fidaleo) Similar approaches could work if interpolation was performed along geodesic curves on the surface, but that would only work for 2 manifolds, and the skin has openings

1. 3. Animating the skin n n Through surface-to-surface mappings we can consistently interpolate motion across approximations of the mesh to be deformed BIDS does precisely that… n n n Motion propagates naturally across a piecewise surface whose patch topology is compatible with that of the face model Smoothness and locality of deformation can be asserted through the construction and conditioning of such approximating surface For further details: n n “Realistic Performance-driven Facial Animation using HW Acceleration” (submitted to To. G) Thesis (soon to come ; )

1. 2. Adapting facial motion

1. 2. Adapting facial motion n Key fact: Different physiognomies Different scale of motion n Conventional Mo. Cap retargeting cannot be applied to soft body deformation The relation between source and target motion is highly non-linear, both with relation to its point of application and its extent and direction We model this by constructing an RBF-based mapping between the embedding of source and target control surfaces… n n capable of representing the two kinds of nonlinearities lacking of boundary separation issues due to the application of BIDS stable under the usual scale of facial motion For further details: n “Use and re-use of facial Mo. Cap” VVG’ 03

1. 1. Labelling the target mesh

1. 1. Labelling the target mesh n Basic idea: Transplanting with minimal distortion the labelling of the ref. mask onto the target mesh n Our approach to this model consists on deforming the ref. mask to fit the target mesh, and then using the resulting control surface n n n this is solved as an iterative process, minimizing the following energy function: the elastic energy terms correspond to the classic definition by Terzopoulos, and they have analytic solutions for Bezier Triangles non-analytic terms require using blind methods (eg simplex downhill) or forward differences for their spatial derivatives

1. Large-scale deformation

2. Fine tissue detail

2. Fine tissue detail

2. a. Fitting a model to photographs

2. a. Fitting a model to photographs n Motivation: n n n A model of the face is needed to act as support of the normal fields captured in 2. b We need to analyse the ratio of deformation for each of the expressions in 2. c We proceed by retrieving the position of markers on the face of the subject with reverse stereometry from 3 viewpoints… and then deform the reference mask using BIDS:

2. b. Shape from shading

2. b. Shape from shading n Consider the simplified BDRF equation: without the specular component, it would be linear… n n use complementary light polarizers on lights and camera lenses, or apply translucent makeup on the subject (cheap option) With a set of 3 exposures to distinct lighting conditions, N is fully determined for any given pose (Rushmeier) The variation of the normal field ( N) is then encoded in the tangent space of the fitted mesh, producing normal maps

2. cd. Deriving a model of wrinkling

2. cd. Deriving a model of wrinkling n The fitted model in every expression not only acts as reference for normal maps, but also allows measuring the ratio of deformation on the face by approximating its strain tensor with that on the control surface n n n Compression is ultimately the cause of facial tissue wrinkling, however… n n n elastic response is not the same in every region, neither in every direction of strain linear compression is much more relevant than shearing We have a sampling representing these properties (normal maps), together with the corresponding compression tensors, so we can build a model relating the two: n it’s a polynomial approximation of N wrt the 3 components of n n represents both magnitude and direction of infinitesimal strain compression (negative strain) can be isolated by chopping off negative eigenvalues the symmetric form of the tensor experimentation reveals linear degree suffixes For any displaced configuration of the control surface, we can evaluate the model and compute the corresponding normal map

Results

Results

Question time…