The University of Auckland Computer Science New Zealand

The University of Auckland | Computer Science | New Zealand PRESENTATION 2009 3 D Human Face Reconstruction and Expression Modelling Alexander Woodward

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Outline q Aim q System overview q Related work q 3 D face reconstruction q Expression modelling q Contributions and future work

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Overview q Aim: Integrated system for 3 D face reconstruction and expression modelling w Vision based not graphics based w Low cost and self-contained q Results can be applied to: ¨Biometrics and security ¨Biomedical visualisation ¨Computer and video games ¨Film ¨Teleconferencing ¨Human computer interaction 3

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand System overview 3 D reconstruction Stati c binocular Active & passive Dynami c stereo Active structured lighting 3 D video scanner Active photometric stereo 3 D data Expression modelling Marker based motion capture Muscle inverse kinematics Video based Sequences from 3 D video scanner

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Related work q Complete systems for face reconstruction and animation are uncommon ¨High hardware requirements ¨Data acquisition, motion capture and animation systems are often provided as disparate packages or only as a service, cf. a stand-alone solution q At least 9 prominent projects aimed toward complete systems ¨Excluding in-house solutions ¨Large body of work in 3 D face research w 3 D reconstruction, expressions, motion capture 04 November 2020 Department of Computer Science 5

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Related work q Borshukov et al (2003 – 2007) Playable Universal Capture approach w 3 D scanner, marker based tracking, optical flow, video texture q Ma et al (2007, 2008) Capture face reflectance w 3 D scanner, photometric stereo, motion capture w Light stage – 156 LED lights over an icosahedron q Image Metrics Inc. & U Sth Carolina Graphics Lab (2008) Digital Emily project w Light stage captures geometry and reflectance w 33 expressions captured; creates an animation 6 rig

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand 3 D reconstruction Stati c binocular Active & passive Dynami c stereo Active structured lighting 3 D video scanner Active photometric stereo 3 D data Expression modelling Marker based motion capture Muscle inverse kinematics Video based Sequences from 3 D video scanner

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand 3 D reconstruction requirements q Off-the-shelf hardware, no special properties ¨Cameras, PC, projector q Low acquisition time – faces move, esp. children q Controlled lighting q Vision based ¨New algorithms q Useful for any type of object 8

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Static 3 D reconstruction Stati c binocular Active & passive Dynami c stereo Active structured lighting 3 D video scanner Active photometric stereo 3 D data Expression modelling Marker based motion capture Muscle inverse kinematics Video based Sequences from 3 D video scanner

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Static 3 D reconstruction q Evaluated approaches: 1. Active & passive binocular stereo 2. Active structured lighting 3. Active photometric stereo ¨ Evaluate effectiveness w Accuracy, time complexity q Determine best approach for dynamic 3 D reconstruction system ¨ 12 algorithms ¨ Database of 15 faces w Alternative test set q Ground truth w Focus on stereo algorithms scanner data: 3 D q Compared to Middlebury, algorithms rank differently for faces q Projected patterns improve and level out performance 10

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Active binocular stereo q Strip colour pattern: much higher accuracy ¨SAD correlation algorithm: Strip pattern SAD - without pattern SAD - with strip pattern 80% 92% ¨Pattern colour should contrast strongly on skin 11

The University of Auckland | Computer Science | New Zealand PRESENTATION 2009 Statistical results BP 73% GC 77% FCV 69% + Grad. + Strip 77% 89% + Grad. + Strip 83% 92% Four Path Shapelet 54% 71% CM 88% SAD 80% + Grad. + Strip 89% 92% + Grad. + Strip 85% 92% DPM + Grad. + Strip 79% 84% 92% SDPS + Grad. 89% 90% + Strip 93% Gray code 97% Ground truth 12

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Dynamic 3 D reconstruction Stati c binocular Active & passive Dynami c stereo Active structured lighting 3 D video scanner Active photometric stereo 3 D data Expression modelling Marker based motion capture Muscle inverse kinematics Video based Sequences from 3 D video scanner

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Dynamic 3 D reconstruction q Reconstruction at video rates → 3 D video! ¨ From static reconstruction best results: w ‘One shot’ active illumination + Symmetric Dynamic Programming (SDPS) w Project pattern every other frame to get a clean texture (2) (3) (1) Monochrome stereo pair of video cameras + 3 rd colour web camera obtains colour texture. 14

2009 The University of Auckland | Computer Science | New Zealand PRESENTATION Colour texture generation + Colour image (reprojected into same reference frame) → Monochrome image Final texture q Low resolution colour information combined with high resolution luminance information q Next step: colour video cameras 15

The University of Auckland | Computer Science | New Zealand PRESENTATION 2009 Videos 16

The University of Auckland | Computer Science | New Zealand Patternless reconstruction PRESENTATION 2009

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Marker based expression modelling 3 D reconstruction Stati c binocular Active & passive Dynami c stereo Active structured lighting 3 D video scanner Active photometric stereo 3 D data Expression modelling Marker based motion capture Muscle inverse kinematics Video based Sequences from 3 D video scanner

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Marker based expression modelling q Data driven: ¨ Stereo web-cameras, face markers. ¨ Head motion - rigid ¨ Expressions - non-rigid q Tracked 3 D points ¨ Unique 3 D face model mapping ¨Virtual muscle animation w 17 active muscles w Muscle inverse kinematics (IK) – Jacobian Transpose 19

The University of Auckland | Computer Science | New Zealand PRESENTATION 2009 Example videos Happiness – easiest to reproduce Surprise 20

2009 The University of Auckland | Computer Science | New Zealand PRESENTATION Anger – needs teeth! Disgust – pursing of mouth & closing of eyes not represented

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Video based expression modelling 3 D Reconstruction Stati c binocular Active & passive Dynami c stereo Active structured lighting 3 D video scanner Active photometric stereo 3 D data Expression modelling Marker based motion capture Muscle inverse kinematics Video based Sequences from 3 D video scanner

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand 3 D video based expression modelling q Image blending q Novel face expressions from multiple video sequences ¨Interactive ¨Low preparation ¨Not data driven q Dense depth data – cf. marker system q Video based → realistic 3 D movement and texture q Reconstruction data directly used for Sub-region masks 11 Control control points 23

2009 The University of Auckland | Computer Science | New Zealand PRESENTATION Synthetic expression results Sadness: lower face region, anger: right eye region, surprise: left eye region Happiness: lower face region, surprise: left and right eye regions 24

2009 The University of Auckland | Computer Science | New Zealand PRESENTATION Synthetic expression results Fear: lower face region, happiness: right eye region, anger: left eye region Disgust: lower face region, anger: left and right eye regions

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Contributions q 3 D face reconstruction and expression modelling system q Unique tool-set ¨Low-cost, off-the-shelf ¨Vision based q To 3 D face reconstruction: w Extensive reconstruction comparison w Face database w Dynamic reconstruction system for 3 D video: SDPS + pattern q To expression modelling: w Marker based performance capture system ¦Muscle based IK animation system, unique mapping approach w Video based expression system – realistic, less flexible 26

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Future work and perspective q Many areas for future research ¨Refine hardware - better reconstructions ( lowcost? ) ¨Markerless motion capture - face ( feature ) tracking w Statistical analysis on video data w Active appearance model (AAM) ¨New animation system (out of scope) ¨Full body → complete character q Synergy of computer vision and computer graphics! ¨Physical models for animation ¨Computer vision tools w Especially 3 D video & markerless motion capture

The University of Auckland | Computer Science | New Zealand PRESENTATION 2009 Questions ?

The University of Auckland | Computer Science | New Zealand PRESENTATION 2009

The University of Auckland | Computer Science | New Zealand PRESENTATION 2009 Timeline of Experiments

Ekman - 1987 The University of Auckland | Computer Science | New Zealand PRESENTATION 2009 Universal expressions Sadness Anger Fear Disgust Happiness Surprise Recognisable in every culture! Used as exemplar expressions to judge my results 31

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Types of binocular stereo algorithm q Local vs global optimisation q WTA ¨ SAD, SSD q Chen-Medioni – ¨ local method with explicit surface constraints ¨ Seed propagation approach q Dynamic programming – 1 D optimisation ¨ SDPS – markov chain ¨ DPM q Cubic algorithms – 2 D optimisation ¨ Markov random field ¨ Energy minimisation ¨ Graph-cut (KZ 1, Roy. Cox), Belief Propagation,

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Types of photometric stereo algorithm q Experiment focused on integration methods ¨Assumes C² continuity – i. e. a smooth second derivative q Local optimisation – based on curve integrals ¨Four path integration ¨Shapelet w Explicit summation of basis functions q Global optimisation ¨FCV – Frankot Chellappa Variant

The University of Auckland | Computer Science | New Zealand PRESENTATION 2009 Structured lighting techniques

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Body modelling and animation q Body: generic skinned animation ¨Skeletal hierarchy, fully articulated • The bones of the hand • Each bone of the • The body model with underlying skeleton 04 November 2020 skeleton has a region of influence, denoted in green Department of Computer Science • Movement of the forearm 35

The University of Auckland | Computer Science | New Zealand PRESENTATION q Interactive personalised avatar creator ¨Input photograph ¨RBF mapping 36 2009

The University of Auckland | Computer Science | New Zealand PRESENTATION 2009 Results 04 November 2020 Department of Computer Science 37

The University of Auckland | Computer Science | New Zealand PRESENTATION 2009 Results

The University of Auckland | Computer Science | New Zealand PRESENTATION 2009 Results

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand 3 D video based expression system overview q Acquire sequences of individual expressions using dynamic 3 D face reconstruction system. w Expression sequences start from a neutral state. w Test subject’s head remains in the same position for every sequence w A reference texture and depth map are taken from the neutral expression and used as the base for all image regions q 11 control points are manually annotated on video sequences. w Future work to automate this process. q Six sub-regions manually defined on the face. 04 November 2020 Department of Computer Science 40

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand System conclusions q Sinusoidal interpolation instead of a linear one. This roughly models the biphasic nature of skin q Realistic animations are created as motion is derived from 3 D video sequences of real-life test subjects. w A user can create unnatural but interesting looking expressions that can convey a comical feel q Texture maps sourced from video sequences solves the loss of detail in the marker based approach w However, apart from the control points that were manually specified, no points on the face surface are Department of Computer Science 04 November 2020 41

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Test subject placement q Subject can be placed with knowledge of required view area, sensor size, and camera lens:

The University of Auckland | Computer Science | New Zealand PRESENTATION 2009 Projector synchronisation

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand RBF mapping approach q Radial Basis Functions q User specified point correspondences on generic model and 3 D face data ¨Specify divergences between data q For each dimension (in 3 D) ¨Find RBF approximation of (1 D) displacements within the 3 D space of specified points. q Using this RBF approximation all 3 D points from the generic model can be mapped to the 3 D face data proportions

The University of Auckland | Computer Science | New Zealand PRESENTATION 2009 Marker tracking

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Rigid and non-rigid motion Anchor markers: Rigid orientation: Remove rigid motion by using transpose of orientation and centre of gravity of anchors

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Muscle inverse kinematics q Forward kinematics is the calculation of a new position g of an end effector by specifying updates to parameters of a kinematic chain q Inverse kinematics is the calculation of parameters for a kinematic chain to meet a desired goal position g, when starting from an initial position e. q Kinematic chain consists of joints q Each joint has DOF’s – its animitable parameters, ¨E. g. 3 -DOF for position, 1 -DOF for orientation around one axis (position of joint implied through kinematic

The University of Auckland | Computer Science | New Zealand PRESENTATION 2009 Jacobian Transpose approach q FK: q IK: e = current end effector position d = change in end effector position First order estimate in positional change: Change in parameters: Jacobian Transpose estimate: g = goal end effector position

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Estimate assured to move closer to the goal g: Always moving in a direction less than 90 degrees from d 04 November 2020 Department of Computer Science 49

The University of Auckland | Computer Science | New Zealand PRESENTATION 2009

The University of Auckland | Computer Science | New Zealand PRESENTATION 2009 Interface between Raw Data and Generic Model q User specifies a ‘minimal’ set of correspondences between raw and generic data q Radial Basis Functions (RBF) used as the interpolant Model with animation system Depth map Correspondences made and mapped via RBF with a final nearest point map and texture projection Results in a custom face with animation system in place • Feature extraction as a goal

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Face Animation Model q Research primarily based on Terzopoulos, Waters, Parke collective work in the field q Physically based model for skin tissue ¨ Mass-Spring system ¨ Epidermal – Fascial – Skull levels of tissue ¨ Forces are applied to the tissue to simulate muscle contractions ¨ Springs bring elasticity, allow forces to propagate-> stretches and pulls! q Abstract muscle definitions ¨ Decoupled from model ¨ Warped via RBF also ¨ Two types

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Face Animation System Forces q Model the behaviour of the tissue w Reactionary over the evolution of applied muscle forces: Skull Penetration Constraint Spring Forces - biphasic spring constant - rest length of spring - current length of spring - spring direction vector Muscle Forces Applied to fascia nodes based on the abstract muscle definitions………. . (explained later)

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Face System Forces Volume Preservation Force - epidermal normal for volume element ‘e’ - current and rest nodal positions with respect to center of mass of element ‘e’ - force scaling constants These forces allows for tissue form restitution

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Linear Muscle q Linear Muscle: ¨Applies forces to nodes inside it’s angular range ¨Influence is weighted by angle and radius from muscle vector Displacement formula: Where and ‘k’ = muscle contraction increment.

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Ellipsoid Muscle q Ellipsoid Muscle: ¨Acts like a string bag ¨Application of force weighted by radius only ¨Defined by major and 2 minor axes ¨Can generate puckering effects Displacement formula:

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Physical Simulation q Layered Tissue Model is a physically based one q Euler integration is used to run the simulation Equations of motion Acceleration Velocity Nodal position Velocity dependent damping co-efficient. Controls the rate of dissipation of kinetic energy which eventually brings the facial mesh to rest.

The University of Auckland | Computer Science | New Zealand PRESENTATION 2009 6 pre-built expressions Happiness Sadness Surprise Fear Disgust Anger

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand General conclusions and future work (old version) q Investigated low-end and cost effective equipment to create self-contained tools that can run entirely on any end user system. ¨A unique solution has been proposed for 3 D face reconstruction and expression modelling with appropriate hardware q Synchronised audio capture of speech sequences would greatly add to the realism. q The attachment of the face model to a body would complete the system, giving a fully realised virtual human. 04 November 2020 Department of Computer Science 59

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand q Conclusions drawn from the static reconstruction experiment formed the basis of a dynamic 3 D face reconstruction system q 3 D face reconstructions have no notion of higher order surface structure and are just a collection of points. w This structure was addressed in the second part of this thesis which investigated face expression modelling. q Marker motion was combined with a muscle inverse kinematics framework to drive the facial animation system. w A static face texture impacts on the visual result, as illumination cues such as wrinkles and shadowing over 60 Department of Computer Science 04 November 2020

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand q To supplement the work on 3 D faces, a body model was created q Interactive 3 D video expression creation system which ties together 3 D face reconstruction and expression modelling. q Main problems faced were dealing with hardware constraints w But focus on low-cost and off-the-shelf solutions q Focused on the computer vision aspects of facial reconstruction and expressions as opposed to computer graphics 04 November 2020 Department of Computer Science 61

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand q A combined marker and dense 3 D reconstruction system could be developed, to incorporate further information for a muscle inverse kinematics system q Highly detailed face animation is best served by taking advantage of real world data in the form of digital images and computer vision processing q Advanced physical models of faces meets the tools and approaches investigated within this thesis 04 November 2020 Department of Computer Science 62

04 November 2020 Department of Computer Science 63 The University of Auckland | Computer Science | New Zealand PRESENTATION 2009

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand q Details in important areas of the face that are not currently modelled include the eyelids, lips, teeth and inner mouth. q Loss of texture detail in the forehead where the wrinkles are lost q Fine tuning of the preset muscle locations and parameters when mapping a new face model was sometimes needed to improve results or correct muscles 04 November 2020 Department of Computer Science 64

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand System analysis q Capable of reproducing facial expressions from marker motion. ¨Low cost hardware, easy retargetting to other models. q Many differences between test-subjects ¨Expression articulation, muscle control, gross face movement q Difficulty in performing when no emotional tie involved. w Easy to understand the need for directors in performance capture situations. w Some user direction was needed to describing to a test subject how an expression should be created 04 November 2020 Department of Computer Science 65

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand q Issue: potentially multiple solutions for a vertex position when influenced by multiple muscles q Illumination conditions affect coloured marker detection q Reflectance properties of the skin surface are important visual cues. ¨Missing in this system ¨Addressed in next chapter. 04 November 2020 Department of Computer Science 66

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Body modelling and animation system q Skinned animation system was chosen for real-time capability and ease of creating new body poses ¨A posable skeleton is associated with a body model (skin surface description), usually in the form of geometric data w Forward and inverse kinematics used for animation w Skin surface under new pose is determined based on skeletal bone local coordinate systems and blending between adjacent bones. q Future work: combine with the face reconstructions and animation systems. 04 November 2020 Department of Computer Science 67

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Static reconstruction experiment q Evaluated three computer vision approaches to 3 -D face reconstruction. ¨ Binocular stereo: passive. ¨ Structured lighting: active. ¨ Photometric stereo: active. q Two main aims: ¨ Determine their effectiveness for 3 D facial reconstruction. w Accuracy, time complexity. ¨ Provide a new and alternative test set for evaluating algorithms. w Database of faces. w We focus on stereo vision algorithms. q Integrated lab environment designed. ¨ 12 algorithms tested in total. ¨ Results compared to ground truth data obtained from a commercial 3 D scanner. q Summary: ¨ Active illumination techniques are most accurate. ¨ Stereo algorithm rankings were different from that expected. ¨ ‘One shot’ active illumination coupled with a traditional stereo 04 November 2020 Department of Computer Science 68

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Photogrammetry Laboratory q Optical ‘Range’. ¨Integrated. ¨Multiple systems view a common scene. q Stereo bench. ¨ Sideways for face capture! q Projector for structured lighting. q Light sources for photometric stereo. q Commercial 3 D scanner. w Solutionix Rexcan 400. Department of Computer Science 04 November 2020 Example Data: Depth map Perspective visualisation 69

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Calibration q System calibration: ¨Estimates intrinsic and extrinsic camera parameters w I. e. camera projection matrices w For cameras: ¦A calibration cube - 63 markings defines a world coordinate system ¦Tsai calibration w For the lights: ¦A calibration sphere estimates directions to lights ¦Simple analytic derivation, inaccurate w Could also calibrate the projector using Tsai’s algorithm 04 November 2020 Department of Computer Science 70

The University of Auckland | Computer Science | New Zealand PRESENTATION 2009 World to image coordinates

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand q Rectification: w The camera calibration matrices were used to rectify images. ¦The resultant image pairs meet the epipolar constraint. q Data processing: w Data must be compared in a common co-ordinate frame. w Alignment done using a semi-automatic process involving 3 D object rigid transformations. w Small number of manual correspondences made. w Data projected into disparity space. 04 November 2020 Department of Computer Science 72

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand q Database of 15 people created ¨Data acquired from all systems ¨Rexcan ground truth q Test-bed for new algorithms 73

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Binocular Stereo q Approach 1: Binocular stereo (stereo vision). ¨ Passive. ¨ Active research area in our department. ¨ Textureless regions cause problems. System Geometry (side view) w Remedy via active illumination. q Test a set of local and global algorithms. Tested algorithms: Sum of Absolute Differences (SAD) Dynamic Programming Method (DPM) q Use two Canon digital SLRs – 6 Mpixels w 1536 x 1024 resolution. Symmetric Dynamic Programming Stereo (SDPS) Graph Cut (GC) Belief–Propagation (BP) Chen and 2020 Medioni 04 November (CM)Department – seed based of Computer Science 74

2009 Structured Lighting The University of Auckland | Computer Science | New Zealand PRESENTATION q Approach 2: Structured Lighting. System Geometry (side view) ¨ Active approach. Depth inferred in the same manner as stereo. ¨ Augment stereo system with a colour projector. w Add structure to scene -> break homogeneity. w Projects 800 x 600 pixel image. ¦Acer PL 111 LCD Projector. 6 of the Gray code projections: q Interested in ‘one shot’ patterns overalgorithms: Gray code. Tested Time-multiplexed structured lighting using Gray code Direct coding - ‘one shot’ colour gradation pattern. 04 November 2020 Department of Computer Science • Add texture to face. • Used with standard stereo algorithms. 75

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Photometric Stereo q Approach 3: Photometric Stereo (PSM). ¨ Face viewed under 3 different known lighting conditions. ¨ Depth by integrating recovered surface orientation map. w Albedo independent approach used. System Geometry (top-down view) ¨ Three 150 W light sources. q Analysed gradient field integration techniques. Tested algorithms: Frankot-Chellappa Variant (FCV) Fourier based integration. Four-Scan method Local integration paths. Shapelets Summation of correlated basis functions. 04 November 2020 Department of Computer Science 76

2009 The University of Auckland | Computer Science | New Zealand PRESENTATION A Collection of Reconstructions q Example depth maps: Ground truth Gray code FCV SAD Structured lighting SDPS GC CM Binocular Stereo Photometric Stereo 04 November 2020 Department of Computer Science 77

The University of Auckland | Computer Science | New Zealand PRESENTATION 2009 Photometric Stereo Results q. Reconstruction accuracy: w 17 test subjects. Percentage of errors less than 2 disparity units Method P <=2, % 69 54 71 97 Gold standard result for accuracy 04 November 2020 Department of Computer Science 78

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Passive Stereo Results q. Reconstruction accuracy: Method P<=2, % 89 79 77 GC 80 73 88 97 04 November 2020 Department of Computer Science 79

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Stereo + Gradation Pattern q. Reconstruction accuracy: Method P<=2, % 90 84 83 GC 85 77 89 97 04 November 2020 Department of Computer Science 80

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Stereo + Strip Pattern q. Reconstruction accuracy: Method P<=2, % 93 92 92 GC 93 89 92 97 04 November 2020 Department of Computer Science 81

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Improvement to Stereo from Active Illumination q Addition of the Strip colour pattern. ¨SAD stereo algorithm: Strip pattern Depth map SAD - without pattern SAD - with strip pattern P<=2 = 80% P<=2 = 93% ¨Pattern colour should avoid skin tones. 04 November 2020 Department of Computer Science 82

The University of Auckland | Computer Science | New Zealand PRESENTATION 2009 Error map example

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand q Gray code approach most accurate. ¨Slower acquisition time. q Look to alternative ‘one shot’ approaches. q Photometric stereo least accurate. q Our test set has high resolution images and large disparity ranges. ¨O(n 3) stereo algorithms – GC, BP – inappropriate. w Long processing time. ¦Parameter setting difficult. q Our results differ from the Middlebury rankings: http: //cat. middlebury. edu/stereo/ 04 November 2020 Department of Computer Science 84

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand q All results contain errors. w Need post processing to clean up data. w Even for the commercial 3 D scanner. q Faces have many unique properties posing a challenge for 3 D reconstruction ¨Human sensitivity to errors in reconstruction - we see faces all the time. ¨For computer vision: w w Specularities. Anistropic reflectance of hair. Sub-surface scattering. Large homogenous regions. 04 November 2020 Department of Computer Science 85

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand Static reconstruction conclusion and analysis q Framework and test-bench for active and passive 3 -D acquisition systems designed. q Three computer vision approaches tested. ¨ 12 algorithms altogether. q Analysed accuracy of algorithms for 3 D face reconstruction. ¨ Data compared to scanner benchmark. q Provided new alternative test set to Middlebury for testing stereo algorithms ¨ High resolution images of faces. q Passive stereo combined with active illumination a promising approach. ¨ Want a one shot approach for faces (moving object). ¨ SDPS + Strip pattern. ¨ Leads to real-time spatio-temporal acquisition. w Acquire 3 D face performance. 04 November 2020 Department of Computer Science 86

04 November 2020 Department of Computer Science 87 The University of Auckland | Computer Science | New Zealand PRESENTATION 2009

2009 PRESENTATION The University of Auckland | Computer Science | New Zealand q A generic face model with an abstract muscle animation system was designed during my Master’s thesis. ¦ Refined for Ph. D thesis. ¦Can be personalised with 3 D data and texture information from the static reconstruction experiment using a custom RBF mapping procedure. • Example of muscle contraction • Generic morphable face with linear and ellipsoid muscles 04 November 2020 Department of Computer Science • A biomechanical tissue model 88