Synthetic faces for eyewitnesses and face recognition research

Synthetic faces for eyewitnesses and face recognition - research and practice Colin Tredoux Department of Psychology University of Cape Town South Africa July 2014 International Congress of Applied Psychology Paris, France

Many thanks to… Heike Schmidt Nils Maskow Taryn Sullivan Peter Schaupp Oliver Oxtoby Patrick Chiroro Dave Nunez Buhle Zuma Yon Rosenthal And many others – Anne Kaiser, Desirée Baumbach, Kate Kempen, Alicia Nortje Also, many thanks to sponsors for grants, support - NRF, URC, SAPS

Part I Introduction

Synthesising faces • There is an abiding societal interest in technologies that create artificial faces – Driven by applications • Assisting eyewitnesses with the creation of a visual likeness of a face they saw • Computer game avatars • Reconstruction of faces of dead people • Economising on bandwidth for telecommunication • Creating stimuli for experimental research • Creating synthetic faces for use in police lineups • Animating faces with emotional expressions

Help witnesses reconstruct faces • Sketching faces from memory a natural extension of portraiture art • Sketch artists are Expensive to train; employ • Thus, the rise of mechanised forms of construction of human faces • ranging from acetate kits (Photofit, Identikit) through computerised versions of the same, to complex software using statistical models

Since much of this talk is about new technology for making faces, it may interest you to look back in time for a similar announcement of new technology… (early Photofit system, USA)

Entertainment • Long history of making faces for movies, going back to animation techniques • Has generated new application for Hindi word ‘avatar‘ • This application particularly concerned in the emulation of motion, speech, facial expression – Toy story, Avatar, FIFA Game Face • Perhaps we can learn something from this – most face recognition experiments in psychology have used static images

Re-animating the dead

Ötzi, the iceman

Richard III

Reconstruction of unknown skeleton on Table Mountain, Cape Town Phillips, Rosendorff & Scholtz (1996)

Telecommunication Bandwidth is limited – Compress faces by using replicas of the talker, that are model-generated Wang & Cohen (2005)

Police lineups • Police lineups are arrays of people containing a suspect and a number of innocent ‘foils’ • It is important to choose the foils so that they bear a sufficient resemblance to the suspect (or description of perpetrator) • This is an arduous task, could be made simpler by using synthetic faces or people

Grist & Tredoux (2013)

Experimental manipulation • Properties of faces are often of interest in research – e. g. distinctiveness, typicality, attractiveness, ‘babyfacedness’, similarity to another face • Stimuli usually selected that vary on one dimension, while controlling for others • Not easy! – if synthetic faces can be created to vary on one dimension only this would be useful

Example – shifting a face on a perceptual dimension Southern Northern Original face

Facial animation/expression • Considerable interest in Psychology in expression/recognition of face emotions • Several attempts to create models of these with synthetic faces Roesch et al. , (2010)

Part II Some methods for synthesising faces

General approaches to synthesis • Generic model-based – e. g. Start by creating a single underlying musculature and skeletal structure – Deform the general model to create an individual face Kähler, Haber, Yamauchi, & Seidel (2002)

Images from Caroline Wilkinson’s facial reconstruction centre, Dundee, Scotland

General approaches to synthesis • Exemplar based-modeling – Create a face through the linear combination of an existing collection of face ‘models’ from Tredoux (2002) The best known of these approaches is probably ‘Eigenface’ modeling (a form of Principal Component Analysis )

10 20 30 50 100 Original from Tredoux (2002)

Eigenface exemplar modeling

Eigenface software EVOFIT Stirling, UK Frowd & Hancock (2004) EIGENFIT (now E-Fit V) Canterbury, UK Gibson & Solomon (2006) ID Cape Town, SA Tredoux, Nunez, Oxtoby, Rosenthal at al. (1999; 2007) Your presenter 27

An alternative eigenface approach: Morphable Models FACEGEN

CAPTURE Protocol developed with Peter Schaupp; available on request

CAPTURE cont. Without standardization, variable quality adds artefacts to face models

CAPTURE cont. Standardization [lighting, exposure, colour temperature, face orientation, all important]

CAPTURE cont. If you don’t control capture, this happens to your face models. . Animations from Peter Hancock

LANDMARK

Auto Landmark Several auto-landmarkers have been developed, following Tim Cootes’ work (2001) Quality not good enough for face synthesis 34

WARP Warp Faces are warped so that texture is mapped to the average shape (called ‘reflectance’ in some approaches e. g. Todorov and colleagues), shape information is saved in a separate file

EIGEN For each face in the set, we have: 1. For each of [whole face, eyes, eyebrows…mouth] 1. A texture file [a string of numbers, concatenated from the xy layout of a 2 D face image to a single vector] 2. A shape file [a set of co-ordinates, interpolated from Bezier curves, for each of the features]

Shape Tesselation structure for mapping texture to shape Texture in standard shape

EIGEN cont. We do PCA on texture and shape for each face (and features within face) produces eigenfaces (shapes and textures), and the coefficients that map the original faces onto the eigenfaces For each feature we combine texture and shape in an appearance model

EIGEN cont. This can all be done in SPSS or MATLAB, but in our experience these are too slow and cumbersome (One image file could have 2. 4 million data points, and a data file with 400 faces > 1 billion data points) Therefore, we developed a suite of bespoke programs for making these models

EIGEN cont. In practice, we build the models in a single step, using a batch process: All software available on request from colin. tredoux@uct. ac. za

Eigenfaces • Principal Component Analysis (PCA) is conducted on the normalized faces, to derive ‘eigenfaces’ • Each face in the dataset can then be expressed as a weighted sum of the eigenfaces – E. g. Face 1 = 0. 12 E 1 + 0. 03 E 2 + … + Ek • Faces not already in the dataset can also be expressed as a weighted sum, but the representation will have a small amount of error 42

Eigenface modeling - key ideas 1 Proposition A – synthetic faces: The model is capable of representing unique faces that we do not know to exist 1. Any vector of coefficients (c 1…ck) [that does not describe a known face in the population] when multiplied through the eigenfaces will make a synthetic face, mistakeable as a member of the population in question 2. SUBJECT TO the constraint that the sums of squares of the coefficients are bounded i. e. a ≤ ci 2 ≤ b in our experience a ≈ 0. 6 and b ≈ 1. 6

Eigenface modeling - key ideas 2 Proposition B – faces from memory: • for every face that exists in the population, there is a representation by a unique set of PC coefficients • Therefore, the task of reconstructing a face from memory is the same as finding the set of coefficients • This is an optimization problem

Optimization algorithms Generally, eigenface synthesis software uses some variant of evolutionary or genetic search i. e. a search strategy that mimics the process of natural selection

Optimization algorithm PBIL • PBIL (population based incremental learning) was identified as one useful technique Rosenthal (1998) – A set of faces is generated, using random estimates for each of the PC coefficients – The witness chooses the best match (i. e. closest to the perpetrator) in this set of faces – The coefficients of the ‘best match’ are used to generate the next sample, or the witness decides that no better match is possible

Optimization algorithm Mchoice DIFS An alternate ‘variance reduction’ algorithm was devised by Tredoux, Nunez, Oxtoby & Prag (2007) as simulations showed PBIL to be very slow 0. 9 0. 8 0. 7 0. 6 0. 5 0. 4 0. 3 0. 2 0. 1 0 Mchoice_face 1 Mchoice_face 2 PBIL_face 1 PBIL_face 2 0 100 200 300 Cycle 48

Demonstration – using ID