TAUCHI Tampere Unit for ComputerHuman Interaction Automated recognition

TAUCHI – Tampere Unit for Computer-Human Interaction Automated recognition of facial expressi ns and identity 2003 UCIT Progress Report Ioulia Guizatdinova Research Group for Emotions, Sociality, and Computing University of Tampere 10. 01. 2004

TAUCHI – Tampere Unit for Computer-Human Interaction Contents • • • Research problems Aims and Tasks Facial landmark extraction : Methods Facial landmark extraction : Results Future Steps

TAUCHI – Tampere Unit for Computer-Human Interaction Research Problems “Automated recognition of facial expressions and identity” • • Face identification Recognition of facial expressions

TAUCHI – Tampere Unit for Computer-Human Interaction Research Problems Face identification • Classification of input face to one of existing face classes stored in database • Rejection of input face as unrecognized/unknown face Face database Input face Face classes still image/video signal Recognition system classification rejection Unrecognized face Person A 1 … N … … Person Z

TAUCHI – Tampere Unit for Computer-Human Interaction Research Problems Recognition of facial expressions • Facial expressions affect face recognition because a variability of facial landmarks in their appearance is high • Humans are good in recognizing facial identity regardless of changes in facial expressions • Computer-aided systems of face recognition are dramatically compromised by changes in facial expressions

TAUCHI – Tampere Unit for Computer-Human Interaction Aim of research • The primary aim of this research is to do theoretical and experimental investigation on the possibilities to automatically recognize facial identity independent of changes in facial expressions • For that purpose two 2 D recognition systems will be developed - Recognition system of facial expressions - Expression-invariant system of facial identity recognition

TAUCHI – Tampere Unit for Computer-Human Interaction Tasks • Extraction and classification of facial landmarks, namely, regions of eyes/eye-brows, nose, and mouth from still images • Detection and recognition of facial expressions - how facial muscle activations can change appearance of a face during emotional reactions? • Expression-invariant recognition of facial identity

TAUCHI – Tampere Unit for Computer-Human Interaction Methods of landmark extraction Feature-b ased method • Uses knowledge on geometrical structure of human faces • Based on geometrical features of facial landmarks, such as position of eyes/eye-brows, nose, and mouth • Four regions of interest have been selected as most informative for further recognition steps – – right eye-brow / eye left eye-brow / eye nose mouth

TAUCHI – Tampere Unit for Computer-Human Interaction Methods of landmark extraction 15 Template-b ase method 0 14 2 13 5 3 22. ° 12 • Represents a face as a feature map/template of original facial image • Local oriented edges are used to construct a feature map of the facial image • Orientation of edges has been determined with step of 22. 5 and encoded as 0, 1, …. . 15 1 4 11 5 10 6 9 8 7 Oriented edges extracted in left eye region

TAUCHI – Tampere Unit for Computer-Human Interaction Methods of landmark extraction Database • Tests were performed using Pictures of Facial Affect [1] • 110 images with 7 basic facial displays: happiness, surprise, fear, anger, disgust and neutral expression • Images were first normalized to three pre-set sizes 100 X 150, 200 X 300 and 300 X 400 in order to test the effect of image size to the operation of the algorithms • In sum 110 x 3 = 330 images were used for algorithm testing [1] Ekman, P. , Friesen, W. V. , & Hager, J. C. (2002) Facial Action Coding System (FACS). Published by A Human Face, Salt Lake City, UTAH: USA

TAUCHI – Tampere Unit for Computer-Human Interaction Facial landmark extraction Pre-processing algorithms • • RGB – grey-level transformation Multiresolution image representation was performed using a recursive Gauss transformation Different resolution levels Resolution level 2 Transformation Input face (RGB) Normalization (greylevel scale) Resolution level 0

TAUCHI – Tampere Unit for Computer-Human Interaction Facial landmark extraction • • Final feature map has been constructed on base of local oriented edges extracted in each point of grey-level image at each resolution level with exception of points which had the contrast values less than threshold Extraction of local edges has been performed by calculation of difference between two oriented Gaussians with shifted kernels, which allows determining both orientation and contrast of local edge Points of interest have been grouped - if the distance between points of interest was less than the threshold the points were grouped - otherwise ignored Map of detected points of interest Final feature map

TAUCHI – Tampere Unit for Computer-Human Interaction Facial landmark extraction • • Detected regions of interest have been compared with orientation portraits of facial landmarks I have constructed earlier Regions which did not correspond to the portraits have been ignored Pre-knowledge about facial structure have been used matching Right Eye Number of points of interest • Orientation portraits of the facial landmarks Left Eye Nose Mouth Edge orientation

TAUCHI – Tampere Unit for Computer-Human Interaction Facial landmark extraction Finally, facial landmarks have been detected! neutral disgust fear Examples of feature maps of high-contrast oriented edges detected from the expressive images

TAUCHI – Tampere Unit for Computer-Human Interaction Results Performance of the facial landmark detection algorithm averaged by all expressions for three image sizes

TAUCHI – Tampere Unit for Computer-Human Interaction Results Performance of the facial landmark detection algorithm averaged by all expressions for three image sizes

TAUCHI – Tampere Unit for Computer-Human Interaction Results (a) (b) Right Eye (c) Left Eye (d) Nose Mouth Performance of the feature detection system for three image sizes. N-neutral; H-happiness; Sd-sadness; F-fear; A-anger; Sr-surprise; D-disgust

TAUCHI – Tampere Unit for Computer-Human Interaction Results (a) (b) Right Eye (c) Left Eye (d) Nose Mouth Performance of the feature detection system for three image sizes. N-neutral; H-happiness; Sd-sadness; F-fear; A-anger; Sr-surprise; D-disgust

TAUCHI – Tampere Unit for Computer-Human Interaction Results Problems • • • Algorithms are slow – about few seconds Errors in groupping points of interest (red rectangles a, b, c) Some landmarks are undetectable (d) (a) (b) (c) (d)

TAUCHI – Tampere Unit for Computer-Human Interaction Results Recommendations • To improve detection of nose and mouth regions two alternatives are proposed - The first one is selection of different thresholds for detection and groupping of points of interest for different resolution levels - The second alternative requires more careful processing of detected regions and searching different landmark parts such as eye and mouth corners and nostrils.

TAUCHI – Tampere Unit for Computer-Human Interaction Future steps • Full article about automated expression-invariant detection of facial landmarks; short article about how emotions affect cognitive functioning and how this knowledge might be implicated for HCI • To improve landmark detection; implement prototype of 2 D recognition system of facial expressions (i. Exp. Rec) • To implement and test i. Exp. Rec • To implement of expression-invariant 2 D facial identity recognition system (i. Face. Rec).

TAUCHI – Tampere Unit for Computer-Human Interaction Thank you f r your attention!