Improving the Recognition of Faces Occluded by Facial

Improving the Recognition of Faces Occluded by Facial Accessories Rui Min Abdenour Hadid Jean-Luc Dugelay Multimedia Communications Dept. EURECOM Sophia Antipolis, France min@eurecom. fr Machine Vision Group University of Oulu, Finland hadid@ee. oulu. fi Multimedia Communications Dept. EURECOM Sophia Antipolis, France jld@eurecom. fr 9/10/2020 1

Outline < Research Problem and Objectives < State of the Art < Framework of the Proposed Approach < Occlusion Detection < Face Recognition < Experiments < Conclusions & Discussions 9/10/2020 2

Research Problem < Facial Occlusions: Sunglasses, Scarf, Medical Mask, Beards etc. < Face Recognition in Non-Cooperative Systems (e. g. Video Surveillance) < Security Issues: Ø Football Hooligans Ø ATM Criminals 9/10/2020 3

Main Goals < Address the Face Recognition under Occlusions caused by Facial Accessories < In Particular: Improving the Recognition of Faces Occluded by Sunglasses and Scarf < Specifically take into account the Occlusion Analysis < Robustness < Simplicity 9/10/2020 4

State of the Art < Holistic approaches are not robust to partial occlusions Ø e. g. : PCA, LDA and ICA < Local feature-based and component based methods Ø Less sensitive to occlusions than the holistic methods Ø e. g. : LFA (local feature analysis), LS-ICA (local salient ICA) < Other new trends: e. g. Sparse Representation, Partial-SVM < (More Recently) Occlusion Analysis prior to Face recognition Ø Observation : Prior knowledge of occlusion significantly improve performance Ø PCA based approach +manually annotated Occlusion information [1] Ø S-LNMF (Selective Local Non-negative Matrix Factorization): automatic occlusion detection + LNMF based FR [2] 9/10/2020 5

Our Framework < Two Step Algorithm: Ø Occlusion Detection in Local Patches Ø Face Recognition based on Local Binary Patterns (LBP) <Recognizing a Probe face Ø Compute the LBP representation Ø Divide the image into local patches Ø Occlusion detection in each patch Ø Non-occluded patches are selected for recognition 9/10/2020 6

Occlusion Detection < Facial Feature based algorithms: l l The information of facial features (such as mouth or skin color) is exploited to decide whether or not a face is occluded. Weakness: – Structure variation of the occluded part might be wrongly categories as mouth/eyes – Color variation of the occluded part might be wrongly recognized as skin color < Learning based algorithms: l A large number of positive (clean faces) and negative (occluded faces) samples to train a classifier, which can predict the label of an unknown face. < Our previous study: <<Robust Scarf Detection prior to Face Recognition>>[3] Ø Learning based method is more robust against texture variations Ø Better tolerate image degradation 9/10/2020 7

Occlusion Detection – cont. Feature extraction Dimensionality reduction SVM-based classification < Image Division < Feature Extraction: Gabor Wavelet filtering < Dimensionality Reduction: Principal Component Analysis (PCA) < Classification: Support Vector Machine (SVM) 9/10/2020 8

Occlusion Detection – cont. < Gabor Wavelet based Feature Extraction: Ø Image Filtering by Gabor Wavelets < Dimensionality Reduction by PCA Ø Construct a data set consists of the features extracted from the occluded and non-occluded patches: Ø Compute the eigenvectors of the covariance matrix of the centered S Ø Project the extracted features onto the eigenspace 9/10/2020 9

Occlusion Detection – cont. < SVM based Occlusion Detection Ø Occlusion Detection is considered as a two-classification problem Ø For a training set consisting of N pairs , is the label indicates occlusion or not. Ø SVM finds the maximum-margin hyper-plane to separate the data by: Ø Kernel in use: Radial Basis Function (RBF) Ø The implementation is provided by LIBSVM (http: //www. csie. ntu. edu. tw/~cjlin/libsvm/) 9/10/2020 10

Face Representation using LBP < Local Binary Patterns Ø Discriminative power Ø Computational simplicity Ø Robustness to monotonic gray scale changes (e. g. illumination variations) Ø Robust to local deformation, geometric transformation and misalignment to some extent. Ø Local patch based approach: easily connected with occlusion analysis 9/10/2020 11

Face Representation using LBP-cont. < Feature Extraction Ø The LBP code for pixel is given by: Ø The Thresholding function: q Face Representation Ø Representing the non-occluded facial components < Recognition Ø Chi-square distance Ø Nearest-neighbor classifier 9/10/2020 12

Experimental Data < AR Face Database [4] Ø A standard testing dataset for occluded face recognition Ø More than 4000 face images of 126 subjects (70 men and 56 women) Ø Facial expressions, illumination conditions and occlusions (sunglasses and scarf) Ø 2 sessions: 14 days interval 9/10/2020 13

Experimental Setup < Faces are Cropped, normalized and down-sampled into 128*128 pixels < LBP operator: (using only uniform patterns, 8 equally spaced pixels on a circle of radius 2) < Face images are divided into 64 blocks (size: 16*16) 9/10/2020 14

Experimental Setup – cont. < Training for the Occlusion Detector Ø Random selection of 150 non-occluded faces, 150 faces occluded by scarf, 150 faces occluded by sunglasses Ø The upper parts of the images are used to train the sunglass detector Ø The lower parts of the images are used to train the scarf detector q. Result of Occlusion Detection 9/10/2020 15

Results – Experiment 1 < Experiment 1 – Justification of the 2 -steps approach Ø Gallery images: 240 non-occluded faces from session 1 (with 3 facial expressions: neutral, smile and anger) Ø Evaluation images: corresponding 240 non-occluded faces from session 2, 240 faces with scarf and 240 faces with sunglasses from session 1 Ø Occluded faces are with three different illumination variations < Algorithms in Comparison: < PCA [5], FA-PCA and LBP [6] < FA-PCA: Occlusion Detection Prior to PCA based Face Recognition 9/10/2020 16

Results – Experiment 1 – cont. < Results 9/10/2020 17

Results – Experiment 2 < Experiment 2 – Comparison with the state of the art approach Ø S-LNMF (Selective Local Non-negative Matrix Factorization) [2] Ø Gallery images: 240 non-occluded faces from session 1 Ø Evaluation images: 240 faces with scarf and 240 faces with sunglasses from session 2 Ø 80 faces with extreme facial expression (scream) from session 2. Ø 80 faces with illumination variations (Right-Light) from session 2. < Results 9/10/2020 18

Conclusions < A novel approach for improving the recognition of occluded faces is proposed < State-of-the art in face recognition under occlusion is reviewed < A new approach to scarf and sunglasses detection is thoroughly described. < Extensive experimental analysis is conducted, demonstrating significant performance enhancement using the proposed approach compared to many other methods under various configurations 9/10/2020 19

Future Works < Address face recognition under general occlusions (sunglasses, scarves, beards, long hairs, caps, extreme facial make-ups etc. ) < Automatic face detection under severe occlusions < Automatic face recognition robust to occlusion in video surveillance 9/10/2020 20

References < [1] A. Rama, F. Tarres, L. Goldmann, and T. Sikora, "More robust face recognition by considering occlusion information, " Automatic Face & Gesture Recognition, 2008. FG '08. 8 th IEEE International Conference on , vol. , no. , pp. 1 -6, 17 -19 Sept. 2008. < [2] H. J. Oh, K. M. Lee, and S. U. Lee, “Occlusion invariant face recognition using selective local non-negative matrix factorization basis images, ” Image Vision Comput. , vol. 26, no. 11, pp. 15151523, Nov. 2008. < [3] R. Min, A. D'angelo, J. -L. Dugelay, “Efficient scarf detection prior to face recognition, ” EUSIPCO 2010, 18 th European Signal Processing Conference, pp 259 -263, Aug. 2010. < [4] A. M. Martinez and R. Benavente, " The AR face database, " Technical report, CVC Technical report, no. 24, 1998. < [5] M. Turk, and A. Pentland, 1991. “Eigenfaces for recognition, ” J. Cognitive Neuroscience, vol. 3, no. 1, pp. 71 -86, Jan. 1991. < [6] T. Ahonen, A. Hadid, and M. Pietikäinen, “Face recognition with local binary patterns, ” Computer Vision, ECCV 2004 Proceedings, Lecture Notes in Computer Science 3021, Springer, 469 -481. 9/10/2020 21

Thank you! 9/10/2020 - - p 22