Face Alignment with PartBased Modeling Vahid Kazemi Josephine

Face Alignment with Part-Based Modeling Vahid Kazemi Josephine Sullivan CVAP KTH Institute of Technology

Objective: Face Alignment • Find the correspondences between landmarks of a template face model and the target face. Annotated images (source: IMM dataset) Test image (source: You. Tube)

Why: Possible Applications • The outcome can be used for: - Motion Capture: by determining head pose and facial expressions. - Face Recognition: by comparing registered facial features with a database. - 3 D Reconstruction: by determining camera parameters using correspondences in an image sequence - Etc.

Global Methods • Overview: - Create a constrained generative template model - Start with a rough estimate of face position. - Refine the template to match the target face. • Properties: - Model deformations more precisely - Arbitrary number of landmarks • Examples: - Active Shape Models [Cootes 95] - Active Appearance Model [Cootes 98] - 3 D Morphable Models [Blanz 99]

Part-Based Methods • Overview: - Train different classifiers for each part. - Learn constraints on relative positions of parts. • Properties: - More robust to partial occlusion - Better generalization ability - Sparse results • Examples: - Elastic Bunch Graph Matching [Wiskott 97] - Pictorial Structures [Felzenszwalb 2003]

Our approach to face alignment • How can we avoid the draw backs of existing models?

Our approach to face alignment • Find the mapping, q, from appearance to the landmark positions: • But q is complex and non-linear…

Linearizing the model • Use piece-wise linear functions

Linearizing the model • Use a part based model

Linearizing the model • Use a suitable feature descriptor Feature Descriptor

Part Selection Criteria • Detect the parts accurately and reliably - Contain strong features • Ensure a simple (linear) model - Minimum variation • Capture the global appearance - Cover the whole object

Part Selection for the face We chose nose, eyes, and mouth as good candidates Image from IMM dataset

Appearance descriptor • Variation of PHOG descriptor - Divide the patch into 8 sub-regions - Recursively repeat for square regions

Part detection • Build a tree-structured model of the face, with nose at the root, and eyes and mouth as the leafs of the tree.

Part detection • Detect the parts by sliding a patch on image and calculating the Mahalanobis distance of the patch from the mean model

Part detection • Find the optimal solution by minimizing the pictorial structure cost function: • We can solve this efficiently by using generalized distance transform [Felzenszwalb 2003] by limiting the cost function

Regression • Model the mapping between the patch’s appearance feature (f) and its landmark positions (x) as a linear function: • Estimate weights from training set using Ridge regression

Regression • Comparison of different regression methods

Robustify the regression function • Why • Compensate for bad part detection • Deformable parts don’t exactly fit in a box • How • Extend training set by adding noise to part positions

Experiments • Use 240 face images from IMM dataset. • Dataset contains still images from 40 individual subjects with various facial expressions under the same lighting settings • 58 landmarks are used to represent the shape of subjects

Results • Comparison of localization accuracy of our algorithm comparing to some existing methods on IMM dataset. * Mean error is the mean Euclidean distance between predicted and ground truth location of landmarks in pixels

Results • The results of cross validation on IMM dataset Predicted Ground truth

Demo More videos: http: //www. csc. kth. se/~vahidk/face/

Conclusion and future work • Part-Based models can be used to simplify complicated models • The choice of parts is very important • HOG descriptors are not fully descriptive

• Questions?