Face Alignment with PartBased Modeling Vahid Kazemi Josephine
![Face Alignment with Part-Based Modeling Vahid Kazemi Josephine Sullivan CVAP KTH Institute of Technology Face Alignment with Part-Based Modeling Vahid Kazemi Josephine Sullivan CVAP KTH Institute of Technology](https://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-1.jpg)
Face Alignment with Part-Based Modeling Vahid Kazemi Josephine Sullivan CVAP KTH Institute of Technology
![Objective: Face Alignment • Find the correspondences between landmarks of a template face model Objective: Face Alignment • Find the correspondences between landmarks of a template face model](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-2.jpg)
Objective: Face Alignment • Find the correspondences between landmarks of a template face model and the target face. Annotated images (source: IMM dataset) Test image (source: You. Tube)
![Why: Possible Applications • The outcome can be used for: - Motion Capture: by Why: Possible Applications • The outcome can be used for: - Motion Capture: by](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-3.jpg)
Why: Possible Applications • The outcome can be used for: - Motion Capture: by determining head pose and facial expressions. - Face Recognition: by comparing registered facial features with a database. - 3 D Reconstruction: by determining camera parameters using correspondences in an image sequence - Etc.
![Global Methods • Overview: - Create a constrained generative template model - Start with Global Methods • Overview: - Create a constrained generative template model - Start with](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-4.jpg)
Global Methods • Overview: - Create a constrained generative template model - Start with a rough estimate of face position. - Refine the template to match the target face. • Properties: - Model deformations more precisely - Arbitrary number of landmarks • Examples: - Active Shape Models [Cootes 95] - Active Appearance Model [Cootes 98] - 3 D Morphable Models [Blanz 99]
![Part-Based Methods • Overview: - Train different classifiers for each part. - Learn constraints Part-Based Methods • Overview: - Train different classifiers for each part. - Learn constraints](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-5.jpg)
Part-Based Methods • Overview: - Train different classifiers for each part. - Learn constraints on relative positions of parts. • Properties: - More robust to partial occlusion - Better generalization ability - Sparse results • Examples: - Elastic Bunch Graph Matching [Wiskott 97] - Pictorial Structures [Felzenszwalb 2003]
![Our approach to face alignment • How can we avoid the draw backs of Our approach to face alignment • How can we avoid the draw backs of](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-6.jpg)
Our approach to face alignment • How can we avoid the draw backs of existing models?
![Our approach to face alignment • Find the mapping, q, from appearance to the Our approach to face alignment • Find the mapping, q, from appearance to the](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-7.jpg)
Our approach to face alignment • Find the mapping, q, from appearance to the landmark positions: • But q is complex and non-linear…
![Linearizing the model • Use piece-wise linear functions Linearizing the model • Use piece-wise linear functions](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-8.jpg)
Linearizing the model • Use piece-wise linear functions
![Linearizing the model • Use a part based model Linearizing the model • Use a part based model](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-9.jpg)
Linearizing the model • Use a part based model
![Linearizing the model • Use a suitable feature descriptor Feature Descriptor Linearizing the model • Use a suitable feature descriptor Feature Descriptor](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-10.jpg)
Linearizing the model • Use a suitable feature descriptor Feature Descriptor
![Part Selection Criteria • Detect the parts accurately and reliably - Contain strong features Part Selection Criteria • Detect the parts accurately and reliably - Contain strong features](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-11.jpg)
Part Selection Criteria • Detect the parts accurately and reliably - Contain strong features • Ensure a simple (linear) model - Minimum variation • Capture the global appearance - Cover the whole object
![Part Selection for the face We chose nose, eyes, and mouth as good candidates Part Selection for the face We chose nose, eyes, and mouth as good candidates](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-12.jpg)
Part Selection for the face We chose nose, eyes, and mouth as good candidates Image from IMM dataset
![Appearance descriptor • Variation of PHOG descriptor - Divide the patch into 8 sub-regions Appearance descriptor • Variation of PHOG descriptor - Divide the patch into 8 sub-regions](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-13.jpg)
Appearance descriptor • Variation of PHOG descriptor - Divide the patch into 8 sub-regions - Recursively repeat for square regions
![Part detection • Build a tree-structured model of the face, with nose at the Part detection • Build a tree-structured model of the face, with nose at the](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-14.jpg)
Part detection • Build a tree-structured model of the face, with nose at the root, and eyes and mouth as the leafs of the tree.
![Part detection • Detect the parts by sliding a patch on image and calculating Part detection • Detect the parts by sliding a patch on image and calculating](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-15.jpg)
Part detection • Detect the parts by sliding a patch on image and calculating the Mahalanobis distance of the patch from the mean model
![Part detection • Find the optimal solution by minimizing the pictorial structure cost function: Part detection • Find the optimal solution by minimizing the pictorial structure cost function:](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-16.jpg)
Part detection • Find the optimal solution by minimizing the pictorial structure cost function: • We can solve this efficiently by using generalized distance transform [Felzenszwalb 2003] by limiting the cost function
![Regression • Model the mapping between the patch’s appearance feature (f) and its landmark Regression • Model the mapping between the patch’s appearance feature (f) and its landmark](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-17.jpg)
Regression • Model the mapping between the patch’s appearance feature (f) and its landmark positions (x) as a linear function: • Estimate weights from training set using Ridge regression
![Regression • Comparison of different regression methods Regression • Comparison of different regression methods](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-18.jpg)
Regression • Comparison of different regression methods
![Robustify the regression function • Why • Compensate for bad part detection • Deformable Robustify the regression function • Why • Compensate for bad part detection • Deformable](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-19.jpg)
Robustify the regression function • Why • Compensate for bad part detection • Deformable parts don’t exactly fit in a box • How • Extend training set by adding noise to part positions
![Experiments • Use 240 face images from IMM dataset. • Dataset contains still images Experiments • Use 240 face images from IMM dataset. • Dataset contains still images](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-20.jpg)
Experiments • Use 240 face images from IMM dataset. • Dataset contains still images from 40 individual subjects with various facial expressions under the same lighting settings • 58 landmarks are used to represent the shape of subjects
![Results • Comparison of localization accuracy of our algorithm comparing to some existing methods Results • Comparison of localization accuracy of our algorithm comparing to some existing methods](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-21.jpg)
Results • Comparison of localization accuracy of our algorithm comparing to some existing methods on IMM dataset. * Mean error is the mean Euclidean distance between predicted and ground truth location of landmarks in pixels
![Results • The results of cross validation on IMM dataset Predicted Ground truth Results • The results of cross validation on IMM dataset Predicted Ground truth](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-22.jpg)
Results • The results of cross validation on IMM dataset Predicted Ground truth
![Demo More videos: http: //www. csc. kth. se/~vahidk/face/ Demo More videos: http: //www. csc. kth. se/~vahidk/face/](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-23.jpg)
Demo More videos: http: //www. csc. kth. se/~vahidk/face/
![Conclusion and future work • Part-Based models can be used to simplify complicated models Conclusion and future work • Part-Based models can be used to simplify complicated models](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-24.jpg)
Conclusion and future work • Part-Based models can be used to simplify complicated models • The choice of parts is very important • HOG descriptors are not fully descriptive
![• Questions? • Questions?](http://slidetodoc.com/presentation_image_h/364d723390002dcbbdad80052189b9f4/image-25.jpg)
• Questions?
- Slides: 25