Face detection face alignment and face image parsing

  • Slides: 44
Download presentation
Face detection, face alignment, and face image parsing Brandon M. Smith Guest Lecturer, CS

Face detection, face alignment, and face image parsing Brandon M. Smith Guest Lecturer, CS 534 Monday, October 21, 2013 CS 534: Computation Photography 10/20/2021 1

Lecture overview • Brief introduction to local features • Face detection • Face alignment

Lecture overview • Brief introduction to local features • Face detection • Face alignment and landmark localization http: //www. ima. umn. edu/2008 -2009/MM 8. 5 -14. 09/activities/Wohlberg-Brendt/sift. png http: //www. noio. nl/assets/2011 -03 -01 -stitching-smiles/violajones. png http: //www. mathworks. com/matlabcentral/fx_files/32704/4/icaam. jpg • Face image parsing http: //homes. cs. washington. edu/~neeraj/projects/face-parts/images/teaser. png CS 534: Computation Photography 10/20/2021 2

Local features: broad goal • What are local features trying to capture? The local

Local features: broad goal • What are local features trying to capture? The local appearance in a region of the image David G. Lowe, "Distinctive image features from scale-invariant keypoints, " International Journal of Computer Vision, 60, 2 (2004) CS 534: Computation Photography 10/20/2021 3

Local features: motivation What are their uses? o Matching http: //docs. opencv. org/_images/Featur_Flann. Matcher_Result.

Local features: motivation What are their uses? o Matching http: //docs. opencv. org/_images/Featur_Flann. Matcher_Result. jpg CS 534: Computation Photography 10/20/2021 4

Local features: motivation What are their uses? o Matching o Image indexing and retrieval

Local features: motivation What are their uses? o Matching o Image indexing and retrieval Shen et al. , CVPR 2012 CS 534: Computation Photography 10/20/2021 5

Local features: motivation What are their uses? o Matching o Image indexing and retrieval

Local features: motivation What are their uses? o Matching o Image indexing and retrieval o Aligning images, e. g. , for panorama stitching http: //www. leet. it/home/lale/files/Garda-pano. jpg CS 534: Computation Photography 10/20/2021 6

Local features: motivation What are their uses? o o Matching Image indexing and retrieval

Local features: motivation What are their uses? o o Matching Image indexing and retrieval Aligning images, e. g. , for panorama stitching Video stabilization CS 534: Computation Photography 10/20/2021 7

Local features: motivation What are their uses? o o o Matching Image indexing and

Local features: motivation What are their uses? o o o Matching Image indexing and retrieval Aligning images, e. g. , for panorama stitching Video stabilization 3 D reconstruction http: //www. nsf. gov/news/special_reports/science_nation/images/virtualrealitymaps/d uomopisa 500. jpg CS 534: Computation Photography 10/20/2021 8

Local features: motivation What are their uses? o o o Matching Image indexing and

Local features: motivation What are their uses? o o o Matching Image indexing and retrieval Aligning images, e. g. , for panorama stitching Video stabilization 3 D reconstruction Object recognition, including face recognition http: //doi. ieeecomputersociety. org/cms/Computer. org/dl/trans/tp/2007/11/figures/i 192714. gif CS 534: Computation Photography 10/20/2021 9

Local features: types Types of features and feature descriptors o Image intensity or gradient

Local features: types Types of features and feature descriptors o Image intensity or gradient patches o Shift Invariance Feature Transform (SIFT) – very popular! o DAISY o SURF o Many more… CS 534: Computation Photography 10/20/2021 12

Face detection: goal Automatically detect the presence and location of faces in images. Shen

Face detection: goal Automatically detect the presence and location of faces in images. Shen et al. , Detecting and Aligning Faces by Image Retrieval, CVPR 2013 CS 534: Computation Photography 10/20/2021 13

Face detection: motivation • Automatic camera focus http: //cdn. conversations. nokia. com. s 3.

Face detection: motivation • Automatic camera focus http: //cdn. conversations. nokia. com. s 3. amazonaws. com/wp-content/uploads/2013/09/Nokia-Pro-Camera-auto-focus_half-press. jpg CS 534: Computation Photography 10/20/2021 14

Face detection: motivation • Automatic camera focus • Easier photo tagging http: //sphotos-d. ak.

Face detection: motivation • Automatic camera focus • Easier photo tagging http: //sphotos-d. ak. fbcdn. net/hphotos-ak-ash 3/163475_10150118904661729_7246884_n. jpg CS 534: Computation Photography 10/20/2021 15

Face detection: motivation • Automatic camera focus • Easier photo tagging • First step

Face detection: motivation • Automatic camera focus • Easier photo tagging • First step in any face recognition algorithm http: //images. fastcompany. com/upload/camo 1. jpg CS 534: Computation Photography 10/20/2021 16

Face detection: challenges • • Large face shape and appearance variation Scale and rotation

Face detection: challenges • • Large face shape and appearance variation Scale and rotation (yaw, roll, pitch) variation Background clutter Occlusions Image noise Efficiency False positives CS 534: Computation Photography 10/20/2021 17

Face detection: Viola-Jones* • Paul Viola and Michael Jones, Robust Real-time Face Detection, International

Face detection: Viola-Jones* • Paul Viola and Michael Jones, Robust Real-time Face Detection, International Journal of Computer Vision (IJCV), 2004. o Feature type? o Which features are important? o Decide: face or not a face * Next few slides are based on a presentation by Kostantina Pall & Alfredo Kalaitzis, available at http: //www 1. cs. columbia. edu/~belhumeur/courses/biometrics/2010/violajones. ppt CS 534: Computation Photography 10/20/2021 18

Face detection: Viola-Jones Feature type? • Useful domain knowledge: o The eye region is

Face detection: Viola-Jones Feature type? • Useful domain knowledge: o The eye region is darker than the forehead or the upper cheeks o The nose bridge region is brighter than the eyes o The mouth is darker than the chin • Encoding o Location and size: eyes, nose bridge, mouth, etc. o Value: darker vs. brighter CS 534: Computation Photography 10/20/2021 19

Face detection: Viola-Jones Feature type? • Rectangle features o Value = ∑(pixels in black)

Face detection: Viola-Jones Feature type? • Rectangle features o Value = ∑(pixels in black) - ∑(pixels in white) o Three types: 2, 3, 4 rectangles o Very fast: integral image CS 534: Computation Photography 10/20/2021 20

Face detection: Viola-Jones • * From http: //www. cs. ubc. ca/~lowe/425/slides/13 -Viola. Jones. pdf

Face detection: Viola-Jones • * From http: //www. cs. ubc. ca/~lowe/425/slides/13 -Viola. Jones. pdf CS 534: Computation Photography 10/20/2021 21

Face detection: Viola-Jones Final decision: face or not a face • Cascade of classifiers

Face detection: Viola-Jones Final decision: face or not a face • Cascade of classifiers 1. Two-feature classifier: >99% recall, >60% precision 2. Five-feature classifier 3. 10 -feature classifier … 10. 200 -feature classifier CS 534: Computation Photography 10/20/2021 22

Face detection: Viola-Jones http: //vimeo. com/12774628# CS 534: Computation Photography 10/20/2021 23

Face detection: Viola-Jones http: //vimeo. com/12774628# CS 534: Computation Photography 10/20/2021 23

Face detection: recent approaches Xiangxin Zhu and Deva Ramanan, Face Detection, Pose Estimation, and

Face detection: recent approaches Xiangxin Zhu and Deva Ramanan, Face Detection, Pose Estimation, and Landmark Localization in the Wild, CVPR 2012. CS 534: Computation Photography 10/20/2021 24

Face detection: recent approaches Shen et al. , Detecting and Aligning Faces by Image

Face detection: recent approaches Shen et al. , Detecting and Aligning Faces by Image Retrieval, CVPR 2013. CS 534: Computation Photography 10/20/2021 25

Face detection: recent approaches Shen et al. , Detecting and Aligning Faces by Image

Face detection: recent approaches Shen et al. , Detecting and Aligning Faces by Image Retrieval, CVPR 2013. CS 534: Computation Photography 10/20/2021 26

Face alignment and landmark localization: goal Goal of face alignment: automatically align a face

Face alignment and landmark localization: goal Goal of face alignment: automatically align a face (usually non-rigidly) to a canonical reference http: //www. mathworks. com/matlabcentral/fx_files/32704/4/icaam. jpg Goal of face landmark localization: automatically locate face landmarks of interests http: //homes. cs. washington. edu/~neeraj/projects/face-parts/images/teaser. png CS 534: Computation Photography 10/20/2021 27

Face alignment and landmark localization: motivation • Preprocess for: o Face recognition o Portrait

Face alignment and landmark localization: motivation • Preprocess for: o Face recognition o Portrait editing wizards o Face image retrieval o… • Face tracking • Expression recognition • Facial pose recognition http: //static 3. businessinsider. com/image/52127 e 2 169 bedd 4 d 60000012 -752 -564/realeyes-facialrecognition. png http: //mission 0 ps. com/wp-content/uploads/2013/04/10 -special-effects. jpg CS 534: Computation Photography 10/20/2021 28

Face alignment and landmark localization: challenges • • • Pose Expression Identity variation Occlusions

Face alignment and landmark localization: challenges • • • Pose Expression Identity variation Occlusions Image noise CS 534: Computation Photography 10/20/2021 29

Face alignment and landmark localization: approaches Parametric appearance models o Cootes, Edwards, and Taylor,

Face alignment and landmark localization: approaches Parametric appearance models o Cootes, Edwards, and Taylor, Active Appearance Models, ECCV 1998 CS 534: Computation Photography 10/20/2021 30

Face alignment and landmark localization: approaches Parametric appearance models o Cootes, Edwards, and Taylor,

Face alignment and landmark localization: approaches Parametric appearance models o Cootes, Edwards, and Taylor, Active Appearance Models, ECCV 1998 CS 534: Computation Photography 10/20/2021 31

Face alignment and landmark localization: approaches Part-based deformable models o Saragih et al. ,

Face alignment and landmark localization: approaches Part-based deformable models o Saragih et al. , Face Alignment through Subspace Constrained Mean-Shifts, ICCV 2009 CS 534: Computation Photography 10/20/2021 32

Face alignment and landmark localization: approaches Part-based deformable models o Saragih et al. ,

Face alignment and landmark localization: approaches Part-based deformable models o Saragih et al. , Face Alignment through Subspace Constrained Mean-Shifts, ICCV 2009 CS 534: Computation Photography 10/20/2021 33

Face alignment and landmark localization: approaches Supervised descent o Xiong and De la Torre,

Face alignment and landmark localization: approaches Supervised descent o Xiong and De la Torre, Supervised Descent Method and its Applications to Face Alignment, CVPR 2013 CS 534: Computation Photography 10/20/2021 34

Face alignment and landmark localization: approaches Exemplar-based/non-parametric methods o Shen et al. , Detecting

Face alignment and landmark localization: approaches Exemplar-based/non-parametric methods o Shen et al. , Detecting and Aligning Faces by Image Retrieval, CVPR 2013. CS 534: Computation Photography 10/20/2021 35

Face image parsing Smith, Zhang, Brandt, Lin, and Yang, Exemplar-Based Face Parsing, CVPR 2013.

Face image parsing Smith, Zhang, Brandt, Lin, and Yang, Exemplar-Based Face Parsing, CVPR 2013. CS 534: Computation Photography 10/20/2021 36

Face image parsing: goal Given an input face image, automatically segment the face into

Face image parsing: goal Given an input face image, automatically segment the face into its constituent parts. CS 534: Computation Photography 10/20/2021 37

Face image parsing: motivation • Like face alignment, can be used as a preprocess

Face image parsing: motivation • Like face alignment, can be used as a preprocess for face recognition, automated portrait editing, etc. • Encodes ambiguity • Generalizes to hair, teeth, ears etc. across datasets CS 534: Computation Photography 10/20/2021 38

Face image parsing: our approach Database … 2 K exemplar images Exemplar labels 11

Face image parsing: our approach Database … 2 K exemplar images Exemplar labels 11 landmarks ~150 SIFT features CS 534: Computation Photography 10/20/2021 39

Face image parsing: our approach Database … 2 K exemplar images Exemplar labels 11

Face image parsing: our approach Database … 2 K exemplar images Exemplar labels 11 landmarks ~150 SIFT features Step 0: Rough alignment & Top exemplar selection … 100 top exemplars Input CS 534: Computation Photography 10/20/2021 40

Face image parsing: our approach Database Step 0: Rough alignment & Top exemplar selection

Face image parsing: our approach Database Step 0: Rough alignment & Top exemplar selection … Step 1: Nonrigid alignment … 2 K exemplar images Exemplar labels 11 landmarks ~150 SIFT features Input CS 534: Computation Photography 10/20/2021 41

Face image parsing: our approach Database Step 0: Rough alignment & Top exemplar selection

Face image parsing: our approach Database Step 0: Rough alignment & Top exemplar selection … Step 1: Nonrigid alignment Step 2: Exemplar label aggregation 2 K exemplar images Exemplar labels 11 landmarks ~150 SIFT features Input = * + CS 534: Computation Photography … 10/20/2021 42

Face image parsing: our approach Step 0: Rough alignment & Top exemplar selection Database

Face image parsing: our approach Step 0: Rough alignment & Top exemplar selection Database … Step 1: Nonrigid alignment Step 2: Exemplar label aggregation 2 K exemplar images Exemplar labels 11 landmarks ~150 SIFT features Step 3: Pixel-wise label selection =Label 1 Output Input w 1 * + … + CS 534: Computation Photography * Label 2 + w 2 * * Label 9 + w 9 * 10/20/2021 43

Face image parsing: quantitative results CS 534: Computation Photography 10/20/2021 44

Face image parsing: quantitative results CS 534: Computation Photography 10/20/2021 44

Face image parsing: qualitative results Input CS 534: Computation Photography Soft segments + Hard

Face image parsing: qualitative results Input CS 534: Computation Photography Soft segments + Hard segments Ground truth 10/20/2021 45

Face image parsing: qualitative results Input CS 534: Computation Photography Soft segments + Hard

Face image parsing: qualitative results Input CS 534: Computation Photography Soft segments + Hard segments Ground truth 10/20/2021 46

Face detection face alignment and face image parsingFace detection face alignment and face image parsing
MAXMOTORS FACE FACE FACE FACE FACE FACE FACEMAXMOTORS FACE FACE FACE FACE FACE FACE FACE
Face detection Face detection Stateoftheart face detection demoFace detection Face detection Stateoftheart face detection demo
Parsing Part II Topdown parsing leftrecursion removal ParsingParsing Part II Topdown parsing leftrecursion removal Parsing
Parsing methods Topdown parsing Bottomup parsing Universal NonParsing methods Topdown parsing Bottomup parsing Universal Non
Parsing BottomUp LR Table Construction Parsing BottomUp ParsingParsing BottomUp LR Table Construction Parsing BottomUp Parsing
Parsing methods Topdown parsing Bottomup parsing Universal EliminatingParsing methods Topdown parsing Bottomup parsing Universal Eliminating
Parsing IV Bottomup Parsing Parsing Techniques Topdown parsersParsing IV Bottomup Parsing Parsing Techniques Topdown parsers
Image Alignment and Mosaicing Image Alignment Applications LocalImage Alignment and Mosaicing Image Alignment Applications Local
Image alignment Image source Alignment applications Panorama stitchingImage alignment Image source Alignment applications Panorama stitching
Ch 7 Face detection KH Wong Face detectionCh 7 Face detection KH Wong Face detection
Standard 7 Alignment Utah Model Alignment Vertical AlignmentStandard 7 Alignment Utah Model Alignment Vertical Alignment
Alignment Most alignment programs create an alignment thatAlignment Most alignment programs create an alignment that
LCLSII cryomodule alignment LCLSII cryomodule alignment Topics AlignmentLCLSII cryomodule alignment LCLSII cryomodule alignment Topics Alignment
Ontology Alignment Ontology Alignment n n Ontology alignmentOntology Alignment Ontology Alignment n n Ontology alignment
Alignment Most alignment programs create an alignment thatAlignment Most alignment programs create an alignment that
Ontology Alignment Ontology Alignment n n Ontology alignmentOntology Alignment Ontology Alignment n n Ontology alignment
Multiple Sequence Alignment Multiple Alignment versus Pairwise AlignmentMultiple Sequence Alignment Multiple Alignment versus Pairwise Alignment
Pairwise sequence Alignment Types of Alignment Global alignmentPairwise sequence Alignment Types of Alignment Global alignment
A Subquadratic Sequence Alignment Algorithm Global alignment AlignmentA Subquadratic Sequence Alignment Algorithm Global alignment Alignment
Image Parsing Unifying Segmentation and Detection Z TuImage Parsing Unifying Segmentation and Detection Z Tu
Image Parsing Unifying Segmentation Detection and Recognition ShaiImage Parsing Unifying Segmentation Detection and Recognition Shai
Image Parsing Unifying Segmentation and Detection Z TuImage Parsing Unifying Segmentation and Detection Z Tu
Image Parsing Unifying Segmentation and Detection Z TuImage Parsing Unifying Segmentation and Detection Z Tu