A Prototype System for 3 D Dynamic Face

Problem Statement n n q q Collect multi-view face video with expressions; Potential researches

Relevant Works n n n Static 2 D face databases: FERET, CMU PIE, ORL,

Highlights n n Total Solution: Both hardware and software Multi. View+Synchronization+Real. Time Flexibility: Flexibly

System Diagram Camera Calibration Video Data Capture Color De-mosaicing Facial Feature Tracking 3 D

Synchronization-Hardware Configuration Dragon. Fly Camera

Synchronization-Software Implementation … Buffers Buffer Overrun? Y … Y Time Stamp Matched? N Buffer

Offline Color De-mosaic Raw Data: Color represented in Sparse (Stippled) Pattern Raw Data Reconstructed

Camera Calibration n Find the intrinsic and extrinsic parameters Use Camera Calibration Toolbox for

Camera Calibration (cont’ed) Camera 1 Camera 2

Camera Calibration (cont’ed) Average re-projection error < 0. 2 pixels (0. 16279, 0. 13482)

Facial Marker Tracking n Simple but effective tracking algorithm

Facial Marker Tracking (cont’ed) n Statistical marker collocation model

3 D Reconstruction: Stereo Triangulation n A bit of theory

3 D Reconstruction: Stereo Triangulation (cont’ed)

Deliveries n n n The data acquisition system of camera array Tools for Color

Outline (4 Ws+2 Hs) n n n Why (do we do this? ) Who

Slides: 22

Download presentation

A Prototype System for 3 D Dynamic Face Data Collection by Synchronized Cameras Yuxiao Hu Hao Tang

Problem Statement n n q q Collect multi-view face video with expressions; Potential researches Co-articulation of facial expression and lip movement: Non-frontal view audio/visual speech recognition-lip reading

Relevant Works n n n Static 2 D face databases: FERET, CMU PIE, ORL, Yale Database, UMIST, etc Static 3 D face databases: 3 D-RMA, Gavab. DB, York. DB, XM 2 VTS database, FRGC database, etc 3 D Dynamic face databases: q q CMU FIA, no markers, no audio Intel Research China Database, not synchronized

Highlights n n Total Solution: Both hardware and software Multi. View+Synchronization+Real. Time Flexibility: Flexibly extended from 2 cameras to 5 cameras; Supplementary Tools: q q camera calibration, color space conversion, 2 D facial feature tracking 3 D face shape recovery

Physical Setup Foam. Head

System Diagram Camera Calibration Video Data Capture Color De-mosaicing Facial Feature Tracking 3 D Shape Reconstruction

Synchronization-Hardware Configuration Dragon. Fly Camera

Synchronization-Software Implementation … Buffers Buffer Overrun? Y … Y Time Stamp Matched? N Buffer Overrun? Re-sync Y N Y Compression AVI

Offline Color De-mosaic Raw Data: Color represented in Sparse (Stippled) Pattern Raw Data Reconstructed RGB color image Reconstructed RGB color video

Camera Calibration n Find the intrinsic and extrinsic parameters Use Camera Calibration Toolbox for Matlab Two-step procedure q Find projection matrix using Direct Linear Transformation q Use as initialization for nonlinear minimization of mean squared re-projection error

Camera Calibration (cont’ed) Camera 1 Camera 2

Camera Calibration (cont’ed)

Camera Calibration (cont’ed) Camera 1

Camera Calibration (cont’ed) Average re-projection error < 0. 2 pixels (0. 16279, 0. 13482) and (0. 16439, 0. 12685) Camera 1 Camera 2

Facial Marker Tracking n Simple but effective tracking algorithm

Facial Marker Tracking (cont’ed) n Statistical marker collocation model

Facial Marker Tracking (cont’ed)

3 D Reconstruction: Stereo Triangulation n A bit of theory

3 D Reconstruction: Stereo Triangulation (cont’ed)

Deliveries n n n The data acquisition system of camera array Tools for Color De-mosaicing The calibration data and tools Some sample data result 3 D ground truth data and labeling tool Technical Report

Outline (4 Ws+2 Hs) n n n Why (do we do this? ) Who (has done the related work? ) What (we proposed to do? ) How (did we achieve our goal? ) Why (we need to do so? ) How (we evaluate our work? )