Digital Face Replacement in Photographs CSC 2530 F

Digital Face Replacement in Photographs CSC 2530 F Project Presentation By: Shahzad Malik January 28, 2003

Face Replacement Motivation n n Currently done manually by graphic artists using photo editing software An automatic system has many potential uses: Hollywood special effects n “Personalized” movies n Framing someone… n

Required Components n Need the following subsystems: Face detection (and tracking for videos) n Head pose estimator n Illumination extractor (*) n Facial expression synthesis n Merging/replacement algorithm (*) n

Light Estimation n Assuming a Lambertian reflectance model: n Any image can then be represented by:

Approximate Skin Tone n n Cannot assume 3 basis images for arbitrary photographs Use an approximate image to generate basis

Fitting a Generic 3 D Model n n n Need geometry to create basis images Fit a generic 3 D face mesh to images “Lift” a texture using planar mapping

Generate Basis Images n n Set 3 linearly independent light positions Relight skin tone model with each light

Determining the Coefficients n Compute a least squares solution to: n Solve separately for each RGB channel

Re-illuminating the Target Face n n Set intensities of the 3 light sources to the coefficient values Render the target face with these lights

Flesh Pixel Detection n n Match non-mesh skin pixels to new skin tone Use a histogram-based skin classifier

Histogram Matching n n n Generate histograms for newly lit face Match the Gaussian distribution from original face to newly lit face For each flesh pixel in original image, choose a new color with a similar location on the Gaussian bell curve

Weighted Color Blending n Blend converted flesh pixels with face mesh pixels

Results

Results (continued)

Results (continued)

Results (continued)

Summary n n n Presented a face replacement system Takes lighting and merging into account Future research areas: Face detection and tracking (for videos) n Expression synthesis n More sophisticated reflectance model n Automatic and precise model-fitting n