Digital Face Cloning David Bennett Christophe Hery Steve

Digital Face Cloning David Bennett Christophe Hery Steve Sullivan George Borshukov J. P. Lewis Lance Williams Paul Debevec Fred Pighin Li Zhang

Seattle San Francisco Los Angeles Boston David Bennett Christophe Hery Steve Sullivan George Borshukov J. P. Lewis Lance Williams Paul Debevec Fred Pighin Li Zhang

Definition • Process of capturing an actor’s performance and/or likeness in a digital face model

E. g. photogrammetry (input)

E. g. photogrammetry (output)

Motivation • Replicate a live actor (e. g. celebrity) – Stunts – Extreme makeup – Younger/older/slimmer clone • Creating realistic faces from first principles is difficult (digitizing is easier than modeling)

Purpose of the course 1. Review the field 2. Suggest directions

Digital Clones: History

CG Humans in the movies • First 3 D actor representation in the movies: Peter Fonda in Future. World (1976!) • Not intended to be photorealistic

1990 s

1990 s • Terminator 2 (brief, texture projection) • Jurrasic Park (legs only, does not count) • Titanic (distant) • Startup attempt: Virtual Celebrity/Global Icons, purchased virtual rights to Sammy Davis Jr. , others

1998: Hollywood is ready • "I don't see them as competing with real actors, ” Richard Masur, head of Screen Actors Guild • Projected uses: de-aging, celebrity vacuum ads. • A television series with a virtual cast was considered

Recent and current

Recent and current • Space Cowboys (2000), Enemy at the Gates (2001): brief and distant, but successful • Disney Facial Cloning test (afternoon session) • Matrix sequels (2003): “superpunch” full-screen for 13 seconds • Spider. Man 2 (afternoon session) (reflectance capture) • Lemony Snicket: CG baby in several shots (afternoon session) • Discovery Channel Churchill/Hitler clones (MPC)

Observation • There has not been a “breakthrough” introduction of digital clones • Rather, clones were initially used in the distance, and briefly; they are getting closer, and withstanding longer study. • Current state of the art: full screen for several seconds (see afternoon Perception session)

Syllabus – morning (theory) 8: 30 Introduction and Overview Fred Pighin 9: 00 Face Scanning and Dense Motion Capture Technologies Li Zhang 10: 00 Break or Mathematical Background (schedule change) 10: 15 Reflectance Modeling and Capture Paul Debevec and J. P. Lewis 11: 15 Facial Parameterization and Cross-Mapping J. P. Lewis and Fred Pighin 12: 15 Lunch

Syllabus – afternoon (practice) 1: 30 Case study: Face Cloning at ILM Steve Sullivan and Christophe Hery 2: 15 Perception Experiment (schedule change) 2: 30 Case Study: Disney Facial Cloning Lance Williams 3: 15 Case study: Leaping the Uncanny Valley with Data (Face Cloning in the Matrix sequels) George Borshukov 4: 00 Break 4: 15 Case study: Polar Express David Bennet 5: 00 Perception of Facial Realism J. P. Lewis 5: 15 Panel on the Future of Digital Face Cloning All

Face Cloning at Siggraph • • • Courses – 10. Realistic Materials in Computer Graphics – 28. From Mocap to Movie: The Making of "The Polar Express“ – 29. High-Dynamic-Range Imaging and Image-Based Lighting Papers (Skin & Faces session on Monday) – Automatic Determination of Facial Muscle Activations From Sparse Motion Capture Marker Data – Face Transfer With Multilinear Models – Postproduction Relighting and Reflectance Transformation with Time-Multiplexed Illumination Sketches – Image-Based Rendering From a Sparse Set of Images – Implementation of Modeling Hair From Multiple Views – Model Flowing: Capturing and Tracking of Deformable Geometry – Performance Geometry Capture for Spatially Varying Relighting – Joint Motion and Reflectance Capture for Relightable 3 D Video