EurographicsSIGGRAPH Symposium on Computer Animation 2003 Unsupervised Learning

  • Slides: 26
Download presentation
Eurographics/SIGGRAPH Symposium on Computer Animation (2003) Unsupervised Learning for Speech Motion Editing Yong Cao

Eurographics/SIGGRAPH Symposium on Computer Animation (2003) Unsupervised Learning for Speech Motion Editing Yong Cao 1, 2 Petros Faloutsos 1 Frederic Pighin 2 University of California, Los Angeles 1 Institute for Creative Technologies, University of Southern California 2

Problem ■ Motion Capture is convenient but lacks flexibility ■ Problem: How to extract

Problem ■ Motion Capture is convenient but lacks flexibility ■ Problem: How to extract the semantics of the data for intuitive motion editing?

Related Work 1. Face motion synthesis ■ Physics-based face model Lee, Terzopoulos, Water Kähler,

Related Work 1. Face motion synthesis ■ Physics-based face model Lee, Terzopoulos, Water Kähler, Haber, Seidel ( SIGGRAPH 1995) (Graphics Interface 2001) ■ Speech motion synthesis Bregler, Covell, Slaney Brand (SIGGRAPH 1997) (SIGGRAPH 1999) Ezzat, Pentland, Poggio (SIGGRAPH 2002) 2. Separation of style and content Brand, Hertzmann (SIGGRAPH 2000) Chuang, Deshpande, Bregler (Pacific Graphics 2002)

Our Contribution ■ New statistical representation of facial motion • Decomposition into style and

Our Contribution ■ New statistical representation of facial motion • Decomposition into style and content • Intuitive editing operations

Our Contribution Original Neutral Motion Edited Sad Motion

Our Contribution Original Neutral Motion Edited Sad Motion

Roadmap ■ Independent Component Analysis (ICA) ■ Facial motion decomposition ■ Semantics of components

Roadmap ■ Independent Component Analysis (ICA) ■ Facial motion decomposition ■ Semantics of components ■ Motion editing

Independent Component Analysis (ICA) ■ Statistical technique ■ Linear transformation ■ Components are maximally

Independent Component Analysis (ICA) ■ Statistical technique ■ Linear transformation ■ Components are maximally independent

Steps of ICA ■ Preprocessing (PCA) ■ Centering ■ Whitening ■ ICA decomposition Reconstruction:

Steps of ICA ■ Preprocessing (PCA) ■ Centering ■ Whitening ■ ICA decomposition Reconstruction: Decomposition:

Roadmap ■ Independent Component Analysis (ICA) ■ Facial motion decomposition ■ Semantics of components

Roadmap ■ Independent Component Analysis (ICA) ■ Facial motion decomposition ■ Semantics of components ■ Motion editing

Speech motion Dataset Speech motion of 113 sentences in 5 emotion moods: Frustrated 18

Speech motion Dataset Speech motion of 113 sentences in 5 emotion moods: Frustrated 18 sentences Happy 18 sentences Neutral 17 sentences Sad 30 sentences Angry 30 sentences Each motion: 109 motion capture markers 2 – 4 seconds

Facial Motion Decomposition and Reconstruction Facial motion Components in ICA space …………

Facial Motion Decomposition and Reconstruction Facial motion Components in ICA space …………

Roadmap ■ Independent Component Analysis (ICA) ■ Facial motion decomposition ■ Semantics of components

Roadmap ■ Independent Component Analysis (ICA) ■ Facial motion decomposition ■ Semantics of components ■ Motion editing

Interpretation of independent components ■ Goal: Find the semantics of each component Classify each

Interpretation of independent components ■ Goal: Find the semantics of each component Classify each component into: Style (emotion) Content (speech) ■ Methodology ■ Qualitatively ■ Quantitatively

Qualitatively changing Style (emotion) Content (speech)

Qualitatively changing Style (emotion) Content (speech)

Quantitatively n Style: Emotion Same speech, different emotion ………… Happy Frustrated

Quantitatively n Style: Emotion Same speech, different emotion ………… Happy Frustrated

Speech Content Grouping of motion markers ■ Mouth motion ■ Eyebrow motion ■ Eyelid

Speech Content Grouping of motion markers ■ Mouth motion ■ Eyebrow motion ■ Eyelid motion

Content: speech related motion Step 1: Using each independent component to reconstruct facial motion

Content: speech related motion Step 1: Using each independent component to reconstruct facial motion 0 Reconstruct 0 0 ………… 0

Content: speech related motion Step 2: Compare according to certain region

Content: speech related motion Step 2: Compare according to certain region

Roadmap ■ Independent Component Analysis (ICA) ■ Facial motion decomposition ■ Semantic meaning of

Roadmap ■ Independent Component Analysis (ICA) ■ Facial motion decomposition ■ Semantic meaning of components ■ Motion editing

Motion Editing with ICA ■ Edit the motion in intuitive ways ■ Translate ■

Motion Editing with ICA ■ Edit the motion in intuitive ways ■ Translate ■ Copy and Replace ■ Copy and Add

Results ■ Changing emotional state by translating

Results ■ Changing emotional state by translating

Conclusion ■ New statistical representation of facial motion • Decomposition into content and style

Conclusion ■ New statistical representation of facial motion • Decomposition into content and style • Intuitive editing operations

The End Thanks to Wen Tien for his help on this paper, Christos Faloutsos

The End Thanks to Wen Tien for his help on this paper, Christos Faloutsos for useful discussions, and Brian Carpenter for his excellent performance. Thanks to the USC School of Cinema – Television and House of Moves for motion capture.

Unsupervised Learning Deep Autoencoder Unsupervised Learning We expectUnsupervised Learning Deep Autoencoder Unsupervised Learning We expect
Unsupervised Learning and Data Mining Unsupervised Learning andUnsupervised Learning and Data Mining Unsupervised Learning and
Unsupervised Learning and Clustering In unsupervised learning youUnsupervised Learning and Clustering In unsupervised learning you
Unsupervised Learning Neural Networks 11 Unsupervised learning simpleUnsupervised Learning Neural Networks 11 Unsupervised learning simple
Unsupervised Learning and Data Mining Unsupervised Learning andUnsupervised Learning and Data Mining Unsupervised Learning and
CLUSTERING TECHNIQUE DATA MINING UNSUPERVISED LEARNING UNSUPERVISED LEARNINGCLUSTERING TECHNIQUE DATA MINING UNSUPERVISED LEARNING UNSUPERVISED LEARNING
Chapter 5 Unsupervised learning Introduction Unsupervised learning TrainingChapter 5 Unsupervised learning Introduction Unsupervised learning Training
Practical Machine Learning Unsupervised Learning Sven Mayer UnsupervisedPractical Machine Learning Unsupervised Learning Sven Mayer Unsupervised
Unsupervised Learning Deep Autoencoder Unsupervised Learning We expectUnsupervised Learning Deep Autoencoder Unsupervised Learning We expect
Animation What is animation The computer animation refersAnimation What is animation The computer animation refers
Unsupervised Learning n Clustering n n n UnsupervisedUnsupervised Learning n Clustering n n n Unsupervised
Animation Through The Ages WHAT IS ANIMATION AnimationAnimation Through The Ages WHAT IS ANIMATION Animation
Animation in flash FramebyFrame Animation l An animationAnimation in flash FramebyFrame Animation l An animation
Animation Two major forms of animation Stopmotion animationAnimation Two major forms of animation Stopmotion animation
Introduction to Animation George Boudman Animation Techniques AnimationIntroduction to Animation George Boudman Animation Techniques Animation
Unity 3 D Animation Animation The Animation SystemUnity 3 D Animation Animation The Animation System
TYPES OF ANIMATION Types of animation Traditional animationTYPES OF ANIMATION Types of animation Traditional animation
ANIMATION IT Survey 1112 WHAT IS ANIMATION AnimationANIMATION IT Survey 1112 WHAT IS ANIMATION Animation
Chapter 5 ANIMATION The Power of Animation AnimationChapter 5 ANIMATION The Power of Animation Animation
CUTOUT ANIMATION CUTOUT ANIMATION Cutout animation puppets canCUTOUT ANIMATION CUTOUT ANIMATION Cutout animation puppets can
Animation in Java 1 Animation Demo Checkout AnimationAnimation in Java 1 Animation Demo Checkout Animation
Types of Animation Animation Animation is about bringingTypes of Animation Animation Animation is about bringing
Animation The Fundamental Principles Early Animation Processes AnimationAnimation The Fundamental Principles Early Animation Processes Animation
Behavioral Animation Procedural Animation Type Behavioral Animation IntroducedBehavioral Animation Procedural Animation Type Behavioral Animation Introduced