Cursive Controlling Expressive Avatar Gesture using Pen Gesture

Cursive: Controlling Expressive Avatar Gesture using Pen Gesture Francesca A. Barrientos John F. Canny UC Berkeley Computer science division 2 October C V E ’ 0 2002 2, S e p t e m. CVE b e r'02 3 0 • - Bonn, O c t o Germany b e r 2 , 2 0 0 2 , B o. Barrientos nn, Germany.

2 Avatar world communication • Verbal communication – Chat – Spoken • Nonverbal communication – Change image or play animation – Controls • Select menu item • Select image from palette of images The palace • Interface creates disruption between verbal and nonverbal blaxxun 2 October 2002 CVE '02 • Bonn, Germany Barrientos

Language embedded in physical world • In physical world, language embedded in matrix of sounds and visuals including movement • Communication through multiple channels is seamless and continuous • Avatar, as a virtual body, can send nonverbal communication 2 October 2002 CVE '02 • Bonn, Germany Barrientos 3

4 Problem • Controlling a highly articulated avatar using simple computer input devices is difficult • Goals for gesture control technique – Seamless with verbal communication – Create expressive movement – Unobtrusive interaction at a desktop or palmtop computer 2 October 2002 CVE '02 • Bonn, Germany Barrientos

5 Avatar gesture from pen gesture • Body gesture has symbolic and qualitative aspects • Pen gesture (handwriting) carries discrete and continuous data • Pen gesture simultaneously selects an avatar gesture and controls expressive movement 2 October 2002 CVE '02 • Bonn, Germany Barrientos

6 Illustration of technique • Writing pen gesture triggers animation • Body gesture is “sweep” to side • Symbol is letter ‘s’ l • Quality being varied is size 2 October 2002 CVE '02 • Bonn, Germany Barrientos

7 Overview • Background – Nonverbal communication – Previous approaches • Interaction technique • Cursive application • Conclusions 2 October 2002 CVE '02 • Bonn, Germany Barrientos

8 Communicative gesture • Closely tied to speech • Encodes ideas shared with speech – Clarifies meaning when speech is ambiguous – Useful when outside noise interferes with speech • Multidimensional in meaning and form – Encodes through shape, timing, and quality of movement • Indicates mood/emotion 2 October 2002 CVE '02 • Bonn, Germany Barrientos

9 Previous approaches • Specialized widgets Vlnet expression panel 1. Select 2. Modulate 3. Trigger • 3 step process Comic. Chat Emotion. Wheel • Select and modulate • Hunt and click diverts attention • Avatar agency Body. Chat and Comic. Chat • Driven by chat text • Lacks direct control 2 October 2002 CVE '02 • Bonn, Germany Barrientos

10 Continuous interaction Computers fragment our thinking by substituting discrete events for continuous actions. -Malcolm Mc. Cullough (Abstracting Craft, p. 53) [Gestures] can. . . enhance the experience of agency through kinesthetic involvement and the feeling of directness. -Brenda Laurel (Computers as Theatre, p. 158) 2 October 2002 CVE '02 • Bonn, Germany Barrientos

11 Why pen gestures • Natural – People doodle while talking and listening • Expressive – Reflects emotional state – Very personal – Can be intentionally manipulated • Dual nature – form: Analog and digital • Multi-dimensional 2 October 2002 CVE '02 • Bonn, Germany Barrientos

12 Interaction technique Library Generated offline Input User writes letter in GUI Generation Selection and synthesis Animation Gesture performed wsrlibrary 2 October 2002 GUI Gesture generator CVE '02 • Bonn, Germany Avatar animator Barrientos

13 Interaction schematic Cursive interaction technique. Copyright © 2002 by Barrientos. 2 October 2002 CVE '02 • Bonn, Germany Barrientos

14 Synthesizing expressive gesture • An avatar gesture is represented by a set of motion samples – Each motion sample has different movement qualities – Samples recorded using human performance capture system – Samples made up of set of joint angle trajectories • Synthesize expressive gesture instance by blending samples 2 October 2002 CVE '02 • Bonn, Germany Barrientos

15 Motion blending example sample 2 sample 1 interpolated gesture instance 2 October 2002 CVE '02 • Bonn, Germany Barrientos

16 Modulation through multilinear interpolation Gβ • Gesture sample I – Rotation angles over time • Gesture type β – Set of 2 n prototypes - G – n style parameters – Prototypes represent extremal trajectories β • Gesture instance I (u, v ) Speed u k – Vector of joint trajectories – Multilinear interpolation on type produces instance Iβ(0, 1) Iβ(1, 1) β I (l, k) I (0, k) Iβ(0, 0) Iβ(1, k) Iβ(1, 0) β l 2 October 2002 CVE '02 • Bonn, Germany Size v Barrientos

17 Interpolating speed • Interpolation along curves of different durations • Time dilation step – Determine duration for interpolated curve – Choose sample rate on interpolated curve – Compress slower curve sample at proportionately slower rate – Sample faster trajectory at proportionately faster rate 2 October 2002 Angle θ CVE '02 • Bonn, Germany t Barrientos

18 Cursive 2 October 2002 CVE '02 • Bonn, Germany Barrientos

Independent of specific avatar world software • Cursive communicates directly with all avatar copies • Avatar driver runs Cursive to control gesturing avatar • Other viewers can see gestures without installing Cursive 2 October 2002 CVE '02 • Bonn, Germany Barrientos 19

20 Communication • • • Driver logs into Vworld server Other viewers receive notification Other viewers request and download avatar copy Avatar opens socket connection to Cursive sends gesture commands via socket viewer host driver host Web server opencommands socket sends login request avatar Download avatar notification VWorld server 2 October 2002 CVE '02 • Bonn, Germany Barrientos

21 Evaluation • Very simple to control one parameter at a time • More complicated to control size and speed simultaneously – Effective usage requires practice – Require further investigation into mapping handwriting features to movement parameters • Viable technique for controlling avatar gesture 2 October 2002 CVE '02 • Bonn, Germany Barrientos

22 Summary • Wanted: Richly expressive, spontaneous gesture in avatar worlds • Solution: An interaction technique employing pen gesture input • Cursive: An implementation of this interaction technique 2 October 2002 CVE '02 • Bonn, Germany Barrientos

23 Future work • User studies • Experiments with public multi-user virtual worlds • Investigate avatar gesture vocabulary – How many gestures – What types of gestures 2 October 2002 CVE '02 • Bonn, Germany Barrientos

24 Conclusion • Transmitting bodily nonverbal communication through the internet is an exciting possibility • Continuous interaction strengthens the connection between the user and the avatar • Pen gestures are a powerful technique for controlling expressive avatar gesture – Expressive writing is natural – Interaction technique is compact and unobtrusive – Maps continuous actions to continuous events 2 October 2002 CVE '02 • Bonn, Germany Barrientos

25 Acknowledgements My gesture model: Erin Dare. 2 October 2002 CVE '02 • Bonn, Germany Barrientos