A Demonstration of Continuous Interaction with Elckerlyc Herwin

A Demonstration of Continuous Interaction with Elckerlyc Herwin van Welbergen, Dennis Reidsma Job Zwiers

Temporal coordination: from turn based interaction to continuous interaction Current interaction with virtual humans • Using speech, facial expression and gestures • Turn-based, ‘push-to-talk’ interaction paradigm

Our goals • Allow continuous interaction • All partners express themselves continuously • In parallel • Needs prediction and anticipation

Continuous interaction Temporal coordination in human behavior - Timing of backchannel - Allowing room for (intrusive) backchannel from partner - Overlap during non-intrusive backchannel - … In this talk - Focus on specifying continuous interaction - Using minor timing and parameter value adjustments in ongoing behavior

Elckerlyc Temporal coordination in interaction with virtual humans • Elckerlyc: open source platform for behavior generation designed for such temporally coordinated behavior (… and other things) • Executes behavior specified in the behavior markup language (BML) • Implements SAIBA Framework

Why BML/SAIBA? > 10 years of research on ECAs • Increasingly sophisticated models – Directed to different aspects of behavior • Building a state-of-the-art ECA entails re-implementing all these models – Allow reuse of models

SAIBA framework FML Intent Planning Behavior Realization Behavior Planning Feedback • • BML Feedback (Smartbody, Greta, ACE, Elckerlyc) Three-level setup of many existing ECAs Separation between stages Each stage is a black box Clear-cut definition of interfaces between stages

BML Example

BML Example

BML Behaviors Gesture Head Gaze ion t a din r o Speech Co Locomotion Posture Facial expression eyes torso legs

Phases and sync-points

Synchronizing behaviors

Example: gaze shift to moving target <bml id="bml 1"> <speech id="bluespeech" start="gaze 1: ready"> <text>I'm staring at the blue cube. </text> </speech> <speech id="greenspeech" start="gaze 2: start"> <text>Look! A green sphere. </text> </speech> <speech id="bluespeech 2" start="gaze 2: end"> <text>Looking at the blue cube again. </text> </speech> <gaze id="gaze 1" start="1" ready="2" type="AT" modality="NECK" target="bluebox"/> <gaze id="gaze 2" start="4" ready="5" relax="6" end="7" type="AT" modality="NECK" dynamic="true" target="greensphere"/> </bml>

Continuous interaction using SAIBA

Continuous Interaction Scenarios Virtual Trainer

Continuous Interaction Scenarios Virtual Trainer <bml id="bml 1"> <bmlt: procanimation id="exercise 1" name="squat"/> <constraint id="c 1"> <synchronize ref="exercise 1: beat 1"> <sync ref="excercise. Anticipator: beat 1 -0. 5"/> </synchronize> <synchronize ref="exercise 1: beat 2"> <sync ref="excercise. Anticipator: beat 2 -0. 5"/> </synchronize>. . . </constraint> </bml>

Demo

Continuous Interaction Scenarios Taking the turn (as soon as possible) <bml id="bml 1"> <speech id="speech 1" start="speech. Stop. Anticipator: stop+x"> <text>Bla bla</text> </speech> </bml>

Demo • Uses pre-planning • Easily allows incremental updates of speech end prediction

Continuous Interaction Scenarios Keeping the turn

Demo

Conclusion • Elckerlyc provides mechanisms for the specification and execution of behaviors synchronized to predicted human behaviour • Flexible pre-planning • Incremental update of user behavior prediction

Open issues Specifying parameter changes <bmlt: setparameter id=“pchange 1" start="10" end="speech 1: end” target="speech 1" parameter="volume" curve="linear" startvalue="25" endvalue="100"/> • Send a parameter specification message – – Allow easy experimentation with parameter change Synchronization to behaviors Parameter value curve (e. g. linear, ease-in ease-out, …) Using BML? • Conceptually doesn’t match very well with other behaviors • Should not modify synchronization constraints • Requires specialized mechanisms to refer to behaviors in previous blocks

Open Issues Creating anticipators • Currently we use placeholders • Real-time music tempo tracking and prediction (Pieter Bos et al) • Predicting listener backchannel relevant moments (Louis. Philippe Morency, Iwan de Kok, . . . ) • Fitness excercise tempo tracking and prediction (Eike Dehling) • Real-time turn-giving prediction (Jonsdottir et al 2008)

Enterface 2010 Backchannel aware guide • Predicts and anticipates completions, acknowledgement and continuers of a user • Response handling strategies – – – Modulated by politeness Ignore (and speak louder) Acknowledge response Wait for acknowledgement before continuing Change route-giving plan Etc • Requires – Adaption of timing of ongoing behavior – Adaptation of parameterization of ongoing behavior – Graceful interruption

Thank you for your attention More information/demo • http: //hmi. ewi. utwente. nl/showcase/Elckerlyc • Contact Dennis, Herwin: {dennisr, welberge}@ewi. utwente. nl Acknowledgments Stefan Kopp’s group Bart van Straalen Ronald Paul Mark ter Maat Zsofia Ruttkay Greta developers Smart. Body developers

END The End.

Enterface 2010 Route recap: dealing with acknowledgements • • A 1: So, we went to the square with the obelisk… A 2: We crossed the river using the suspension bridge… A 3: Crossed the square with the arch monument… A 4: And took the first street to the right. Do not deal with it: simply elicit the feedback, pause a bit, then continue speaking, whatever the subject does Deal with it: elicit feedback, pause a bit, if feedback is there, wait till it is finished before acknowledging (smile, nod? ) and continuing Deal with it, the “continuous interaction way”: elicit feedback, wait, but if feedback comes and it is non-intrusive, immediately acknowledge the feedback (smile, nod? ) and continue speaking even while the user is still finishing the feedback Evaluate responsiveness, attentiveness, politeness, etc

Enterface 2010 Eliciting of completion A: Do you know the way to the square with the… euhm… [makes thinking gestures]… euhm… that thing [makes iconic gesture for the arch monument]… euhm… [looks at the listener inviting a response]… euhm… how do you call it? . . . etc. [correct completion: ] U: arch monument? A: Yes, the arch monument. [Incorrect completion: ] U: Obelisk? A: No, I mean… euhm. . . the square with the arch monument.

Some boring technical facts • Written in Java • ≈80, 000 lines of code (Greta: 45, 000; Smart. Body: 35, 000) – including ≈ 30, 000 lines code for Collada loading and custom graphics – ≈ 200 (J)Unit test cases, ≈ 1100 tests • Takes advantage of multiple processors/cores – Physical simulation – Rendering – Text To Speech (Ode. Java) (Jogl) (MS Speech API, Mary TTS, Text-based)

Licensing • Pre-release under the GPL v 3 license – No source repository yet, contact Herwin for code – ‘Official’ versioned release soon • Inverse dynamics code is available under MIT license – http: //www. herwinvanwelbergen. nl/index. php? selected=phd

Elckerlyc design concerns Continuous interaction • Allow fast and flexible planning and re-planning • Allow specification of synchronization to predictions Flexibility • Easily exchange internal modules, add new modalities, … Motion generation • Not so much focus on designing (procedural) animation • But on combining motion from all kinds of paradigms – In sequence and parallel – Physical simulation, mocap/keyframe animation, procedural motion, biomechanical models, … – New: mixed dynamics

Elckerlyc Architecture Highlights • Scheduling and playing stages • Scheduler can easily be exchanged • Scheduler resolves timing of a sync point per constraint • Sync points can slightly adjusted

Animation plan • Result of planned BML • Specification of motion to be executed by the player • Contains timed motion units • Timing linked to time pegs

Organization of motion Motion Units • (≈LMPs, gestures, controllers, …) • Parameters, linked to BML parameters • Can be executed, typically rotate joints – At canonical time (0. . 1) • Contain phases, key times can be aligned to time pegs – Key times are assigned canonical time values • Physical, procedural, mocap/keyframe, specialized

Procedural motion units • Function of time (0. . 1) and parameter vector • of end effector position and/or joint rotation – Custom functions for perlin noise, splines, … – Example: <End. Effector local="false" target="r_wrist” translation="0; (1 -t)*starty+t*endy; 0. 3"/> • Key frames • Mocap • Can be imported from Greta gestures

Physical motion units Physical controllers • Input: desired state • Minimize discrepancy between current and desired state • Output: joint torques that guides the VH closer to this state • Can cope with external perturbation Video from Abe et al. 2007

Physical simulation for gesture Advantages • Models force transference between body parts • Models momentum effects – Overshoot – Models pendulum like motion easily • Realistic balance • Desired state specification fits some motions well – Loosely hanging arm (desired state: slightly damp arm motion) – Balance (desired state: projected Co. M in support polygon) – Spine (desired state: spine pose)

Physical simulation for gesture Disadvantages • Precise timing and limb placement is an open issue – It is unknown if and when a controller achieves a desired limb position – These aspects are crucial for (speech accompanying) gesture

Mixed dynamics • Use kinematic motion on body parts that require precise timing/placement • Use physical simulation the remaining part of the body • Calculated the forces the kinematic parts apply on the physical part – Using inverse dynamics • Allow switching between different kinematic/physical representations – Depending on the needs of the moment

Comparison with mocap

Executing a Greta gesture

Executing a Smart. Body gesture

Physical spine (experimental)

Switching

Transition motion units • • • Define a transition animation from physics to kinematics Or kinematics to kinematics Flexible start state Predicted end state Only define type, joints and start and end time <bmlt: transition id="trans 1" class="Hermite. Spline. Physical. Transition" start="armhang: end" end="7"> <bmlt: parameter name="joints" value="l_shoulder, l_elbow, l_wrist"/> </bmlt: transition>

Prediction Animation predictor • Can predict a joint configuration at a certain time • Uses a copy of the animation plan with predictable timed motion units • Predictable timed motion units – Deterministically define pose at each time • Procedural motion unit – Determinism during certain motion phases • Stroke of gesture • Gaze at static target

Custom kinematic motion units Adhere to the motion unit interface • Move joints on the basis of time (0. . 1) and parameter values (can be changing during motion) • Provide canonical key positions for synchronization • Motion should be smooth (C 0 continuous) • Can make use of predicted motion • Can use a flexible start position • Currently: – Gaze, pointing • Planned: MURML motion units

Animation player • Executes an animation plan • Timing is linked to time pegs • Time warping: converts ‘real’ time to canonical time • Combines physical simulation and kinematic motion automatically

Scheduling • Determine synchronization constraints between behaviors • Distribute behavior over planners • BML => plans • Assign preliminary times to pegs

Scheduling • Conflicting constraints • Stretching/skewing/skipping if necessary

Scheduling in Elckerlyc Using the Smart. Body scheduling algorithm • Assumes that the behaviors in the XML are ordered by time importance • The timing of the 1 st behavior is never adapted • The timing of the 2 nd behavior is adapted to fit the time constraints imposed by the 1 st behavior • The timing of the 3 rd behavior is adapted to fit the time constraints imposed by the 1 st and 2 nd behavior • Etc • So, ‘later’ behaviors never change the timing of earlier behaviors

Scheduling • Modularity: separate parser and uni-modal planners • Each planner – Can provide timing information on BML behaviors given time constraints • Desired duration, requested timing – Can construct (motion) units from BML behaviors – Can validate the current plan • Allows us to easily exchange the scheduler – Use ACE’s chunk based scheduler?

Animation Planner Animation planner • Construct the animation plan • BML => timed motion units • Resolves execution times for unknown Time Pegs for a timed motion unit , given constraints • Can validate the current animation plan

From BML to timed motion units Gesture binding • Maps behaviors to motion units • Handles parameter assignment in motion units – Maps BML parameter – Or provide defaults • Allows easy modification of BML realization • Allows easy addition of new BML behaviors • Mapping is currently 1 to 1

Replanning • Moving Time Pegs – Keys of timed motion units are linked to time pegs => retiming is handled automatically – Planner can be queried for the validity of the motion plan after moving time pegs • Access to motion units, speech units, ability to remove or adapt them

Anticipation Specify synchronization to predicted times <bml id="bml 1" xmlns: bmlt="http: //hmi. ewi. utwente. nl/bmlt"> <bmlt: procanimation id="conduct 1" name="3 -beat"/>. . . <bmlt: procanimation id="conduct 8" name="3 -beat"/> <bmlt: controller id="balance 1" class="Balance. Controller"/> <constraint id="c 1"> <synchronize ref="conduct 1: start"> <sync ref="metronome 1: tick 1"/> </synchronize> <synchronize ref="conduct 1: beat 2"> <sync ref="metronome 1: tick 2"/> </synchronize>. . . <synchronize ref="conduct 8: end"> <sync ref="metronome 1: tick 25"/> </synchronize> </constraint> </bml>

Demo with anticipation

Other uses for Time Pegs • Chunk based/last minute scheduling – MORE? ? ? • Communication between Engines – Communicate time drift of TTS in Speech Engine to Face Engine

Further work Implementing Anticipators for turn-taking • End of turn, interruption point, user wants turn, . . Combining Timed Motion Units How to best implement interruption? • Directly hook into Elckerlyc? • Define some BML strategies? – Core BML can only replace, append and insert behaviors – No synchronization between different BML requests – Create strategy for replace, tighter merge, change?

Further work Specifying parameter value changes • Using BML? ? <bml id="bml 2" scheduling="tight-merge"/> <bmlt: setparameter id="reparam 1" start="10" end="speech 1: end" target="speech 1" parameter="volume" curve="linear" startvalue="25" endvalue="100"/> </bml>

Collaboration Exchanging Motion Units, other modules? Anticipation/prediction • Predicting listener backchannels with USC/ICT – (Louis-Philippe Morency, Iwan de Kok, . . . ) Rendering/output generation • Bonebus interface for Elckerlyc?

Some boring technical facts • Written in Java • ≈80, 000 lines of code (Greta: 45, 000; Smart. Body: 35, 000) – including ≈ 30, 000 lines code for Collada loading and custom graphics – ≈ 200 (J)Unit test cases, ≈ 1100 tests • Takes advantage of multiple processors/cores – Physical simulation – Rendering – Text To Speech (Ode. Java) (Jogl) (MS Speech API)

Licensing • Pre-release under the GPL v 3 license – No source repository yet, contact Herwin for code – ‘Official’ versioned release soon • Inverse dynamics code is available under MIT license – http: //www. herwinvanwelbergen. nl/index. php? selected=phd

Thank you for your attention More information/demo • http: //hmi. ewi. utwente. nl/showcase/Elckerlyc • Contact Dennis, Herwin: {dennisr, welberge}@ewi. utwente. nl Acknowledgments Job Zwiers Ronald Paul Mark ter Maat Zsofia Ruttkay Greta developers Smart. Body developers

END The End.

RESTJES De volgende dingen horen typisch in de afdeling Easter Eggs: • Scheduling • Hoe maak je je eigen controllers • Hoe maak je je eigen engine • Hoe neem je een nieuwe avatar op in het systeem • Hoe bouw je procedurele animaties • Hoe maak je mocap animaties (daar is ergens een oude herwin tutorial van, no? • …

FUTURE WORK • OPEN: Multiple BML blocks, BML streams, …? Hoe gaan we dat oplossen? Wat waren de mogelijkheden ook weer? • OPEN: meer, zie einde deze presentatie

BML Behaviors

More behaviors…

Open Issues

Handling multiple BML Requests • How to handle both long sequences and graceful interruption of behavior? Granularity of BML Requests is left open • Full monologue • Or short spurts of behavior – Single gaze shift – Speech clause + related gestures

Handling multiple BML Requests Handling successive blocks • • In development Required for BML core compliance: – Replace – Append (but what about persistent behaviors? ) – Merge (default) merges the new BML block into the current BML block(s). Behaviors in this request start immediately, provided they do not conflict with on-going behaviors (and behaviors scheduled in the future) of prior BML requests (for the given character? ). If such a potential conflict is detected with required behaviors, the realizer must not perform any behaviors of the new request and notify via an error / exception. If only non-required behaviors potentially conflict with prior behaviors, the realizer may choose not to perform these behaviors, as long as warning notifying of the omission are sent. If this results in the omission of all behaviors, an error/exception must be sent instead of a warning. Alternatively, the schedule of these non-required behaviors may be adjusted to avoid conflicts, as long as required synchronization constraints are not violated. – tight-merge: (Elckerlyc specific) Like merge, but the new BML block may refer to behaviors in the current block(s). – mergeandreplace (Elckerlyc specific): like tight-merge, but replaces all behaviors from the current block(s) with those in the new block if their ids match. Optionally allows the (fluent) replacement of currently running behaviors. – removeandreplace (Elckerlyc specific): removes all current behaviors except those with ids matching behaviors in the new block. Then replaces those behaviors with the new behaviors and adds the new behaviors. This is different from replace in that it allows replacing behaviors that are currently running and retains persistent behavior. – TODO: wat is nodig voor enterface?

Stuff from old presentation

Animation Player

Open issues Conflict resolution • What if multiple behaviors want to use the right hand? • Resource allocation in the planner – Use left hand if right hand is in use • Combination of animations – Store in motion units how they combine with other motion units • Nod motion should be additively blended • Manipulative hand gesture should constrain hand position – In planner => hierarchical motion plans (as in Smart. Body) – In realizer => final combination phase (as in EMBR) – Use existing computer animation techniques for the combination (see Welbergen et al 2009 for an overview)

Open issues Easily obtaining more motion units • Because authoring motion units is a lot of work • With other research groups? – Cooperation? – Automatic conversion of Greta gestures to our proc animation

End of stuff from old presentation

Feedback from the realizer FML Intent Planning BML Behavior Planning Feedback To the planner • • • BML performance start BML performance stop Sync-Point progress Exception Warning Behavior Realization Feedback

Advanced synchronization <constraint> Allows synchronization of sync points • Before another sync-point • After another sync-point • At another sync-point

Advanced synchronization <constraint id="before_example"> <before ref="speech_1: start"> <sync ref="gaze_1: stroke"/> <sync ref=“nod_1: stroke"/> </before> <constraint id=“sync_example"> <synchronize ref="speech_1: start"> <sync ref="gaze_1: stroke"/> <sync ref=“nod_1: stroke"/> </synchronize> </constraint> • Before/after not implemented in any realizer • Elckerlyc supports synchronize

Gesture behavior • Coordinated movement with arms and hands

Gesture types: Point

Gesture types: beats

Gesture types: conduit

Gesture types: lexicalized

<speech>

Persistent behaviors No end time • • • Posture Gaze Point (? ) ‘New’ behaviors overwrite old behaviors TODO, moet dit hier? In Elckerlyc: implemented with replacement groups

BML Design • • • Describes occurrence of behaviors Relative timing and synchronization of behaviors Form of behaviors Realizer-independent But allows extensions for realizer-dependent behavior

BML Design: Realizer independence • Cannot rely on skeleton joints, speech synthesis systems, available animations, . . . • Refers to body-parts, lexicalized locations, common verbs – speech, face, gesture, center, left • Allows specification of nuanced/detailed behavior through extensions

BROADER VIEW • wat zijn VH toepassingen en waar zijn die goed voor? • hoe zit de methodische cyclus in elkaar van observeren, modelleren en nabootsen van menselijk gedrag? • Wat is continuous interaction, waar is dat goed voor, waarom moeten VHs dat ook kunnen?

ELCKERLYC • SAIBA Global architecture SAIBA (met name: de feedback loops met warnings enzo) • BML Wat zijn de belangrijkste issues bij behavior generation die door BML worden aangepakt? Wat is BML en hoe werkt het? Synchronisatie van behaviors, parallel specificeren van allerlei behaviors, abstractie (!), WAT ZIJN BEHAVIORS (voorbeelden)? • Design concerns voor Elckerlyc • Globale architectuur Elckerlyc met peg board, planners, players, etc. Feedback loops… • Feature: organisation of motion • Bridge: The Animation Player (& Animation Plan) • Feature: Physics and mixed dynamics switching & mixed dynamics & physics (waarom uberhaupt? ) • Feature: persistent behaviors & replacement groups • Feature: prediction of motion (voor transitiecontrollers enzo) • Extension: Hoe maak je je eigen motion units (voor bielefeld relevant). Uitleg procmotionunits, uitleg mocap (kort, = pmu), voorbeeld greta en smartbody, vraag “en Max? ” • Feature: gesture binding en planning

ELCKERLYC IN USE , CONCLUDING REMARKS • hoe wordt Elckerlyc gebruikt voor een VH toepassing? Wat is de relatie tov intent&behavior planning? Die vallen buiten Elckerlyc. Als je behavior gepland hebt: (1) maak elckerlyc, stuur BML (2) maak Elckerlyc, maak MUs en pas die aan en voeg die toe aan player • Wat kan Elckerlyc dat andere realizers niet kunnen? (of: wat zijn de tekortkomingen van andere realizers die je aan begin presentatie noemt als ‘het doel van Elckerlyc’? ) Refereer terug naar de design concerns • Hoe blijkt die continuous interaction nu daadwerkelijk in Elckerlyc te zitten? Hebben we al goede voorbeelden? Wanneer wel? • Boring technical facts

Switching <bml id="bml 1"> <!-– Some conducting gestures --> <bmlt: procanimation id="conduct 1" <bmlt: procanimation id="conduct 2" <bmlt: procanimation id="conduct 3" <bmlt: procanimation id="conduct 4" <bmlt: procanimation id="conduct 5" name="3 -beat"/> name="3 -beat" start="conduct 1: end"/> name="3 -beat" start="conduct 2: end"/> name="3 -beat" start="conduct 3: end"/> name="3 -beat" start="conduct 4: end"/> <!–- Physical controllers --> <bmlt: controller id="balance 1" class="Balance. Controller"/> <bmlt: controller id="armhang" class="Compound. Controller" name="leftarmhang" start="3. 5" end="6. 5"/> <!–- Transtion --> <bmlt: transition id="trans 1" class="Hermite. Spline. Physical. Transition" start="armhang: end" end="7"> <bmlt: parameter name="joints" value="l_shoulder, l_elbow, l_wrist"/> </bmlt: transition> </bml>

Continuous interaction • • • Observation and interpretation Cognition and deliberation Multi-modal Expression Occurring in parallel Expression timed tightly, from moment to moment, to perceived movements and behavior of other E. g. : Conduct an orchestra; Count along with exercising user; Talk about co -occurring physical task performed by user (device maintenance, tutoring) or about fixed media (presenting info about multi media content in IR or QA) Feedback loop.