To Beat or Not To Beat Beat Gestures

  • Slides: 24
Download presentation
To Beat or Not To Beat? Beat Gestures in Direction Giving Chris Brandhorst &

To Beat or Not To Beat? Beat Gestures in Direction Giving Chris Brandhorst & Mariët Theune University of Twente

Overview • • Beats and other gesture types Research context: The Virtual Guide A

Overview • • Beats and other gesture types Research context: The Virtual Guide A small direction giving corpus How to recognize beats? The Beat Filter When are beats used? Concept categories A (very) simple beat usage model Conclusions and future work Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 1

Beats and other gestures Gesture types • Deictic: pointing at an object’s location •

Beats and other gestures Gesture types • Deictic: pointing at an object’s location • Iconic: representing the shape of a concrete object • Metaphoric: depicting an abstract object using metaphor • Beat: indicating discourse structure; emphasis Mc. Neill (1992) “Hand Mind” p. 93: beats made up 44. 7% of used gestures in a cartoon narration corpus. Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 2

Context: The Virtual Guide • An embodied direction giving agent in a 3 D

Context: The Virtual Guide • An embodied direction giving agent in a 3 D environment Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 3

Video (link naar filmpje hier) Try out the Virtual Guide “live” here: http: //wwwhome.

Video (link naar filmpje hier) Try out the Virtual Guide “live” here: http: //wwwhome. ewi. utwente. nl/~hofs/dialogue/ Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 4

Gesture generation • Keyword-based • Turns (“left”, “right”, etc. ): fixed pointing gesture in

Gesture generation • Keyword-based • Turns (“left”, “right”, etc. ): fixed pointing gesture in turn direction • Objects (“the coffee counter”, etc. ): – pointing gesture to absolute 3 D object location (gesture is computed dynamically) – pointing gesture to relative object location, from viewpoint along the route (fixed gesture; like Turns) – iconic gesture reflecting object shape (fixed gesture from “gestionary”) Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 5

When to use beat gestures? • In the BEAT system (Cassell et al. ,

When to use beat gestures? • In the BEAT system (Cassell et al. , 2001) gestures are used to mark new information and to contrast items • Beats are given low priority: they are only used when no other gesture type is available • In the Virtual Guide, in almost all cases a pointing or iconic gesture is available – so, no “need” for beats? • No: human direction givers do often use beat gestures, as shown in our small video corpus. Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 6

Direction giving video corpus Used for this study: • 15 short video clips (±

Direction giving video corpus Used for this study: • 15 short video clips (± 45 sec. each) • 4 different Dutch speakers, 3 -4 clips each • 2 different destinations in our building • 2 versions of each clip (except one): with a listener present or to the camera • 133 gestures; 124 annotated (others not clearly visible) Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 7

How to recognize beats? Beat characteristics: • “A simple flick of the hand” •

How to recognize beats? Beat characteristics: • “A simple flick of the hand” • Short and quick • Only 2 gesture phases: preparation and retraction (no stroke) • No “tensed stasis” • Formless hand shape Formal coding, based on shape only: the Beat Filter (Mc. Neill, 1992) “filters out” beats from other gestures. Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 8

The Beat Filter 1. Does the gesture have other than 2 movement phases? 2.

The Beat Filter 1. Does the gesture have other than 2 movement phases? 2. How often does tensed stasis or finger movement appear? 3. If the first movement is in non-center space, is any other movement in center space? 4. If there are exactly 2 movement phases, are they in different spaces? • • Add 1 point for each “yes” answer to Questions 1, 3, 4 to the number given in answer to Question 2. The lower the score, the more likely the gesture is a beat. Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 9

Using the Beat Filter (1) • 109 gestures were scored with the Beat Filter*

Using the Beat Filter (1) • 109 gestures were scored with the Beat Filter* • 95 of those were annotated by two annotators (the other 14 were used as test items) • Annotator agreement on the Beat Filter questions was very low: – – Question 1: K = 0. 43 Question 2: K = 0. 31 Question 3: K = 0. 18 Question 4: answer is dependent on Q 1, so computing reliability makes no sense *15 gestures were considered to be “obvious” (other gesture types than beats) and not “filtered” by annotator A …! Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 10

Using the Beat Filter (2) Agreement on total Beat Filter scores: • 44. 2%

Using the Beat Filter (2) Agreement on total Beat Filter scores: • 44. 2% same score (but possibly on different grounds!) • 36. 8% difference of 1 • 16. 8% difference of 2 • 2. 1% difference of 3 In the end, only the scores of annotator A were used. Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 11

Annotating gesture types • Gesture types based on global shape information: resemblance to mentioned

Annotating gesture types • Gesture types based on global shape information: resemblance to mentioned object, finger pointing, directional component, etc. (in combination with speech) Annotator agreement: • Agreed on 83. 3% of gesture types (102 of 124), K=0. 73 – Of these, 33. 3% are beats (34 of 102) • Disagreed on 17. 7% of gesture types (22 of 124) – Most confused were point and iconic (45. 5%) – Next most confused were beat and point (13. 6%) Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 12

Gesture types and beat score 0 beat 1 3 2 30 3 4 5+

Gesture types and beat score 0 beat 1 3 2 30 3 4 5+ NF* # 1 ‘multi-beat’ 34 27. 4 4 3. 2 2 10 8. 1 4 iconic point % 4 2 2 14 7 12 8 3 10 54 43. 5 not agreed 1 3 5 7 2 1 3 22 17. 7 # 4 47 17 25 12 4 15 124 100 • Beats do have lower Beat Filter scores • Many pointing gestures have low scores too *NF (Not Filtered) gesture types were not entirely obvious after all…! Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 13

When are beat gestures used? Some direction giving concept categories were defined: • Directions

When are beat gestures used? Some direction giving concept categories were defined: • Directions (up, down, left, right, …) • (Other) Spatial information (through, in, on, at, across, …) • Duration & Timing (all the way, continue, immediately, …) • Landmarks – Nouns (windows, a square, the hallway, …) – Pronouns (that, the same, this, they, it, …) • Points in Time or Space (now, then, here, there, …) • Hesitations (uh, uhm, I would say, something like that, maybe, …) Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 14

Concepts / gestures overview Concept # Beat (%) Other (%) Total (%) Spatial info

Concepts / gestures overview Concept # Beat (%) Other (%) Total (%) Spatial info 45 6 (13. 3%) 3 (6. 7%) 9 (20. 0%) Hesitations 95 9 (9. 5%) 4 (4. 2%) 13 (13. 7%) Duration & timing 37 3 (8. 1%) 1 (2. 7%) 4 (10. 8%) 7 (4. 0%) 30 (17. 0%) 37 (21. 0%) Landmarks 176 pronoun 49 6 (12. 2%) 9 (18. 3%) 4 (3. 1%) 25 (19. 7%) 29 (22. 8%) 102 4 (3. 9%) 12 (11. 8%) 16 (15. 7%) 84 1 (1. 2%) 39 (46. 4%) 40 (47. 6%) Total 539 30 (5. 6%) 89 (16. 5%) 119 (22. 1%) noun 127 Points in time/space Directions Other n/a 3 (6. 1%) 4 (n/a) 10 (n/a) Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 14 (n/a) 15

Concept Categories Relative frequency of concept categories: • Landmarks are most frequently mentioned •

Concept Categories Relative frequency of concept categories: • Landmarks are most frequently mentioned • Directions are only in fourth place Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 16

Concepts and gestures SI = Spatial Information; H = Hesitations; DT = Duration &

Concepts and gestures SI = Spatial Information; H = Hesitations; DT = Duration & Timing; L = Landmarks; PTS = Points in Time or Space; D = Directions • Not fitting into these categories: 4 beats, 10 “other gestures” Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 17

Landmarks: pronoun or noun • Landmarks as pronouns: fewer gestures, relatively more beats Brandhorst

Landmarks: pronoun or noun • Landmarks as pronouns: fewer gestures, relatively more beats Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 18

A simple beat usage model The probability that a beat gesture B is generated

A simple beat usage model The probability that a beat gesture B is generated to accompany an utterance u (and modelling speaker s): P(B|u) = P(B|Cu) x ms where • Cu is the concept category of u • P(B|Cu) is the probability of B accompanying Cu based on corpus data • ms is an optional multiplier for speaker s (weight factor) Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 19

Toward a better model Other factors than just corpus frequency should be taken into

Toward a better model Other factors than just corpus frequency should be taken into account. For example: • First or second time the same directions are given? • Listener present or not? • Context: influence of preceding and following concepts / gestures • Etc. And of course, more (and more reliable!) corpus data are needed. Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 20

Conclusions How can we recognize beats? • Applying Beat Filter to recognize beat gestures

Conclusions How can we recognize beats? • Applying Beat Filter to recognize beat gestures may not give reliable results • “Impressionistic” gesture type annotation was more reliable • Add “directionality” and “hand shape” to Beat Filter? When are beats used? • “Other” gestures don’t always take precedence over beats • Beats mark spatial information, hesitations, duration and timing more often than other gestures do Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 21

Future work • More data • More reliable annotation • Investigate when beats or

Future work • More data • More reliable annotation • Investigate when beats or other gestures are used given a concept category • Better / more general concept categories? → Implement in the Virtual Guide Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 22

The End Questions? Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture

The End Questions? Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop 2009 23