TurnYielding Cues in TaskOriented Dialogue Agustn Gravano 1

Turn-Yielding Cues in Task-Oriented Dialogue Agustín Gravano 1, 2 Julia Hirschberg 1 (1) Columbia University, New York, USA (2) Universidad de Buenos Aires, Argentina

Introduction Interactive Voice Response Systems • Quickly spreading. • “Uncomfortable”, “awkward”. • ASR+TTS account for most IVR problems. • Other problems revealed. • Coordination of system-user exchanges. • Long pauses after user turns; interruptions. • Modeling turn-taking behavior should lead to improved system-user coordination. Agustín Gravano SIGdial 2009 2

Introduction Goal • Learn when the speaker is likely to end her/his conversational turn. • Find turn-yielding cues. • Cues displayed by the speaker when approaching a potential turn boundary. • This should improve the coordination of IVRs: • Speech understanding: Detect the end of the user’s turn. • Speech generation: Display cues signalling the end of system’s turn. Agustín Gravano SIGdial 2009 3

Talk Outline • • • Previous work Material Method Results Conclusions Agustín Gravano SIGdial 2009 4

Previous Work on Turn-Taking • Duncan 1972, 1973, 1974, inter alia. • Hypothesized 6 turn-yielding cues in face-to-face dialogue. • Conjectured a linear relation between the number of displayed cues and the likelihood of a turn-taking attempt. • Studies formalized and verified some of Duncan’s hypotheses. [For&Tho 96; Wen&Sie 03; Cut&Pea 86; Wic&Cas 01] • Implementations of turn-boundary detection. • Simulations [Fer&al. 02, 03; Edl&al. 05; Sch 06; Att&al. 08; Bau 08] • Actual systems: Let’s Go! [Rau&Esk 08] • Exploiting turn-yielding cues improves performance. Agustín Gravano SIGdial 2009 5

Material Columbia Games Corpus • • 12 task-oriented spontaneous dialogues. Standard American English. 13 subjects: 6 female, 7 male. Series of collaborative computer games. No eye contact. No speech restrictions. 9 hours of dialogue. Manual orthographic transcription, alignment. Manual prosodic annotations (To. BI). Agustín Gravano SIGdial 2009 6

Material Columbia Games Corpus Player 1: Describer Agustín Gravano Player 2: Follower SIGdial 2009 7

Turn-Yielding Cues • Cues displayed by the speaker when approaching a potential turn boundary. Agustín Gravano SIGdial 2009 8

Turn-Yielding Cues Method • IPU (Inter Pausal Unit): Maximal sequence of words from the same speaker surrounded by silence ≥ 50 ms. Hold Speaker A: IPU 1 Smooth switch IPU 2 IPU 3 Speaker B: • Smooth switch: Speaker A finishes her utterance; speaker B takes the turn with no overlapping speech. • Trained annotators distinguished Smooth switches from Interruptions and Backchannels using a scheme based on Ferguson 1977, Beattie 1982. Agustín Gravano SIGdial 2009 9

Turn-Yielding Cues Method Hold Speaker A: IPU 1 Smooth switch IPU 2 IPU 3 Speaker B: • To find turn-yielding cues, we compare: • IPUs preceding Holds, • IPUs preceding Smooth switches. • ~200 features: acoustic, prosodic, lexical, syntactic. Agustín Gravano SIGdial 2009 10

Turn-Yielding Cues Individual Cues Final intonation: 1. • Faster speaking rate. 2. • Reduction of final lengthening. Lower intensity level. Lower pitch level. Higher jitter, shimmer, NHR. 3. 4. 5. • 6. Falling (L-L%) or high-rising (H-H%). Related to perception of voice quality. Longer IPU duration (seconds and #words). Agustín Gravano SIGdial 2009 11

Turn-Yielding Cues Individual Cues 7. Textual completion (independent of intonation). (1) Manually annotated a portion of the data. Labelers read up to the end of a target IPU (no right context), judged whether it could constitute a ‘complete’ utterance. 400 tokens. K=0. 81. (2) Trained an SVM classifier. 19 lexical + syntactic features. Accuracy: 80%. Maj-class baseline: 55%. Human agreement: 91%. (3) Labeled all IPUs in the corpus with the SVM model. Before smooth switches: Before holds: 18% 82% Incomplete 47% 53% Complete (X 2 test, p ~ 0) Agustín Gravano SIGdial 2009 12

Turn-Yielding Cues Individual Cues 1. 2. 3. 4. 5. 6. 7. Final intonation: L-L% or H-H%. Faster speaking rate. Lower intensity level. Lower pitch level. Higher jitter, shimmer, NHR. Longer IPU duration. Textual completion. Agustín Gravano SIGdial 2009 13

Turn-Yielding Cues Defining Presence of a Cue • 2 -3 representative features for each cue: Final intonation Abs. pitch slope over final 200 ms, 300 ms. Speaking rate Syllables/sec, phonemes/sec over IPU. Intensity level Mean intensity over final 500 ms, 1000 ms. Pitch level Mean pitch over final 500 ms, 1000 ms. Voice quality Jitter, shimmer, NHR over final 500 ms. IPU duration Duration in ms, and in number of words. Textual completion Complete vs. incomplete (binary). • Define presence/absence based on whether the value is closer to the mean before S or H. Agustín Gravano SIGdial 2009 14

Top Frequencies of Complex Cues digit == cue present dot == cue absent Turn-yielding cues: 1: Final intonation 2: Speaking rate 3: Intensity level 4: Pitch level 5: IPU duration 6: Voice quality 7: Completion Agustín Gravano SIGdial 2009 15

Turn-Yielding Cues Percentage of turn-taking attempts Combined Cues r 2 = 0. 969 Number of cues conjointly displayed Agustín Gravano SIGdial 2009 16

Turn-Yielding Cues IVR Systems • After each IPU from the user: if estimated likelihood > threshold then take the turn • To signal the end of a system’s turn: Include as many cues as possible in the system’s final IPU. Agustín Gravano SIGdial 2009 17

Summary • Study of turn-yielding cues. • Objective, automatically computable. • Combined cues. • Improve turn-taking decisions of IVR systems. • Results drawn from task-oriented dialogues. • Not necessarily generalizable. • Suitable for most IVR domains. • Interspeech 2009: Study of backchannelinviting cues. Agustín Gravano SIGdial 2009 18

Special thanks to… • Julia Hirschberg • Thesis Committee Members • Maxine Eskenazi, Kathy Mc. Keown, Becky Passonneau, Amanda Stent. • Speech Lab at Columbia University • Stefan Benus, Fadi Biadsy, Sasha Caskey, Bob Coyne, Frank Enos, Martin Jansche, Jackson Liscombe, Sameer Maskey, Andrew Rosenberg. • Collaborators • Gregory Ward and Elisa Sneed German (Northwestern U); Ani Nenkova (UPenn); Héctor Chávez, David Elson, Michel Galley, Enrique Henestroza, Hanae Koiso, Shira Mitchell, Michael Mulley, Kristen Parton, Ilia Vovsha, Lauren Wilcox. Agustín Gravano SIGdial 2009 19