Turntaking and Disfluencies Julia Hirschberg CS 4706 1162022

Turn-taking and Disfluencies Julia Hirschberg CS 4706 1/16/2022 1

Today • Turn-taking behaviors – Conversational Analysis – Importance in real systems • Disfluencies – How to model? Detect? – Role in human-human interaction – Importance in real systems? 1/16/2022 2

Turn-taking • Expected patterns of behavior – Deviation is significant – How do we find the patterns? • • Ordinary conversation Telephone talk Meetings Email? – Who looks for these? 1/16/2022 3

• Terminology – Adjacency pairs – Preference – Pre-sequence – Repair • Examples: – Telephone openings, closings – Broadcasts 1/16/2022 4

Could this be useful when we build SDS? • What do we expect to hear? • What should we produce? 1/16/2022 5

Auditory Cues to Turn-Taking • M. Schegloff “Reflections on studying prosody in talk-in-interaction, ” Language and Speech 41, 1999. (Michael Mu. ) • H. Koiso et al ‘ 99 “An analysis of turntaking and backchannels based on prosodic and syntactic features…, ” Language and Speech 41, 1999. (Sarah) 1/16/2022 6

Disfluencies and Self-Repairs • Are these just ‘noise’? – For people • S. Brennan & M. Williams, “The Feeling of Another’s Knowing, ” J Memory and Language 34, 1995. (Judd) • S. Brennan & Schober, “How listeners compensate for disfluencies in spontaneous speech, ” J Memory and Language 44, 2001. (Aron) – For parsers – For speech recognizers 1/16/2022 7

Hindle ’ 83: Finding the Edit Signal • If we have it, can we ‘repair’ the self-repair automatically? – Builds a correcting parser, Fidditch, for spontaneous speech – Given a string with an edit signal marked, produces a ‘repaired’ version I was * I am really annoyed – If X 1 * X 2 are similar linguistic elements separated by an edit signal, replace X 1 w/X 2 1/16/2022 8

What does it mean to be the same • Same surface string Well if they’d * if they’d… • Same category I was just that * the kind of guy… • Same constituent I think that you get * it’s more strict in Catholic schools • Restarts are completely different… I just think * Do you want something to eat? 1/16/2022 9

Bear et al ’ 92: Detecting and Correcting Self-Repairs • Use multiple knowledge sources but not edit signal – Lexical pattern matching – Parsing failure + pattern matching + re-parsing – Acoustic information: pause, peak F 0, – Cue words: well, no – Fragments 1/16/2022 10

But…is there an edit signal? 1/16/2022 11

1/16/2022 12

RIM Model of Self-Repairs (Nakatani & Hirschberg ’ 94) • ATIS corpus – 6414 turns with 346 (5. 4%) repairs, 122 speakers – Hand-labeled for repairs and prosodic features • Findings: – Reparanda: 73% end in fragments, 30% in glottalization, co-articulatory gestures – DI: pausal duration differs significantly from fluent boundaries, small increase in f 0 and amplitude 1/16/2022 13

Does it identify self-repairs reliably? • CART prediction: 86% precision, 91% recall – Duration of interval, presence of fragment, pause filler, p. o. s. , lexical matching across DI • Are there edit signals? 1/16/2022 14

Next Week • Spoken Dialogue Systems • Andy, David and Vera reporting 1/16/2022 15