Dialogue Systems Julia Hirschberg CS 4705 12302021 1

Dialogue Systems Julia Hirschberg CS 4705 12/30/2021 1

Today • Examples from English and Swedish • Controlling the dialogue flow – State prediction • Influencing user behavior – Entrainment • Learning from human-human dialogue – User feedback • Role of ‘personality’ in SDS • Evaluating SDS 12/30/2021 2

Issues in Designing SDS • Coverage: functionality and vocabulary • Dialogue control: – System – User – Mixed initiative • Confirmation strategies: – Explicit – Implicit – None 12/30/2021 3

The Waxholm Project at KTH • Tourist information • Stockholm archipelago • time-tables, hotels, hostels, camping and dining possibilities. • Mixed initiative dialogue • speech recognition • multimodal synthesis • Graphic information • pictures, maps, charts and time-tables • Demos at http: //www. speech. kth. se/multimodal 12/30/2021 4

The Waxholm system 12/30/2021 There Information Which are. When IIs IWaxholm am lots Which think it day This Where looking possible of Ido about Where of want about Iboats ishotels the want the Thank a is can Thank table hotels the evening to The for week to the shown from isto go are Ieat restaurants boats Waxholm? you find hotels city go oftomorrow you is in Stockholm do in the to on boats shown too Waxholm? hotels? Waxholm? to you Waxholm boats. . . in this Waxholm want depart? in map in. Waxholm to this to. Waxholm go? table is on a Friday, Fromis. At shown where shown whatin do time inthis you this do table want table youto want go to go? 5

Dialogue Control: Predicting Dialogue State • Dialog grammar specified by a number of states – Each state associated with an action – Database search, system question… … – Probable state determined from semantic features • Train transition probabilities from state to state • Dialog control design tool with a graphic interface 12/30/2021 6

Waxholm Topics TIME_TABLE Task: get a time-table. Example: När går båten? (When does the boat leave? ) SHOW_MAP Task : get a chart or a map displayed. Example: Var ligger Vaxholm? (Where is Vaxholm located? ) EXIST Task : display lodging and dining possibilities. Example: Var finns det vandrarhem? (Where are there hostels? ) OUT_OF_DOMAIN Task : the subject is out of the domain. Example: Kan jag boka rum. (Can I book a room? ) NO_UNDERSTANDING Task : no understanding of user intentions. Example: Jag heter Olle. (My name is Olle) END_SCENARIO Task : end a dialog. Example: Tack. (Thank you. ) 12/30/2021 7

Topic selection FEATURES TOPIC EXAMPLES TIME TABLE SHOW MAP OBJECT QUEST-WHEN QUEST-WHERE FROM-PLACE AT-PLACE . 062. 188. 062. 250. 062 . 312. 031. 688. 031. 219 . 073. 024. 390. 024. 293 . 091 . 067 . 091 TIME PLACE OOD END HOTEL HOSTEL ISLAND PORT MOVE . 312. 091. 062. 333. 125. 875 . 031. 200. 031. 556. 750. 031 . 024. 500. 122. 024. 488. 122. 062. 244. 098 . 091 . 067. 933. 067 . 091. 909. 091 12/30/2021 FACILITY NO UNDER- OUT OF STANDING DOMAIN { p(ti | F )} argmax i END 8

Topic prediction results % Errors 15 12, 9 8, 8 10 5 12, 7 8, 5 All “no understanding” excluded 3, 1 2, 9 0 complete parse 12/30/2021 raw data no extra linguistic sounds 9

Entrainment (Adaptation, Accommodation, Alignment) • Hypothesis: over time, people tend to adapt their communicative behavior to that of their conversational partner • Issues – What are the dimensions of entrainment? – How rapidly do people adapt? – Does entrainment occur (on the human side) in human/computer conversations? – Can this be used to the system’s advantage? The user’s? 12/30/2021 10

Varieties of Entrainment… • Lexical: S and H tend over time to adopt the same method of referring to items in a discourse A: It’s that thing that looks like a harpsichord. B: So the harpsichord-looking thing…. . B: The harpsichord… • Phonological – Word pronunciation: voice/voiceless /t/ in better • Acoustic/Prosodic – Speaking rate, pitch range, choice of contour • Discourse/dialogue/social – Marking of topic shift, turn-taking 12/30/2021 11

The Vocabulary Problem • Furnas et al ’ 87: the probability that 2 subjects will producing the same name for a command for common computer operations varied from. 07 -. 18 – Remove a file: remove, delete, erase, kill, omit, destroy, lose, change, trash – With 20 synonyms for a single command, the likelihood that 2 people will choose the same one was 80% – With 25 commands, the likelihood that 2 people who choose the same term think it means the same command was 15% • How can people possibly communicate? – They collaborate on choice of referring expressions 12/30/2021 12

Early Studies of Priming Effects • Hypothesis: Users will tend to use the vocabulary and syntax the system uses – Evidence from data collections in the field • Systems should take advantage of this proclivity to prime users to speak in ways that the system can recognize well 12/30/2021 13

User Responses to Vaxholm The answers to the question: “What weekday do you want to go? ” (Vilken veckodag vill du åka? ) • • • 22% 11% 7% 6% • - 12/30/2021 Friday (fredag) I want to go on Friday (jag vill åka på fredag) I want to go today (jag vill åka idag) on Friday (på fredag) I want to go a Friday (jag vill åka en fredag) are there any hotels in Vaxholm? (finns det några hotell i Vaxholm) 14

Verb Priming: How often do you go abroad on holiday? Hur ofta åker du utomlands på semestern? Hur ofta reser du utomlands på semestern? jag åker en gång om året kanske jag åker ganska sällan utomlands på semester jag åker nästan alltid utomlands under min semester jag åker ungefär 2 gånger per år utomlands på semester jag åker utomlands nästan varje år jag reser en gång om året utomlands jag reser inte ofta utomlands på semester det blir mera i arbetet jag reser utomlands på semestern vartannat år jag reser utomlands en gång per semester jag reser utomlands på semester ungefär en gång per år jag brukar resa utomlands på semestern åtminståne en jag åker utomlands på semestern varje år jag åker utomlands ungefär en gång om året jag är nästan aldrig utomlands en eller två gånger om året en gång per semester kanske en gång per år ungefär en gång per år åtminståne en gång om året nästan aldrig gång i året en gång per år kanske en gång vart annat år varje år vart tredje år ungefär nu för tiden inte så ofta varje år brukar jag åka utomlands 12/30/2021 15

Results no no reuse 4% 2%answer other 24% reuse 52% 18% ellipse 12/30/2021 16

Lexical Entrainment in Referring Expressions • Choice of Referring Expressions: Informativeness vs. availability (basic level or not) vs. saliency vs. recency • Gricean prediction – People use descriptions that minimally but effectively distinguish among items in the discourse • Garrod & Anderson ’ 87 Output/Input Principle – Conversational partners formulate their current utterance according to the model used to interpret their partner’s most recent utterance • Clark, Brennan, et al’s Conceptual Pacts – People make Conceptual Pacts wrt appropriate referring expressions made with particular conversational partners – They are loath to abandon these even when shorter expressions possible 12/30/2021 17

Entrainment in Spontaneous Speech S 13: the orange M&M looking kind of scared and then a one on the bottom left and a nine on the bottom right S 12: alright I have the exact same thing I just had it's an M&M looking scared that's orange S 13: yeah the scared M&M guy yeah S 12: framed mirror and the scared M&M on the lower right S 13: and it's to the right of the scared M&M guy S 13: yeah and the iron should be on the same line as the frightened M&M kind of like an L S 12: to the left of the scared M&M to the right of the onion and above the iron 12/30/2021 18

Extraterrestrial vs Alien I s 11: okay in the middle of the card I have an extraterrestrial figure… s 11: okay middle of the card I have the extraterrestrial … s 10: I've got the blue lion with the extraterrestrial on the lower right s 11: okay I have the extraterrestrial now and then I have the eye at the bottom right corner s 10: my extraterrestrial's gone 12/30/2021 19

Extraterrestrial vs. Alien II S 03: okay I have a blue lion and then the extraterrestrial at the lower right corner S 11: mm I'll pass I have the alien with an eye in the lower right corner S 03: um I have just the alien so I guess I'll match that ---------------------------------------S 10: yes now I've got that extraterrestrial with the yellow lion and the money … S 12: oh now I have the blue lion in the center with our little alien buddy in the right hand corner S 10: with the alien buddy so I'm gonna match him with the single blue lion okay I've got our alien with the eye in the corner 12/30/2021 20

Timing and Voice Quality • Guitar & Marchinkoski ’ 01: – How early do we start to adapt to others’ speech? – Do children adapt their speaking rate to their mother’s speech? • Study: – 6 mothers spoke with their own (normally speaking) 3 yr-olds (3 M, 3 F) – Mothers’ rates significantly reduced (B) or not (A) in AB-A-B design • Results: – 5/6 children reduced their rates when their mothers spoke more slowly 12/30/2021 21

• Sherblom & La Riviere ’ 87: How are speech timing and voice quality affected by a non-familiar conversational partner? • Study: – 65 pairs of undergraduates asked to discuss a ‘problem situation’ together – Utter a single sentence before and after the conversation – Sentences compared for speaking rate, utterance length and vocal jitter • Results: – Substantial influence of partner on all 3 measures – Interpersonal uncertainty and differences in arousal influenced degree of adaptation 12/30/2021 22

Amplitude and Response Latency • Coulston et al ’ 02: – Do humans adapt to the behavior of non-human partners? – Do children speak more loudly to a loud animated character? • Study: – 24 7 -10 -yr olds interacted with an extroverted, loud animated character and with an introverted, soft character (TTS voices) – Multiple tasks using different amplitude ranges – Human/TTS amplitudes and latencies compared • Results: – 79 -94% of children adapted their amplitude, bidirectionally – Also adapted their response latencies (mean 18. 4%), 23 12/30/2021 bidirectionally

Social Status and Entrainment • Azuma ’ 97: Do speakers adapt to the style of other social classes? • Study: Emperor Hirohito visits the countryside – Corpus-based study of speech style of Japanese Emperor Hirohito during chihoo jyunkoo (`visits to countryside‘), 1946 -54 – Published transcripts of speeches – Findings: • Emperor Hirohito converged his speech style to that of listeners lower in social status – Choice of verb-forms, pronouns no longer those of person with highest authority – Perceived as like those of a (low-status) mother 12/30/2021 24

Socio-Cultural Influences and Entrainment • Co-teachers adapt teaching styles (Roth ’ 05) – Social context • High school in NE with predominantly African. American student body • Cristobal: Cuban-African-American teacher • Chris: new Italian-American teacher – Adaptation of Chris to Cristobal • Catch phrases (e. g. right!, really hot) and their production: pitch and intensity contours • Pitch ‘matching’ across speakers – Mimesis vs entrainment 12/30/2021 25

Conclusions for SDS • Systems can make use of user tendency to entrain to system vocabulary • Should systems also entrain to users? – CMU’s Let’s Go adapts confirmation prompts to non-native speech: Finds closest match to user input in system vocabulary 12/30/2021 26

Evidence from Human Performance • Users provide explicit positive and negative feedback • Corpus-based vs. laboratory experiments – do these tell us different things? 12/30/2021 27

The August system 12/30/2021 People IWhat IStrindberg IYes, Over call The can How come Strindberg The Perhaps myself answer that information who amany from Royal million was live we Strindberg, questions the people was Institute ain will people smart department glass married ishere? shown live thing about of houses live but in I Yes, When What it. Do You do might you Thank you Good were are is was like your be do welcome! bye! you! that born for it name? born? we ameet inliving? 1849 will! Strindberg, ofdon’t Speech, should in the really Technology! three Stockholm? on soon Stockholm not KTH Music tothe have throw say! times! again! map and a surname Stockholm stones area Hearing 28

Adapt – demonstration of ”complete” system 12/30/2021 29

Feedback and ‘Grounding’: Bell & Gustafson ’ 00 • Positive and negative – Previous corpora: August system • 18% of users gave pos or neg feedback in subcorpus • Push-to-talk • Corpus: Adapt system – 50 dialogues, 33 subjects, 1845 utterances – Feedback utterances labeled w/ • Positive or negative • Explicit or implicit • Attention/Attitude • Results: – 18% of utterances contained feedback – 94% of users provided 12/30/2021 30

– 65% positive, 2/3 explicit, equal amounts of attention vs. attitude – Large variation • Some subjects provided at almost every turn • Some never did • Utility of study: – Use positive feedback to model the user better (preferences) – Use negative feedback in error detection 12/30/2021 31

The HIGGINS domain This is a 3 D test environment • • The primary domain of HIGGINS is city navigation for pedestrians. Secondarily, HIGGINS is intended to provide simple information about the immediate surroundings. 12/30/2021 32

Initial experiments • Studies on human-human conversation • The Higgins domain (similar to Map Task) • Using ASR in one direction to elicit error handling behaviour User 12/30/2021 Speaks ASR Listens Vocoder Reads Speaks Operator 33

Non-Understanding Error Recovery (Skantze ’ 03) • Humans tend not to signal non-understanding: O: Do you see a wooden house in front of you? U: ASR: YES CROSSING ADDRESS NOW (I pass the wooden house now) O: Can you see a restaurant sign? • This leads to – Increased experience of task success – Faster recovery from non-understanding 12/30/2021 34

Personality and Computer Systems • Early-pc-era reports that significant others were jealous of the time their partners spent with their computers. • Reeves & Nass, The Media Equation How People Treat Computers, Television, and New Media Like Real People and Places, 1996 – Evolution explains the anthropomorphization of the pc • Humans evolved over millions of years without media • Proper response to any stimulus was critical to survival • Human psychology and physiological responses well developed before media invented • Ergo, our bodies and minds react to media, immediately and fundamentally, as if they were real 12/30/2021 35

People See ‘Personality’ Everywhere • Humans assess personality of another (human or otherwise) quickly, with minimal clues • Perceived computer personality strongly affects how we evaluate the computer and information it provides • Experiments: – Created “dominant” and “submissive” computer interfaces and asked subjects to use to solve hypothetical problems – Max (dominant) used assertive language, showed higher confidence in the information displayed (via a numeric scale), always presented its own analysis of the problem first – Linus (submissive) phrased information more tentatively, rated its own information at lower confidence levels, and allowed human to discuss problem first – Each used alternately by people whose personalities previously identified as being either dominant or submissive 12/30/2021 36

User Reactions • Users described Max and Linus in human terms: aggressive, assertive, authoritative vs. shy, timid, submissive – Users correctly identified machines more like themselves – Users rated machines more like themselves as better computers even though content received exactly the same. – Users rated their own performance better when machine’s personality matched theirs • People more frank when rating a computer if questionnaire presented on another machine • Subjects thought highly of computers that praised them, even if praise clearly undeserved 12/30/2021 37

Personality in SDS • Mairesse & Walker ’ 07 PERSONAGE (PERSONAlity GEnerator) – ‘Big 5’ personality trait model: extroversion, neuroticism, agreeableness, conscientiousness, openness to experience – Attempts to generate “extroverted” language based on traits associated with extroversion in psychology literature – Demo: find your personality type 12/30/2021 38

12/30/2021 39

Conclusions for SDS • Systems can be designed to convey different personalities • Can they recognize users’ personalities and entrain to them? • Should they? 12/30/2021 40

Evaluating Dialogue Systems • PARADISE framework (Walker et al ’ 00) • “Performance” of a dialogue system is affected both by what gets accomplished by the user and the dialogue agent and how it gets accomplished Maximize Task Success 12/30/2021 Minimize Costs Efficiency Measures Qualitative Measures 41

Task Success • Task goals seen as Attribute-Value Matrix ELVIS e-mail retrieval task (Walker et al ‘ 97) “Find the time and place of your meeting with Kim. ” Attribute Value Selection Criterion Kim or Meeting Time 10: 30 a. m. Place 2 D 516 • Task success defined by match between AVM values at end of with “true” values for AVM 12/30/2021 42

Metrics • Efficiency of the Interaction: User Turns, System Turns, Elapsed Time • Quality of the Interaction: ASR rejections, Time Out Prompts, Help Requests, Barge-Ins, Mean Recognition Score (concept accuracy), Cancellation Requests • User Satisfaction • Task Success: perceived completion, information extracted 12/30/2021 43

Experimental Procedures • Subjects given specified tasks • Spoken dialogues recorded • Cost factors, states, dialog acts automatically logged; ASR accuracy, barge-in hand-labeled • Users specify task solution via web page • Users complete User Satisfaction surveys • Use multiple linear regression to model User Satisfaction as a function of Task Success and Costs; test for significant predictive factors 12/30/2021 44

User Satisfaction: Sum of Many Measures • Was Annie easy to understand in this conversation? (TTS Performance) • In this conversation, did Annie understand what you said? (ASR Performance) • In this conversation, was it easy to find the message you wanted? (Task Ease) • Was the pace of interaction with Annie appropriate in this conversation? (Interaction Pace) • In this conversation, did you know what you could say at each point of the dialog? 12/30/2021 (User Expertise) • How often was Annie sluggish and slow to reply to you in this conversation? (System Response) • Did Annie work the way you expected her to in this conversation? (Expected Behavior) • From your current experience with using Annie to get your email, do you think you'd use Annie regularly to access your mail when you are away from your desk? (Future Use) 45

Performance Functions from Three Systems • ELVIS User Sat. =. 21* COMP +. 47 * MRS -. 15 * ET • TOOT User Sat. =. 35* COMP +. 45* MRS -. 14*ET • ANNIE User Sat. =. 33*COMP +. 25* MRS +. 33* Help – COMP: User perception of task completion (task success) – MRS: Mean recognition accuracy (cost) – ET: Elapsed time (cost) – Help: Help requests (cost) 12/30/2021 46

Performance Model • Perceived task completion and mean recognition score are consistently significant predictors of User Satisfaction • Performance model useful for system development – Making predictions about system modifications – Distinguishing ‘good’ dialogues from ‘bad’ dialogues • But can we also tell on-line when a dialogue is ‘going wrong’ 12/30/2021 47

Next • Generation: Summarization 12/30/2021 48