MultiModal Dialogue in HumanRobot Conversation Information States in

Multi-Modal Dialogue in Human-Robot Conversation Information States in a Multimodal Dialogue System for Human-Robot Conversation (Lemon et al. 2001) Multi-tasking and Collaborative Acitivities in Dialogue Systems (Lemon et al. 2002) Edith Klee June 3, 2003

Overview • Mobile Robots • Requirements for Human-Robot conversation • The WITAS Dialogue System • Dialogue Management in WITAS • Message Generation in WITAS • Demo videoclip

Mobile Robots • move about their environment using wheels, legs, or similar mechanisms, e. g. unmanned land vehicles (ULV), unmanned air vehicles (UAV), autonomous underwater vehicles (AUV) • device sensors (e. g. camera, mouse gesture input) may give rise to new information at any time • no predictable course of actions or conversation • no strict endpoint to action or conversation • simple form-filling/data-base query style dialogue not sufficient

Requirements on dialogue management in conversational interaction with mobile robots • Asynchronicity events in the dialogue scenario can happen at overlapping time periods • Mixed task-initiative operator and system will introduce issues for discussion • Open-ended no clear start/end points for (sub-)dialogues, no rigid pre-determined goals for interchanges • Resource-bounded participants' actions must be generated and produced in time enough to be effective dialogue contributions • Simultaneous participants can produce and receive actions simultaneously

The WITAS Dialogue System • robot helicopter (UAV, 'Unmanned Aerial Vehicle') • mission goals provided by a human operator, conversation via spoken dialogue • Open Agent Architecture (OAA 2) NL Gemini Parser and Generator DM Dialogue Manager OAA 2 facilitator SR Nuance Speech Recognizer TTS Festival Speech Synthesizer ROBOT Control and Report GUI Interactive Map Display ROBOT

Dialogue Management in WITAS • Dialogue Management: 1) dialogue modelling (representation) 2) dialogue control (algorithm) • main tasks of the DM in WITAS: - interpretation spoken language and map-gestures (mouse click) inputs as commands - interpretation of queries, responses, declarations to the robot - generation of synthesized speech and graphical output to express the robot's responses questions, reports about the environment • The current DM supports: - ambiguity resolution - presupposition checking - processing of anaphoric and deictic expressions - command revision - report generation, confirmation backchannel

Dialogue Manager in WITAS (contd. ) • implemetation of a dynamic information state model • creates and updates Information States (IS) • dialogue moves have the effect of updating information states • moves can be initiated by both the operator and the robot • parts of an information state (2001): Issues Raised (IR) stack – a stack or public unresolved issues raised in the dialogue System Agenda – issues to be raised by system Salience List – the objects referenced in the dialogue so far Modality Buffer – keeps track of mouse gestures Databases – dynamic objects, planned routes, geographical information, names

Parts of an Information State (2002) • Dialogue Move Tree (DMT) structured history of dialogue moves and ‘threads’ + list of ‘active nodes’ • Activity Tree (AT) temporal and hierarchical structure of activities specified by the user or planned/initiated by the system + their execution status • System Agenda (SA) stores the communicative goals of the system • Pending List (PL) stores question that the system has asked, but which the user has not answered, they may be re-raised • Salience List (SL) objects referenced in the dialogue so far, ordered by recency • Modality Buffer (MB) keeps track of mouse gestures (until bound to deictic expressions or recognized as purely gestural expressions)

Dialogue Manager Architecture INDEXICAL INFORMATION SALIENCE LIST (NPs, Activities) ACTIVITY MODEL Conversational Move Inputs (parsed human speech) DIALOGUE MOVE TREE (Active Node List) SYSTEM AGENDA ACTIVITY TREE MODALITY BUFFER Map Display Inputs (mouse clicks) PENDING LIST MESSAGE GENERATION (Selection and Aggregation) DEVICE ACTIVITY LAYER

Example INDEXICAL O: Try to find the red car! U: Do you mean the big red car? O: Oh look! The car is here! S: So you mean the big red car? O: Yes! That car. ACTIVITY MODEL INFORMATION SALIENCE LIST (NPs, Activities) DIALOGUE MOVE TREE (Active Node List) SYSTEM AGENDA ACTIVITY TREE MODALITY BUFFER Map Display Inputs (mouse clicks) PENDING LIST MESSAGE GENERATION (Selection and Aggregation) DEVICE ACTIVITY LAYER

A Dialogue Move tree is. . . • constructed for any ongoing dialogue to represent the current state of the conversation • input: logical forms from the parsing process, e. g. "go to the tower": command([go], [param-list ([pp-loc(to, arg([np(det([def], the), [n(tower, sg)])]) • a history ("message board") of dialogue contributions, organized by "thread", based on activities • classifies whether incoming utterances can be interpreted in the current dialogue context, delimits a space of possible IS update functions • has an Active Node List which controls the order in which this function space is searched • classifies how incoming utterances are to be interpreted in the current dialogue context

Example Dialogue Operator: Fly to the tower! UAV: I plan to fly to the tower. UAV: Now taking off. UAV: I have taken off. UAV: Now flying there. Operator: Make that the police station. UAV: I plan to fly to springfield police station. UAV: I have cancelled flying to the tower UAV: Now flying to springfield police station. .

Example Dialogue Move Tree active nodes are red; position on active node list in parens [0=most active] • Root (1) Root • Command (0) "Fly to the tower" command([go], [param_list([pp_loc(to, arg([np(det([def], the. . . • Report "I plan to fly to the tower" report(inform, agent([np([n(uav, sg)])]), confirm_activity. . .

Example Dialogue Move Tree active nodes are red; position on active node list in parens [0=most active] • Root (1) Root • Command (0) "Fly to the tower" command([go], [param_list([pp_loc(to, arg([np(det([def], the. . . • Report "Now taking off" report(inform, agent([np([n(uav, sg)])]), curr_activity. . . • Report "I plan to fly to the tower" report(inform, agent([np([n(uav, sg)])]), confirm_activity. . .

Example Dialogue Move Tree active nodes are red; position on active node list in parens [0=most active] • Root (1) Root • Command (0) "Fly to the tower" command([go], [param_list([pp_loc(to, arg([np(det([def], the. . . • Report "Now taking off" report(inform, agent([np([n(uav, sg)])]), curr_activity. . . • Report "I have taken off" report(inform, agent([np([n(uav, sg)])]), compl_activity. . . • Report "I plan to fly to the tower" report(inform, agent([np([n(uav, sg)])]), confirm_activity. . .

Example Dialogue Move Tree active nodes are red; position on active node list in parens [0=most active] • Root (1) Root • Command (0) "Fly to the tower" command([go], [param_list([pp_loc(to, arg([np(det([def], the. . . • Report "Now taking off" report(inform, agent([np([n(uav, sg)])]), curr_activity. . . • Report "I have taken off" report(inform, agent([np([n(uav, sg)])]), compl_activity. . . • Report "Now flying there" report(inform, agent([np([n(uav, sg)])]), curr_activity. . . • Report "I plan to fly to the tower" report(inform, agent([np([n(uav, sg)])]), confirm_activity. . .

Example Dialogue Move Tree active nodes are red; position on active node list in parens [0=most active] • Root (1) Root • Command (0) "Fly to the tower" command([go], [param_list([pp_loc(to, arg([np(det([def], the. . . • Report "Now taking off" report(inform, agent([np([n(uav, sg)])]), curr_activity. . . • Report "I have taken off" report(inform, agent([np([n(uav, sg)])]), compl_activity. . . • Report "Now flying there" report(inform, agent([np([n(uav, sg)])]), curr_activity. . . • Report "I plan to fly to the tower" report(inform, agent([np([n(uav, sg)])]), confirm_activity. . . • Revision "Make that the police station" revision([replace([arg([np(det([def], the), [n(police. . .

Example Dialogue Move Tree active nodes are red; position on active node list in parens [0=most active] • Root (1) Root • Command (0) "Fly to the tower" command([go], [param_list([pp_loc(to, arg([np(det([def], the. . . • Report "Now taking off" report(inform, agent([np([n(uav, sg)])]), curr_activity. . . • Report "I plan to fly to springfield police station" report(inform, agent([np([n(uav, sg)])]), confirm. . . • Report "I have taken off" report(inform, agent([np([n(uav, sg)])]), compl_activity. . . • Report "Now flying there" report(inform, agent([np([n(uav, sg)])]), curr_activity. . . • Report "I plan to fly to the tower" report(inform, agent([np([n(uav, sg)])]), confirm_activity. . . • Revision "Make that the police station" revision([replace([arg([np(det([def], the), [n(police. . .

Message Generation • the system has to communicate about: - its perceptions of a changing environment - progress towards user-specific goals - execution status of activities or tasks - its own internal state changes - the progress of the dialogue itself • problems that may arise: - multi-tasking robot, different activities at the same time - dialogue contribution arise incrementally - generation of too many utterances: overload of information - generation of too little utterances: no establishment and maintenance of an appropriate context. - large pieces of text not appropriate

Requirements for Message Generation • flexibility: message selection and generation needs to be more flexible than template-based approaches • aggregation: aggregation rules must be sensitive to incremental aspects • relevance/recency filtering: message filtering according to relevance • echoing: usage of the same language as the user, i. e. usage of anaphoric expressions where possible • variability: expressing the same content in a variety of ways • real time generation to keep the conversation natural

Message Selection - Filtering • selection and generation module • input: logical forms, describing the communicative goals of the system - context tags (e. g. activity identifier, dialogue move tree node) - content logical form (e. g. report, wh-question) - priority tag (e. g. warn, inform) - e. g. report(inform, agent(Agent. ID), cancel-activity(Activity. ID)), when Agent. ID = robot and Activity. ID = tower the result is: "I cancelled flying to the tower". • items considered for generation are placed on the system agenda or pending list • messages more urgent than others carry a label "warning" and pass any filtering • Echoing (using anaphoric expressions) is achieved by accessing the Salience List whenever possible

Thank You!