Nespoles Experiment on Multimodality Summer 2001 Erica Costantini

Nespole!’s Experiment on Multimodality (Summer 2001) Erica Costantini (University of Trieste) Fabio Pianesi (ITC-irst, Trento) Susanne Burger (CMU)

Experiment Objectives • Testing hypothesis concerning the added value of multimodality in a ‘true’ speech-tospeech translation environment • Usability evaluation of the Nespole! System E. Costantini, S. Burger, and F. Pianesi, NESPOLE! Multilingual and Multimodal Corpus, in Proceedings of LREC 2002, Las Palmas, Spain. E. Costantini, F. Pianesi, S. Burger. The Added Value of multimodality in the NESPOLE! Speech-to-Speech Translation System: an Experimental Study. To be published in proceedings of ICMI 2002. S. Burger, E. Costantini, and F. Pianesi, NESPOLE! Deliverable D 5 - Study on Multimodality, part 1, 2002. In NESPOLE! Project website: .

Multimodality Speech + Pen-based Gestures

Multimodality Previous Research 1. The advantages of multimodal input over spoken input includes fewer errors, fewer spontaneous disfluences, briefer and less complex language, greater satisfaction 2. Multimodal interaction occurs more frequently in case of spatial location commands 3. When combined with spoken input, pen-based input can disambiguate badly understood sentences S. L. . Oviatt, “Multimodal interactive maps: Designing for human performance”, Human-Computer Interaction, 1997, pp. 93 -129 (special issue on "Multimodal interfaces"). S. L. . Oviatt, A. De Angeli, and K. Kuhn, “Integration and synchronization of input modes during multimodal human computer interaction”, in Proc. of CHI ‘ 97. ACM Press, New York, 1997, pp. 415 -422. S. L. . Oviatt, “Mutual disambiguation of recognition errors in a multimodal architecture”, in Proc. of CHI '99, ACM Press, New York, 1999, pp. 576 -583.

Multimodality Our Hypothesis 1. Multimodality can increase the probability of successful interaction even with prototypes of “real” multilingual systems, especially when spatial information is the focus of the communicative exchange 2. Multimodality can support a faster recovery from recognition and translation errors.

The Experiment - Method Experimental Design Comparison between the performances of two versions of the system: Ø SO (Speech-Only) version Multilingua Multimedia Only spoken input Ø MM (Multimodal) version: Multilingua Multimedia Spoken and pen-based input

The Experiment - Method Participants 1. CUSTOMERS: 28 1. 14 English and 14 German speakers 2. similar level of computer literacy and web expertise 3. paid volunteers 2. AGENTS: 7 1. Italian speakers trained to act as Trentino tourist board agents 2. researchers univolved with Nespole! Project 3. All participants: • same level of computer literacy and web expertise • sex balanced across conditions and languages

The Experiment - Method Task and Instructions Customers and agents received written instructions concerning the system and the task. Ø Customer’s task: to ask for information in order to choose an appropriate location and a hotel within constraints specified a priori. Ø Agent’s task: to provide the necessary information on the basis of the available materials and following the instructions.

The Experiment - Data Recordings and Transcriptions 1. 56 audio stereo files were recorded 2. Dialogue transcriptions have been done: 1. in accordance to VERBMOBIL conventions 2. using the Trans. Edit annotation tool 3. Besides orthographic words, dialogue transcriptions contain: 1. annotations for spontaneous phenomena 2. annotations for gestures (as a three-line comment added at the end of the corresponding turn)

The Experiment - Data Spoken input Ø Average values and variance for turns, tokens, types, disfluences and spontaneous phenomena are similar for agents and customers and across Languages and experimental conditions (SO and MM). Ø Average dialogue duration: 35 minutes (Range 19– 59 minutes) Turns per dialogue Italian agent German customer English customer 37 39 33 Tokens per dialogue 258 254 218 Types per dialogue 101 103 82 Tokens per turn 6. 98 6. 50 6. 60

The Experiment - Data Pen-Based Input Ø Low number of performed gestures (one every 8 speech turns) Ø Almost all gestures were performed by the agents (98, 1%) Ø Most of the drawing gesture (79%) were performed immediately before speech (no one during the speech). due to push-to-talk procedure? Ø Few or no deictics were used due to time-lag between speech and gestures?

The Experiment - Data Transcriptions alignment The two halves of each dialogue transcriptions were aligned in order to: 1. compare original and translated turns; 2. classify turns into: 1. Successful turns (good translation) 2. Partially successful turns (poor or bad translation, but the target could react properly) 3. Non-successful turns (incomprehensible output) 3. count turn repetitions

The Experiment - Conclusions Summary of the results 1. 86% of achieved goals (customers chose a target hotel) 2. Strong preference for MM 3. Multimodality seems to enhance dialogue effectiveness: 1. it helps in decreasing the number of non-successful turns and repetitions, especially when spatial information is conveyed; 2. it helps in solving misunderstandings; 3. it provides for a better dialogue fluency; 4. it is preferred by users.

The Experiment - Conclusions Discussion 1. We used a real system prototype instead of applying a Wizard of Oz procedure (weaker evidences) 2. The task was the best compromise between the system capabilities and the need to provide for true pen -based gestures 3. The Nespole! Corpus is a relevant source of information in the field of multilingual and multimodal interaction in realistic scenario

The End Questions?

The Experiment - Data Annotation for Gestures Gesture annotation include: Ø gesture identification: • Progressive number • User (agent or customer) • Temporal relationship with the spoken turn (before, during, after) Ø Gesture description: (based on the used White Board commands) Ø Gesture goal: • selection • pointing • connection • words

The Experiment - Data Pen-Based Input Drawings Selection % on all 61% Mode free-hand 65% elliptical 31% rectangular Pointing Connection Words 19% 12% 8% % on class 4% free-hand 100% free-hand 47% line 53% free-hand 100%

The Experiment - Results Turn Successfulness Ø There is a tendency towards fewer non-successful turns in spatial turns all turns legal turns spatial turns Successful turns 29% 31% 29% Partially successful turns 32% 35% 41% Non-successful turns 39% 34% 29%

The Experiment - Results Turn Successfulness Ø There is a tendency towards less non-successful turns in MM condition Ø The tendency is more evident in case of spatial turns Percentages of non-successful turns Eng. Ger. SO MM All turns 33% 27% 34% Legal turns 33% 26% 29% 25% Spatial turns 30% 19% 31% 18%

The Experiment - Results Turn Repetitions Ø There is a tendency towards less repeated turns in MM condition Ø The tendency is more evident in case of spatial turns Percentages of repeated turns Eng. SO MM Ger. SO Ger. MM All turns 16% 20% 18% Legal turns 15% 20% 17% Spatial turns 17% 11% 23% 18%