Multimodal Information Access Using Speech and Gestures Norbert

Multimodal Information Access Using Speech and Gestures Norbert Reithinger norbert. reithinger@dfki. de 22. Mai 2001 1. Projektlenkungssitzung, Sony, Stuttgart-Wangen

Natural Access to Digital Resources • Speech dialog systems already-to-market • Next step: multimodal interfaces • Add modalities beyond speech – Enable interactions using – Gestures – Pointing – Haptics. . . 17. 02. 2004 Chennai 3

Advantages of Multimodality for the Users • Natural use of preferred modalities • Mutual disambiguation using multiple modalities under adverse conditions • Environment may elicit preferred interaction modality • Product realization e. g. as multimodal information kiosk • Example: Smart. Kom 17. 02. 2004 Chennai 4

Smart. Kom: Intuitive Multimodal Interaction Project Budget: Project Duration: € 25. 5 million (partly funded by the German Ministry of Education and Research - BMBF) 4 years (September 1999 – September 2003) The Smart. Kom Consortium: Main Contractor DFKI Saarbrücken Uinv. Of Munich Media. Interface Berkeley Dresden Saarbrücken Heidelberg Univ. of Stuttgart Munich Univ. of Erlangen Aachen 17. 02. 2004 Chennai European Media Lab Ulm Stuttgart 5

Smart. Kom: The Three Scenarios Application Layer Public: Mobile: Car and Pedestrian Navigation Multimodal Dialogue Backbone Home: Smart. Kom-Mobile: Mobile Travel Companion that helps with navigation Cinema, Phone, Fax, Mail, Biometrics Consumer Electronics EPG Smart. Kom-Home: Infotainment Companion that helps to select media content 17. 02. 2004 Chennai Smart. Kom-Public: Communication Companion that helps to keep in touch and to get information 6

Smart. Kom’s Multimodal Input and Output Devices Multimodal Control of TV-Set Infrared Camera for Gestural Input, Tilting CCD Camera for Scanning, Video Projector Microphone Multimodal Control of VCR/DVD Player 3 dual Xeon 2. 8 Ghz processors with 1. 5 GB main memory 17. 02. 2004 Chennai Camera for Facial Analysis Projection Surface Speakers for Speech Output 7

Smart. Kom`s SDDP Interaction Metaphor SDDP = Situated Delegation-oriented Dialogue Paradigm IT Services User Personalized Interaction specifies goal Agent Service 1 delegates task cooperate on problems Service 2 . . . asks user presents results 17. 02. 2004 Chennai Service N 8

Smart. Kom Understands Multimodal Input Please reserve here. 17. 02. 2004 Chennai 9

Smart. Kom: 14 applications 52 functionalities (Source: Reithinger et al: Smart. Kom - Adaptive and Flexible Multimodal Access to Multiple Applications. In ICMI ’ 03) 17. 02. 2004 Chennai 10

Generic Technologies Used • • • 17. 02. 2004 Chennai Speech and gesture recognition Language and gesture understanding Modality fusion Dialog processing Information extraction/retrieval, e. g. from Internet sources Biometry Presentation planning Answer generation Speech synthesis Interactive presentation 11

Interactive Biometric Authentication by Hand Contour Recognition Please place your hand with spread fingers on the marked area. 17. 02. 2004 Chennai 12

Adaptation to Another Language: Smart. Kom Mobile English Smart. Kom’s modular architecture encapsulates language specific knowledge in few language processing modules Smart. Kom system overview: Module was … Modified Not modified Not used Minor modifications 17. 02. 2004 Chennai 13

Conclusion • Multimodal interaction enables natural access to digital resources • Advantageous for many users • Smart. Kom realizes an exemplary multimodal information kiosk • Adaptation to different languages relatively easy 17. 02. 2004 Chennai 14

Thank you very much for your attention! • Please find more information at http: //www. smartkom. org • Other multimodal projects with participation of DFKI – MIAMM (EU): Multidimensional Information Access using Multiple Modalities: http: //www. miamm. org – COMIC (EU): COnversational Multimodal Interaction with Computers: http: //www. hcrc. ed. ac. uk/comic – Virtual. Human (BMBF): Virtual agents for education http: //www. virtual-human. org 17. 02. 2004 Chennai 15