Multimodal Interaction Modalities vs Media Modalities are ways

  • Slides: 30
Download presentation
Multimodal Interaction

Multimodal Interaction

Modalities vs Media ¢ Modalities are ways of encoding information l e. g. graphics

Modalities vs Media ¢ Modalities are ways of encoding information l e. g. graphics l ¢ Media are instantiations of modalities l l e. g. a particular image

How Do Multimodal Systems Differ? Domain/application ¢ Available media ¢ Modeling of context/environment ¢

How Do Multimodal Systems Differ? Domain/application ¢ Available media ¢ Modeling of context/environment ¢ Modeling of user ¢ Focus of research ¢

Example Multimodal Systems ¢ Not speech-centric l MIT paintbrush, soundbrush • http: //www. youtube.

Example Multimodal Systems ¢ Not speech-centric l MIT paintbrush, soundbrush • http: //www. youtube. com/watch? v=04 v_v 1 gny. O 8 • http: //www. youtube. com/watch? v=i. Zbe 3 t 8 YSf 4 • http: //www. youtube. com/watch? v=18 RY 8 Jgid 20 l Wearables • http: //www. gatech. edu/innovations/wearable/

Example Multimodal Systems ¢ Speech-centric MSOIP l COMIC l Smart. Kom l

Example Multimodal Systems ¢ Speech-centric MSOIP l COMIC l Smart. Kom l

MSOIP Keywords Multimodal mobile dialog ¢ Integration of speech and pen input ¢ User

MSOIP Keywords Multimodal mobile dialog ¢ Integration of speech and pen input ¢ User modeling for presentations ¢ Johnston et al. 2001

MATCH Video http: //www. research. att. com/~johnston/ Scroll down to the bottom of the

MATCH Video http: //www. research. att. com/~johnston/ Scroll down to the bottom of the page

About MATCH What input modalities? ¢ What output modalities? ¢ What application(s)? ¢ What

About MATCH What input modalities? ¢ What output modalities? ¢ What application(s)? ¢ What aspects of context? ¢

COMIC Keywords Ambient intelligence ¢ HHI/HCI research ¢ l Collaborative problem solving User modeling

COMIC Keywords Ambient intelligence ¢ HHI/HCI research ¢ l Collaborative problem solving User modeling ¢ Avatar ¢ Alexandersson et al. 2004

COMIC Video http: //www. hcrc. ed. ac. uk/comic/demos/facial-animation/

COMIC Video http: //www. hcrc. ed. ac. uk/comic/demos/facial-animation/

COMIC Video http: //www. hcrc. ed. ac. uk/comic/demos/slot/

COMIC Video http: //www. hcrc. ed. ac. uk/comic/demos/slot/

About COMIC What input modalities? ¢ What output modalities? ¢ What applications? ¢ What

About COMIC What input modalities? ¢ What output modalities? ¢ What applications? ¢ What aspects of context? ¢

Smart. Kom Keywords ¢ Multimodal dialog across l l l ¢ ¢ applications devices

Smart. Kom Keywords ¢ Multimodal dialog across l l l ¢ ¢ applications devices and situations Avatar Situation aware Alexandersson et al. , Reithinger et al. 2003

Smart. Kom Video http: //www. smartkom. org/start_en. html I showed the SK-Mobile one, but

Smart. Kom Video http: //www. smartkom. org/start_en. html I showed the SK-Mobile one, but the other one is also interesting.

About Smart. Kom What input modalities? ¢ What output modalities? ¢ What applications? ¢

About Smart. Kom What input modalities? ¢ What output modalities? ¢ What applications? ¢ What aspects of context? ¢

Parts of a Multimodal System Text In Gesture In Speech Out Speech In Interpreter

Parts of a Multimodal System Text In Gesture In Speech Out Speech In Interpreter Present Out Generator Dialog Manager Knowledge Base Text Out

HCI and Multimodal Systems Input integration/fusion ¢ Representations ¢ Effective help ¢ Quality presentations

HCI and Multimodal Systems Input integration/fusion ¢ Representations ¢ Effective help ¢ Quality presentations ¢ Managing context ¢ Understanding the user ¢

Different Uses of Modalities ¢ Concurrent or sequential Redundant or ¢ Complementary or ¢

Different Uses of Modalities ¢ Concurrent or sequential Redundant or ¢ Complementary or ¢ Contradicting ¢

Input Integration/Fusion ¢ Key elements: Time l Multiple uses of some modalities l Error

Input Integration/Fusion ¢ Key elements: Time l Multiple uses of some modalities l Error rates l ¢ Typical approach is to map straight to semantics if possible

Representation ¢ Increasing use of XML-based languages (SMIL, EMMA) l ¢ But these don’t

Representation ¢ Increasing use of XML-based languages (SMIL, EMMA) l ¢ But these don’t solve the semantic problems Keep ‘backbone’ knowledge separate from ‘peripheral’ information (Alexandersson et al. )

Effective Help ¢ How do each of the systems provide the user with: Explicit

Effective Help ¢ How do each of the systems provide the user with: Explicit help? l Implicit help? l

Quality Presentations ¢ Talking heads Advantages l Disadvantages l Informative presentations are key ¢

Quality Presentations ¢ Talking heads Advantages l Disadvantages l Informative presentations are key ¢ User modeling/adaptive presentations are a bonus ¢ These systems go beyond scripts ¢

Managing Context ¢ What kinds of context are there in a mobile multimodal interaction?

Managing Context ¢ What kinds of context are there in a mobile multimodal interaction?

Understanding the User What kinds of information can we gather about users in general?

Understanding the User What kinds of information can we gather about users in general? ¢ About one user in particular? ¢ How can we use this information? ¢

Commercial Multimodal Systems ¢ Most are for research l Military • Training and battlefield

Commercial Multimodal Systems ¢ Most are for research l Military • Training and battlefield l Education • Tutoring systems ¢ Commercial ones include: l l Wii: http: //www. youtube. com/watch? v=n 4 n. ZVAE eit. U Microsoft surface: http: //www. youtube. com/watch? v=r. P 5 y 7 yp 0 6 n 0

Trade. Offs ¢ You get: More intuitive technology l More information, more easily l

Trade. Offs ¢ You get: More intuitive technology l More information, more easily l Less (dumb stuff) for you to do l ¢ You trade: Privacy l Control l

Towards the Future ¢ Design Multimodal systems in virtual worlds, or crossing over from

Towards the Future ¢ Design Multimodal systems in virtual worlds, or crossing over from virtual to real worlds l Ambient multimodal interaction l ¢ Implementation Mashups – user controlled l Pervasive multimedia l

Towards the Future http: //www. youtube. com/watch? v=FM Jw. URqp. FWs ¢ http: //www.

Towards the Future http: //www. youtube. com/watch? v=FM Jw. URqp. FWs ¢ http: //www. programmableweb. com/m ashups ¢

Sci. Fi? ¢ ¢ Lathe of Heaven by Ursula Le. Guin Summa Technologiae by

Sci. Fi? ¢ ¢ Lathe of Heaven by Ursula Le. Guin Summa Technologiae by Stanislaw Lem Fast Times at Fairmont High by Vernor Vinge The Human Machine Merger, talk by Raymond Kurzweil (at http: //www. kurzweilai. net/meme/frame. html? main=memelist. html? m=6%23581)

Additional Info http: //search. techrepublic. com/se arch/multimodal+system. html ¢ http: //www. w 3. org/2002/mmi/

Additional Info http: //search. techrepublic. com/se arch/multimodal+system. html ¢ http: //www. w 3. org/2002/mmi/ ¢

1 Multimodal Transport Multimodal Transport 2 MTOGA Multimodal1 Multimodal Transport Multimodal Transport 2 MTOGA Multimodal
Multimodal Transport What is Multimodal transport Multimodal transportMultimodal Transport What is Multimodal transport Multimodal transport
Modalities The ways audiences reactions to modalities shiftModalities The ways audiences reactions to modalities shift
Human Computer Interaction The Interaction The Interaction interactionHuman Computer Interaction The Interaction The Interaction interaction
Mechanical Modalities Therapeutic Modalities in Athletic Rehabilitation MechanicalMechanical Modalities Therapeutic Modalities in Athletic Rehabilitation Mechanical
Athletic Training Modalities 2 Modalities Thermotherapy Types ofAthletic Training Modalities 2 Modalities Thermotherapy Types of
Introduction 22 Comparison of Modalities Review Modalities XrayIntroduction 22 Comparison of Modalities Review Modalities Xray
Chapter 12 Therapeutic Physical Modalities Therapeutic Modalities TheseChapter 12 Therapeutic Physical Modalities Therapeutic Modalities These
Modalities are Fun Sports Med 2 Therapeutic ModalitiesModalities are Fun Sports Med 2 Therapeutic Modalities
Chapter 15 Therapeutic Modalities Therapeutic Modalities n InChapter 15 Therapeutic Modalities Therapeutic Modalities n In
Athletic Training Modalities Modalities Objectives Legal ramifications ofAthletic Training Modalities Modalities Objectives Legal ramifications of
Chapter 15 Therapeutic Modalities Therapeutic modalities can beChapter 15 Therapeutic Modalities Therapeutic modalities can be
Therapeutic Modalities Introduction Therapeutic modalities create an optimalTherapeutic Modalities Introduction Therapeutic modalities create an optimal
Chapter 17 Therapeutic Modalities Therapeutic Modalities Various methodsChapter 17 Therapeutic Modalities Therapeutic Modalities Various methods
Chapter 12 Therapeutic Physical Modalities Therapeutic Modalities TheseChapter 12 Therapeutic Physical Modalities Therapeutic Modalities These
Introduction to Modalities Cryotherapy Introduction o Modalities areIntroduction to Modalities Cryotherapy Introduction o Modalities are
Chem Multimodal Analysis of current status of multimodalChem Multimodal Analysis of current status of multimodal
Multimodal Interface IT 20013325 whiteship 2000gmail com MultimodalMultimodal Interface IT 20013325 whiteship 2000gmail com Multimodal
Multimodal Texts Digital texts for literacy Multimodal TextsMultimodal Texts Digital texts for literacy Multimodal Texts
UNIVERSAL DESIGN2 Multimodal vs Multimedia Multimodal systems areUNIVERSAL DESIGN2 Multimodal vs Multimedia Multimodal systems are
THE MULTIMODAL TEXT What are multimodal texts ATHE MULTIMODAL TEXT What are multimodal texts A
THE MULTIMODAL TEXT What are multimodal texts ATHE MULTIMODAL TEXT What are multimodal texts A
Textul multimodal Banda desenat Textul multimodal combin douTextul multimodal Banda desenat Textul multimodal combin dou
Multimodal Interests Incremental understanding and feedback Mobile multimodalMultimodal Interests Incremental understanding and feedback Mobile multimodal