Multimodal Design Technologies Sidney Fels THE UNIVERSITY OF

Multimodal Design & Technologies Sidney Fels THE UNIVERSITY OF BRITISH COLUMBIA

Overview • • Introduction Human I/O: Interface Examples Bringing Modalities Together Summary Fels: Multimodal Design and Technologies 2

Introduction • Two messages 1. knowledge of modalities motivates design 2. modalities can complement each other 1. challenges and pitfalls 1. Applications in virtual environments – communication of experience – intimacy and embodiment Fels: Multimodal Design and Technologies 3

Human Information Processing • Input • 5 usual senses – vision, hearing, touch, taste, smell • position and motion sensing systems • Output • intentional – neuromuscular, movable, verbal • non-intentional / biopotentials – galvanic skin response (GSR), heart rate, brain, muscle activation • Cognitive – memory, decision making, tracking, learning Fels: Multimodal Design and Technologies 4

Visual Display Technologies • examples of designs exploiting human capabilities – Virtual Retinal Display (VRD) – Cubby – CAVE Automatic Virtual Environment Fels: Multimodal Design and Technologies 5

Virtual Retinal Delay (VRD) From HIT lab, U. Of Washington (Furness et al. , 1991) Microvision continuing work Fels: Multimodal Design and Technologies 6

Cubby (Djajadiningrat and Gribnau, 2000) Fels: Multimodal Design and Technologies 7

CAVE Illustration (U. of Illinois, 1992) Fels: Multimodal Design and Technologies 8

Haptics Display Technologies • examples of touch and force feedback – Pantograph – Phantom – Cyber. Force Fels: Multimodal Design and Technologies 9

Pantograph (Hayward, Mc. Gill) • 2 dof translation - 10 cm X 10 cm • FF/B mouse • acceleration best for shock and hard contacts Fels: Multimodal Design and Technologies 10

Phantom (Massey, Sens. Able Devices) • thimble on finger – single point force feedback – 3 degrees of freedom Fels: Multimodal Design and Technologies 11

Cyber. Force (Kramer, Immersion Corp. ) • Exoskelton – arm forces • Tendons – grasp forces Fels: Multimodal Design and Technologies 12

tricks: visual / aural illusions examples of metacognitive gap • visual: never show the point penetrating the surface, even if it is actual: • aural: play a crisp contact sound on contact this makes the surface appear stiffer/harder (very robust effect) displayed: BUT: if time offset too great, opposite effect Fels: Multimodal Design and Technologies 13

Other Haptic Devices • Tactile feedback • Temperature • Sheer forces Fels: Multimodal Design and Technologies 14

challenges for haptic interaction design • continuous vs. discrete manual control • displaying interaction potential • embedding haptic interfaces • ensuring tight sensory coupling • interacting with other modalities Fels: Multimodal Design and Technologies 15

Olfactory Devices • How do we smell? – different theories: • i. e. chemical, infrared absorption – different perceptual mappings: • • smell prism four odors: fragrant, acrid, burnt and caprylic domain specific: wine, beer, etc. many, many others. . . – Acuity is great - 10, 000 times more sensitive than taste – negative adaptation occurs • you get used to the smell Fels: Multimodal Design and Technologies 16

Olfactory Interfaces • Smell camera – Susnick & Raknow, Nature, 2000, 406, 710714. • Smell display – liquids, gels, microencapsulation • Application Research: – Joe Kaye: • in. Stink, Dollars & Scents, Scent Reminder, and more. . . – Fels, Gauthier, Smith: Interactive Yoga system • Smell Display Research and Products: – Digi. Scents (bankrupt), Tri. Senx (bankrupt) – DIVEpak (Southwest Research Institute, 1993) – see notes Fels: Multimodal Design and Technologies 17

Olfaction Display Challenges • smell synthesis – many smells vs. mixing base smells • control breathing space – – – sealed room with air filtration air control in front of and behind user sealed pod tethered mask tubes into an HMD from pack built into HMD Fels: Multimodal Design and Technologies 18

Taste • taste buds for – sensations of sour, salty, bitter, sweet and umami – receptors not completely resolved • umami receptor (Zucker et al. , 2002) – extremely complex and poorly understood • interacts with olfaction • display = food? Fels: Multimodal Design and Technologies 19

Biopotentials: Examples • Galvanic Skin Response (GSR) – Affective Computing, i. e. Galvactinator, Scheirer and Picard • Heart Rate (HR) – 2 Hearts Musical System, (Mc. Caig and Fels, 2002) • Brain activity (EEG) – Brain Computer Interface (BCI) • Muscle activity (EMG) Fels: Multimodal Design and Technologies 20

Summary of Human I/O • Multitude of input/output systems – all active at once • I/O mechanisms usually depend upon – cognitive context – emotional contexts • All these systems available for applications – complement each other • Multimodal design looks at: – integration – substitution – complement Fels: Multimodal Design and Technologies 21

Multimodal Design: Bringing it Together • User centred and non-user centred • Intimacy and Embodiment – automatic behaviour – sources of aesthetics Fels: Multimodal Design and Technologies 22

Intimacy and Embodiment • Want interfaces that feel “good” to use • Humans and machines intimately linked – degree of intimacy supported may determine success • Types of relationships: – human to human – human to machine Fels: Multimodal Design and Technologies 23

Intimacy • Intimacy is a measure of subjective match between the behaviour of an object and the control of that object. – extension of “control intimacy” from electronic musical instruments analysis (Moore, 1997) • High intimacy implies: – object feels like an extension of self – satisfaction derives from interacting with object – emotional expression flows • requires cognitive effort to prevent Fels: Multimodal Design and Technologies 24

Intimacy, Embodiment and Aesthetics Case 1: Object disembodied from Self self Aesthetic object Result Case 2: Self embodies Object self object Control Case 3: Self disembodied from Object object self Reflection Case 4: Object embodies Self object self Fels: Multimodal Design and Technologies Belonging 25

Intimacy and Embodiment Design Examples • Many excellent examples out there of interesting designs – Many at Siggraph: • Jam-o-Drum (Blaine et al. ), Wooden Mirror (Rozin), etc. – Tangible Bits work (Ishii et al. ) – Ubicomp (Weiser and more) – Wearable computing (Mann and others) – Art, Entertainment – Virtual reality Fels: Multimodal Design and Technologies 26

Application: Iamascope (Fels and Mase, 1997) Fels: Multimodal Design and Technologies 27

Iamascope Overview Video Fels: Multimodal Design and Technologies 28

Flow. Field: Semantics of Caress (Chen, Fels and Min, 2003) • Whole hand interaction in VR – use multi-touch sensitive pad – immersive display • Idea: – allow users direct manipulation of fluid • use particle simulation for fluid – aesthetics was important • hand manipulation on hard surface mapped to obstructions in the flow field Fels: Multimodal Design and Technologies 29

Flow. Field: Semantics of Caress Fels: Multimodal Design and Technologies 30

Summary • Multimodal interfaces need to consider – human information processing – matching interface to task • use complementary modes where appropriate – intimacy and embodiment • Plenty of research opportunities Fels: Multimodal Design and Technologies 31

Fels: Multimodal Design and Technologies 32

Thank-you! Please fill out your comments forms. www. icics. ubc. ca/hci/multimodal/ THE UNIVERSITY OF BRITISH COLUMBIA