Technische Universitt Mnchen Acoustic Interfaces for Improving Situational

Technische Universität München Acoustic Interfaces for Improving Situational Awareness Monika Srdic 01. 04. 2020 Final: Bachelor Informatics – Games Engineering Supervisor: Prof. Gudrun Klinker, Ph. D. Advisor: Daniel Dyrda, M. Sc.

Fachgebiet Augmented Reality Technische Universität München Introduction / Motivation • User Interfaces are essential in human–computer interaction and in IT • User Interface: “the way a computer gives information to a user or receives instructions from a user“ Ø Providing feedback to the user Ø Enhancing situational awareness Monika Srdic: Acoustic User Interfaces for Improving Situational Awareness 2

Fachgebiet Augmented Reality Technische Universität München Introduction / Motivation User / Input Feedback Semantics + Context Different Feedback forms (animation, sound, image/graphic, …) Monika Srdic: Acoustic User Interfaces for Improving Situational Awareness 3

Fachgebiet Augmented Reality Technische Universität München • Situational awareness: system state known in respect to time and space give proper (auditive) feedback in respect to the situation Monika Srdic: Acoustic User Interfaces for Improving Situational Awareness 4

Fachgebiet Augmented Reality Technische Universität München Ø Imagine you are using a graphic editor. How can auditory feedback improve the situational awareness? • Focus on state of the system (interactable vs not interactable, . . . ) Photoshop, unity, rendering, … Monika Srdic: Acoustic User Interfaces for Improving Situational Awareness 5

Fachgebiet Augmented Reality Technische Universität München Problem Description: Issues with UI feedback • Information overload • Keeping optimal experience • Keep overview of contextual situation/ system state • Acceptance of different feedback forms under different circumstances / situations • Functionality over design -> principles / usability guidelines (Nielsen Norman Group, . . ) Monika Srdic: Acoustic User Interfaces for Improving Situational Awareness 6

Fachgebiet Augmented Reality Technische Universität München Existing Solutions / Related Work 1. Assisting the Visually Impaired: Obstacle Detection and Warning System by Acoustic Feedback [1] Ø Focus on spatial awareness 2. “Writing with Music”: Exploring the Use of Auditory Feedback in Gesture Interfaces [2] Ø Focus on the design and effect of auditory feedback 3. Physically Based Sound Effects for Interactive Simulation and Animation [3] Ø Focus on immersion and mimicking the real world Monika Srdic: Acoustic User Interfaces for Improving Situational awareness 7

Fachgebiet Augmented Reality Technische Universität München Goals of this Thesis • Overview of Important Principles and guidelines in designing auditory feedback for future research • Overview of different designs of accoustic UI for improving situational awareness • Analyzing important aspects of situational awareness in context of different User feedback forms • Discussing different interaction methods in respect to auditory feedback Monika Srdic: Accoustic User Interfaces for improving situational awareness 8

Fachgebiet Augmented Reality Technische Universität München Proposed Work / Approach • Ø Ø Ø Analysing the three important topics UI Situational Awareness Audio • Designing auditory feedback Ø Discussion and analysis of possibilities Monika Srdic: Accoustic User Interfaces for improving situational awareness 9

Fachgebiet Augmented Reality Technische Universität München User Interfaces • Design principles: Ø Aesthetics: minimalism vs realism Ø functionality/ effect: flexibility, consistency, invoke high user engagement and usability, intuitive, principle of least surprise Ø Trends Monika Srdic: Accoustic User Interfaces for improving situational awareness 10

Fachgebiet Augmented Reality • Ø Ø Ø Technische Universität München UI feedback: Multimodality Occurence of interactions, urgency, time, system state Start, end, duration • Current Situation of GUI and UI feedback in Editors: Ø User centered, mostly graphical and visual feedback Monika Srdic: Accoustic User Interfaces for improving situational awareness 11

Fachgebiet Augmented Reality Technische Universität München Situational Awareness • Creation of mental model: conceptual map of input and already set up associations / feelings, . . • Feedback based on that conceptual mapping • Shared vs. Individual Situational Awareness • Clear, adaptive, accurate, mirror reality • + flexible, interoperable • - scalable only to limit, uncertainty Monika Srdic: Accoustic User Interfaces for improving situational awareness 12

Fachgebiet Augmented Reality Technische Universität München • Situational awareness and spatial awareness both are mental models based on different focus • In graphical editors: awareness about system behaviour and system state, no related work • Feedback needs to be appropriate to situation and help him get clarity about his role in it and about the system Monika Srdic: Accoustic User Interfaces for improving situational awareness 13

Fachgebiet Augmented Reality Technische Universität München Audio • = any sound or noise which the human ear is capable of hearing [15] • Hearing gives feedback about environment in real life • Properties of sound: Ø Tone , pitch, intervals, duration, rhythm, chord, melody, dynamics, sound combinations, silence and pauses, foreground and background sound, instruments, sound system and quality, already existing sound effects, voice or singing, … Monika Srdic: Accoustic User Interfaces for improving situational awareness 14

Fachgebiet Augmented Reality • • • Technische Universität München + commonly liked and accepted + high immerision + works on deep levels of consciousness + continuous / richness in sound + enhances learning + easy sound production • - danger of information overload or loss • - sound perception is individual • - can lower performance Monika Srdic: Acoustic User Interfaces for Improving Situational Awareness 15

Fachgebiet Augmented Reality Technische Universität München • Acoustic feedback is not used very much in graphical editors so far • Ø Ø Ø Key factors in audio design: Main purpose of sound? What emotions or user behaviour should be triggered? How frequent is interaction that causes the feedback? What is the context and the situation? What type of sound can be used? What technology is useable? Monika Srdic: Acoustic User Interfaces for Improving Situational Awareness 16

Fachgebiet Augmented Reality Technische Universität München • For graphical editor nothing too complex since it is work environment, rather distracting if too much • Audio feedback is fufilling as soon as it doesnt become noise to the user Monika Srdic: Accoustic User Interfaces for improving situational awareness 17

Fachgebiet Augmented Reality Technische Universität München Specification of design principles Abstraction of a Process Monika Srdic: Accoustic User Interfaces for improving situational awareness 18

Fachgebiet Augmented Reality Technische Universität München Parallel Usage – successful start Monika Srdic: Accoustic User Interfaces for improving situational awareness 19

Fachgebiet Augmented Reality Technische Universität München Distinct end of action Monika Srdic: Accoustic User Interfaces for improving situational awareness 20

Fachgebiet Augmented Reality Technische Universität München Input Validation Monika Srdic: Accoustic User Interfaces for improving situational awareness 21

Fachgebiet Augmented Reality Technische Universität München Validation on every frame update Monika Srdic: Accoustic User Interfaces for improving situational awareness 22

Fachgebiet Augmented Reality Technische Universität München Action Focus Monika Srdic: Accoustic User Interfaces for improving situational awareness 23

Fachgebiet Augmented Reality Technische Universität München Performance update Monika Srdic: Accoustic User Interfaces for improving situational awareness 24

Fachgebiet Augmented Reality Technische Universität München Critical Research Issues • The auditory UI is not very suitable or the designs don‘t add to higher usability • The solutions are too complex – information overflow • Too much related work – keeping the common theme due to much information regarding this topic Ø Should be tested in future user study Monika Srdic: Accoustic User Interfaces for improving situational awareness 25

Fachgebiet Augmented Reality Technische Universität München Discussion / Suggested Future Work • User study for different use cases • Focusing on specific editors • Collecting more design guidelines for accoustic feedback Monika Srdic: Accoustic User Interfaces for improving situational awareness 26

Fachgebiet Augmented Reality Technische Universität München Conclusion • Success in generalizing common situations in graphical editors and creating overview according to the situation/ system state • Success in designing auditive feedback according to the situation/ system state • Only one to three tones maximum, not much decorating • Ustudy needed to confirm theory Ø Detailed design of acoustic feedback with in depth knowledge about sound, user interfaces, user feedback and the environment improves situational awareness. Monika Srdic: Accoustic User Interfaces for improving situational awareness 27

Fachgebiet Augmented Reality Technische Universität München List of References Monika Srdic: Accoustic User Interfaces for improving situational awareness 28

Fachgebiet Augmented Reality Technische Universität München List of References Monika Srdic: Accoustic User Interfaces for improving situational awareness 29