Interaction Techniques for Common Tasks in Immersive Virtual

Interaction Techniques for Common Tasks in Immersive Virtual Environments Design, Evaluation, and Application Doug A. Bowman April 27, 1998

Vision Introduction Methodology Travel Selection/Manipulation Application Remaining Work Immersive VEs for productivity Complex applications for real work Example: immersive modeling and design Doug Bowman - VE Interaction Techniques 2

Definitions Introduction Methodology Travel Selection/Manipulation Application Remaining Work Interaction Technique (IT): Method used to complete a task via a human-computer interface (hardware & software) Immersive VE: A real-time 3 D synthetic environment that appears to surround the user in space √ HMD with head tracking, CAVE X “Fishtank VR”, MUDs, Multimedia apps Doug Bowman - VE Interaction Techniques 3

A Brief History of VEs Introduction Methodology Travel Selection/Manipulation Application Remaining Work 1968: Sutherland’s Ultimate Display Hardware advances – displays – trackers – input devices – haptics – 3 D graphics – 3 D audio Software advances – view culling – level of detail – VE toolkits – collision detection Doug Bowman - VE Interaction Techniques 4

VE Applications Introduction Methodology Travel Selection/Manipulation Application Remaining Work In Use: – architectural walkthrough – phobia treatment – games (e. g. 1 st person shooter) Proposed: – information visualization and retrieval – modeling and design – constructivist education Doug Bowman - VE Interaction Techniques 5

Interaction: the Distinguishing Factor Introduction Methodology Travel Selection/Manipulation Application Remaining Work Current applications – may involve movement through VE – may involve shooting or pointing Proposed applications – require 3 D navigation and selection – require 6 DOF manipulation (object placement) – require large command spaces Doug Bowman - VE Interaction Techniques 6

How to improve VE Interaction Introduction Methodology Travel Selection/Manipulation Application Remaining Work better design of techniques systematic evaluation (formative and summative) in the context of applications and requirements Doug Bowman - VE Interaction Techniques 7

Universal Tasks: Travel Introduction Methodology Travel Selection/Manipulation Application Remaining Work Viewpoint Motion Control: The user’s interactive control of the position and orientation of his viewpoint Wayfinding: Cognitive process of determining a route, using landmarks, maps, etc. Navigation: VMC + Wayfinding Doug Bowman - VE Interaction Techniques 8

Universal Tasks: Selection & Manipulation Introduction Methodology Travel Selection/Manipulation Application Remaining Work Selection: Specification of one or more objects from a set – as the object of a command – to begin manipulation Manipulation: Specification of the position, orientation, and/or scale of an object Doug Bowman - VE Interaction Techniques 9

Why not natural interaction? Introduction Methodology Travel Selection/Manipulation Application Remaining Work Term “VR” implies replication of real world Why not use well-developed human skills to accomplish tasks in VEs? – travel: walking or driving – selection & manipulation: grasp and place These mappings are intuitive, but too limited Doug Bowman - VE Interaction Techniques 10

Interaction Techniques and Input Devices Introduction Methodology Travel Selection/Manipulation Application Remaining Work input devices are only the hardware component of an IT input device does not determine IT many ITs can be implemented with a single input device we will not design or evaluate devices we will design and evaluate ITs for common VE input devices Doug Bowman - VE Interaction Techniques 11

Problem Statement: I will. . . Introduction Methodology Travel Selection/Manipulation Application Remaining Work analyze universal tasks and create taxonomies of techniques design new techniques based on these formal frameworks design, implement, and conduct formal evaluations of IT performance apply the results to a complex and useful VE application Doug Bowman - VE Interaction Techniques 12

Design and Evaluation Methodology Introduction Methodology Travel Selection/Manipulation Application Remaining Work Taxonomization and Categorization Guided Design Performance Measures Range of Evaluation Methods Testbed Evaluation Doug Bowman - VE Interaction Techniques 13

Taxonomization and Categorization Task Subtask Introduction Methodology Travel Selection/Manipulation Application Remaining Work Task analysis Consider techniques for low-level subtasks Promotes deeper understanding of task Framework for design Framework for evaluation Technique Component Doug Bowman - VE Interaction Techniques 14

Guided Design 1 2 3 Task Subtask 4 Introduction Methodology Travel Selection/Manipulation Application Remaining Work Design new techniques based on taxonomy, not simply intuition Choose a component for each low-level subtask Easy to see holes in design space Technique Component Doug Bowman - VE Interaction Techniques 15

Evaluation Methods Introduction Methodology Travel Selection/Manipulation Application Remaining Work Range of performance metrics (quantitative and qualitative; productivity and user-centric) Range of methods (user studies, usability evaluation, formal experiments) Consideration of outside factors (characteristics of task, environment, user, system that might affect performance) Doug Bowman - VE Interaction Techniques 16

Testbed Evaluation Introduction Methodology Travel Selection/Manipulation Application Remaining Work testbed: representative set of tasks and environments evaluate techniques for overall performance in a wide range of situations vary technique components and outside factors measure several performance variables generalizable and replicable Doug Bowman - VE Interaction Techniques 17

Summary of Methodology Initial Evaluation and Design Taxonomies Perf. Metrics speed accuracy comfort. . . Doug Bowman - VE Interaction Techniques Introduction Methodology Travel Selection/Manipulation Application Remaining Work Outside Factors environment density user’s reach task difficulty. . . 18

Summary of Methodology Initial Evaluation and Design Taxonomies Perf. Metrics speed accuracy comfort. . . Introduction Methodology Travel Selection/Manipulation Application Remaining Work Outside Factors environment density user’s reach task difficulty. . . requirements Applications Doug Bowman - VE Interaction Techniques 19

Summary of Methodology Initial Evaluation and Design Taxonomies Perf. Metrics speed accuracy comfort. . . TESTBED EVALUATION Introduction Methodology Travel Selection/Manipulation Application Remaining Work Outside Factors environment density user’s reach task difficulty. . . requirements Applications Doug Bowman - VE Interaction Techniques 20

Summary of Methodology Initial Evaluation and Design Taxonomies Perf. Metrics speed accuracy comfort. . . TESTBED EVALUATION Performance Measurements/ Models Doug Bowman - VE Interaction Techniques choice of techniques Introduction Methodology Travel Selection/Manipulation Application Remaining Work Outside Factors environment density user’s reach task difficulty. . . requirements Applications 21

Informal Evaluation Introduction Methodology Travel Selection/Manipulation Application Remaining Work based on observations – default gaze-directed steering – lack of published work based on our own applications – Conceptual Design Space – Virtual GIS Doug Bowman - VE Interaction Techniques 22

Initial Taxonomy Introduction Methodology Travel Selection/Manipulation Application Remaining Work Task: Move from the current location to the desired location Direction/Target Selection Viewpoint Motion Control Velocity/Acceleration Selection Conditions of Input Doug Bowman - VE Interaction Techniques gaze-directed pointing physical props gesture slow in, slow out physical props start/stop buttons automatic start/stop constant movement 23

Performance Measures Introduction Methodology Travel Selection/Manipulation Application Remaining Work Quantitative (e. g. speed, accuracy) Qualitative (e. g. presence) User-Centric (e. g. ease of use, comfort) IT Doug Bowman - VE Interaction Techniques Quality Factors -speed -accuracy -cognitive load -presence -spatial awareness -. . . Apps 24

Simple Experiments (Bowman, Koller, and Hodges, VRAIS ‘ 97) Introduction Methodology Travel Selection/Manipulation Application Remaining Work Absolute Motion – no difference between gaze and pointing Relative Motion – pointing superior to gaze Spatial Awareness – teleportation causes disorientation – any continuous motion does not Doug Bowman - VE Interaction Techniques 25

Expanded Framework Absolute vs. Relative Motion – same techniques, different results – highlights need to consider outside factors Consider task, user, system, and environment characteristics Doug Bowman - VE Interaction Techniques Performance Introduction Methodology Travel Selection/Manipulation Application Remaining Work absolute relative Task gaze-directed steering pointing 26

Complex Experiment Introduction Methodology Travel Selection/Manipulation Application Remaining Work Does travel IT affect cognitive load? Task: gather as much info as possible Variables: » IT: gaze, pointing, torso » Environment: 1 -, 2 -, or 3 dimensional » System: collision detection Doug Bowman - VE Interaction Techniques 27

Guided Design Introduction Methodology Travel Selection/Manipulation Application Remaining Work Taxonomy: “tour” technique – environmental target selection – gesture-based velocity selection – explicit or automatic stop inputs Intuition: travel based on manipulation – cross-task technique – still fits in taxonomy Doug Bowman - VE Interaction Techniques 28

Final Framework and Testbed Introduction Methodology Travel Selection/Manipulation Application Remaining Work Rework taxonomy to be more general – task analysis: 2 basic position-setting methods are specifying destination, specifying trajectory – distinction allows better fitting of techniques VMC Testbed – still in design stage – based on evaluation framework Doug Bowman - VE Interaction Techniques 29

Initial Taxonomy Introduction Methodology Travel Selection/Manipulation Application Remaining Work Based on metaphor, not task Arm-extension metaphor – touch and place object with virtual hand – hand may extend beyond normal range Ray-casting metaphor – point at object to select – manipulate by attaching to virtual light ray Doug Bowman - VE Interaction Techniques 30

Informal Evaluation (Bowman and Hodges, I 3 DG ‘ 97) Introduction Methodology Travel Selection/Manipulation Application Remaining Work Studied six techniques (4 AE, 2 RC) Simple user study (comments, observations) Eleven subjects used techniques to place furniture in a room Results – AE excels at manip. , RC better at selection – selection & manipulation should be separated Doug Bowman - VE Interaction Techniques 31

HOMER Techniques Hand-Centered Object Manipulation Extending Ray. Casting Hybrid technique Select: ray-casting Manipulate: v. hand Doug Bowman - VE Interaction Techniques Introduction Methodology Travel Selection/Manipulation Application Remaining Work Time 32

HOMER Techniques Hand-Centered Object Manipulation Extending Ray. Casting Hybrid technique Select: ray-casting Manipulate: v. hand Doug Bowman - VE Interaction Techniques Introduction Methodology Travel Selection/Manipulation Application Remaining Work Time 33

HOMER Techniques Hand-Centered Object Manipulation Extending Ray. Casting Hybrid technique Select: ray-casting Manipulate: v. hand Doug Bowman - VE Interaction Techniques Introduction Methodology Travel Selection/Manipulation Application Remaining Work Time 34

HOMER Techniques Hand-Centered Object Manipulation Extending Ray. Casting Hybrid technique Select: ray-casting Manipulate: v. hand Doug Bowman - VE Interaction Techniques Introduction Methodology Travel Selection/Manipulation Application Remaining Work Time 35

Formal Taxonomy Selection Indication of Object touch occlude Indication to Select button gesture Attachment Manipulation Release Doug Bowman - VE Interaction Techniques Positioning Introduction Methodology Travel Selection/Manipulation Application Remaining Work hand moves to object user scales to touch object 1 -to-1 hand motion mapping Orientation match tracker orientation indirect control Indication to Release button gesture Final Object Position/Orientation remain in place use physics model 36

Evaluation Framework Introduction Methodology Travel Selection/Manipulation Application Remaining Work Performance measures similar to travel Important outside factors: – task characteristics: DOFs to manipulate – user characteristics: reach, spatial ability – system characteristics: constraints used Doug Bowman - VE Interaction Techniques 37

Guided Design Introduction Methodology Travel Selection/Manipulation Application Remaining Work testbed implemented to allow arbitrary combinations of technique components 4608 possible combinations - reduced to 667 via dependencies and constraints Taxonomy: gaze-based HOMER with separate positioning and orientation Intuition: manipulation based on travel (cross-task technique) Doug Bowman - VE Interaction Techniques 38

Selection/Manipulation Testbed Introduction Methodology Travel Selection/Manipulation Application Remaining Work Tasks that test all important aspects of a select/manip. IT Selection variables: distance, size, density Manip. variables: distance, accuracy, DOFs required Doug Bowman - VE Interaction Techniques 39

Application Case Study: Immersive Design Introduction Methodology Travel Selection/Manipulation Application Remaining Work Verify evaluation results in a complex VE application Design system involves all universal tasks Choose ITs based on testbed results and specified application requirements Doug Bowman - VE Interaction Techniques 40

Interaction Requirements Introduction Methodology Travel Selection/Manipulation Application Remaining Work Travel – exploration and goal-based movements – spatial awareness, info gathering, ease of use Selection – accuracy at a distance, speed, comfort Manipulation – expressibility, accuracy, ease of use Doug Bowman - VE Interaction Techniques 41

Three Levels of Interaction Design Introduction Methodology Travel Selection/Manipulation Application Remaining Work Naive design – taken from CDS application (in D. Bertol, Designing Digital Space) – gaze-directed steering, ray-casting “Intuitive” design iteration – current implementation – pen & tablet, pointing, Go-Go technique Final, systematic design Doug Bowman - VE Interaction Techniques 42

Remaining Work Introduction Methodology Travel Selection/Manipulation Application Remaining Work complete design and evaluation framework for travel design, implement, and run travel testbed complete and run selection/manipulation testbed modify application interaction design and verify with a usability study Doug Bowman - VE Interaction Techniques 43

Contributions formal understanding of tasks/techniques testbeds for future evaluations performance results and models new interaction techniques useful and usable immersive design application Doug Bowman - VE Interaction Techniques 44