CHAPTER 7 Direct Manipulation and Immersive Environments Designing

CHAPTER 7: Direct Manipulation and Immersive Environments Designing the User Interface: Strategies for Effective Human-Computer Interaction Sixth Edition Ben Shneiderman, Catherine Plaisant, Maxine S. Cohen, Steven M. Jacobs, and Niklas Elmqvist Addison Wesley is an imprint of in collaboration with Nicholas Diakopoulos © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved.

Direct Manipulation and Immersive Environments Topics 1. Introduction (Ryan) 2. What is Direct Manipulation? (Robert) 3. Some examples of Direct Manipulation (Ethan) 4. 2 D and 3 D Interfaces (Natalie) 5. Teleoperation and Presence (Richard) 6. Augmented and Virtual Reality (Lee) 1 -2 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -2

Introduction • Positive feelings associated with good user interfaces ☺ – Mastery of the interface – Competence in performing tasks – Ease in learning the system originally and in assimilating advanced features – Confidence in the capacity to retain mastery over time – Enjoyment in using the system – Eagerness to show the system off to novices – Desire to explore more powerful aspects of the system 1 -3 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -3

Robert’s presentation outline *** – Definition of Direct Manipulation – Context for the term Direct Manipulation – Principles of Direct Manipulation – Attributes of Direct Manipulation • Direct Manipulation Categorization Continuum 1 -4 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -4

Slide 3 -19 from lecture on Chapter 3. 1 -5 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 3 -5

What is Direct Manipulation? *** 1 -6 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. https: //www. nngroup. com/articles/direct-manipulation/ 7 -6

History of term Direct Manipulation *** ● Term was originally coined by Ben Shneiderman in the early 1980 s 1 -7 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -7

Direct Manipulation *** Direct Manipulation is a central concept of: ● GUI’s ○ WYSIWYG ■ “What You See Is What You Get” 1 -8 https: //www. nngroup. com/articles/direct-manipulation/ © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -8

Principles of Direct Manipulation 1. Continuous representations of the objects and actions of interest with meaningful visual metaphors. 2. Physical actions or presses of labeled buttons, instead of complex syntax. 3. Rapid, incremental, reversible actions whose effects on the objects of interest are visible immediately. 1 -9 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -9

Attributes of Direct Manipulation • Novices can learn basic functionality quickly, usually through a demonstration by a more experienced user • Experts can work rapidly to carry out a wide range of tasks, even defining new functions and features – Knowledgeable intermittent users can retain operational concepts – Error messages are rarely needed • Users – Immediately see whether their actions are furthering their goals, and, if the actions are counterproductive, they can simply change the direction of their activity – Experience less anxiety because the interface is comprehensible and because actions can be reversed easily – Gain a sense of confidence and mastery because they are the initiators of action, they feel in control, and they can predict the 1 -10 interface’s responses © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -10

Attributes of Direct Manipulation (continued) • One way of trying to understand categorize the direct manipulation metaphor is by looking at the translational distance between users and the representation of the metaphor, which will be referred to as strength – Strength can be perceived along a continuum from weak to immersed – This can be further described as the level of indirectness between the user’s physical actions and the actions in the virtual space 1 -11 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -11

Attributes of Direct Manipulation Systems (continued) • Multi-touch allows new actions to be assigned to various combinations of finger touches – The 2 -finger actions like zoom in/out are intuitive, but others must be learned and take longer to discover – This accounts for why a young child can easily learn to tap, change screens, and touch on a tablet (the intuitive actions), but does not have the skills to re-arrange the icons on the screen (the learned actions) 1 -12 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -12

1 -13 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 13

Attributes of Direct Manipulation Systems (concluded) *** Example of Translational Distance Continuum https: //en. wikipedia. org/wiki/D-pad https: //www. oculus. com/en-us/press-kithardware/ https: //www. seeedstudio. com/5 inch-800 x 480 -Capacitive. Touch. Screen. html https: //www. pinterest. com/pin/4944812 34059551062/? lp=true Weak Medium Strong Direct Manipulation Categorization Continuum © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. Immersive 1 -14 7 -14

Direct Manipulation Systems (example) • Three users working concurrently on a large tabletop touch device. – They can use their hands and fingers to manipulate the objects on the device – Note the use of the different hand gestures 1 -16 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -16

Direct Manipulation Systems (example) • A tangible user interface for molecular biology, developed in Art Olson’s Laboratory at The Scripps Research Institute – Utilizes auto-fabricated molecular models tracked with the Augmented Reality Toolkit (from the University of Washington Human Interface Technology Lab) – The video camera on the laptop captures the molecule’s position and orientation, enabling the molecular modeling software to display information such as the attractive/repulsive forces surrounding the molecule 1 -17 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -17

Discussion of Direct Manipulation Problems with direct manipulation • Spatial or visual representations can be too spread out • High-level flowcharts and database-schema can become confusing • Designs may force valuable information off of the screen • Users must learn the graphical representations • The visual representation may be misleading • Typing commands with the keyboard may be faster 1 -18 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -18

Direct Manipulation Systems (example) • GPS solutions – This is a screenshot from Google Street. View of the inside of the University Center at Nova Southeastern University in Florida – On the bottom is a scrollable image of other views on campus. In the left hand corner is a more conventional static map showing the physical street location of the campus – Users can move the “person” to a different location on campus and the views will change accordingly 1 -19 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -19

Direct Manipulation Systems (another example)*** • Video games – Pong • One of the first popular games was Pong • Used knobs to manipulate the white paddles on the screen • Interface objects of Pong – – Paddle Ball Player Rudimentary sound – Games have made many advancements since Pong • • Various controls Multiple objects of interest Dynamic, detailed environments Multiplayer capabilities * The two visual aids used were taken from the Wikimedia Commons © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 1 -20 20

Direct Manipulation Systems (another example) • Video games – Woman playing World of Warcraft, using both her keyboard and mouse, while hearing sounds of the game via her headset 1 -21 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -21

Direct Manipulation Systems (other examples)*** • Computer-aided design and fabrication – Computer-aided design (CAD) systems are used to assist in the design of various things – Autodesk • Computer-aided manufacturing (CAM) – Honeywell’s Experion Process Knowledge System Orion • Home automation • 3 D Printing 1 -22 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 22

Direct Manipulation Systems (another example) • 3 D Printing – This shows astronaut (Bruce Wilmore) onboard the International Space Station with the ratchet wrench that was created with Made in Space’s 3 D printer – This device was designed, qualified, tested, and printed in space in less than one week 1 -23 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -23

Direct Manipulation Systems (another example)*** • Auto. CAD * Video taken from the official Autodesk website. https: //www. autodesk. com/products/autocad/overview © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 1 -24 24

2 D and 3 D Interfaces • “Pure” 3 D interfaces have strong utility in some contexts, for example: – Medical – Product design – Scientific visualization • In some situations, 2 D may actually be preferable to simplify interactions • The power of 3 D interfaces lies in applying them in the appropriate domain or context where the added dimension provides more understanding and improves task outcomes 1 -26 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -26

3 D Interfaces • By using a medical simulation inserted into a large scale visualization (using CAVE technology), physicians were able to find a solution, that would not have been possible without doing the actual surgery 1 -27 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -27

Grape Surgery! 1 -28 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 28

3 D Interfaces (continued) Features for effective 3 D – – – – – Use occlusion, shadows, perspective, and other 3 D techniques carefully Minimize the number of navigation steps for users to accomplish their tasks Keep text readable Avoid unnecessary visual clutter, distraction, contrast shifts, and reflections Simplify user movement Prevent errors Simplify object movement Organize groups of items in aligned structures to allow rapid visual search Enable users to construct visual groups to support spatial recall 1 -29 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -29

3 D Interfaces (continued) Guidelines for inclusion of enhanced 3 D features: – Provide overviews so users can see the big picture – Allow teleoperation – Offer X-ray vision so users can see into or beyond objects – Provide history keeping – Permit rich user actions on objects – Enable remote collaboration – Give users control over explanatory text and let users select for details on demand – Offer tools to select, mark, and measure 1 -30 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -30

3 D Interfaces (concluded) Guidelines for inclusion of enhanced 3 D features (continued): – Implement dynamic queries to rapidly filter out unneeded items – Support semantic zooming and movement – Enable landmarks to show themselves even at a distance – Allow multiple coordinated views – Develop novel 3 D icons to represent concepts that are more recognizable and memorable 1 -31 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -31

Teleoperation and Presence*** • The control of a device or machine remotely • Forms of feedback – – live video haptic touch temperature auditory 1 -32 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 32

1 -33 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 33

Teleoperation and Presence • Two “parents”: – direct manipulation in personal computers – process control in complex environments – common tasks include operating power/chemical plants, manufacturing, and steering vehicles • Physical operation is remote • Complicating factors in the architecture of remote environments: – Time delays • transmission delays • operation delays – Incomplete feedback – Feedback from multiple sources – Unanticipated interferences © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 1 -34 7 -34

Teleoperation and Presence (example) • • Nurse demonstrates using the Telehealth cart otoscope to conduct a real-time tympanic (ear) membrane exam On screen is Physician Assistant who, from a remote location, can see and evaluate the patient and provide an appropriate plan of care 1 -35 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -35

***Teleoperation and Presence • Telepresence – allows a person to feel as if they were present – facilitates formal or informal conversations 1 -36 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 36

***Teleoperation and Presence (example) • Presence led to development of mobile remote presence systems (MRP) 1 -37 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 37

Teleoperation and Presence (example) • Immerse. Board allows two users to be co-located and work on the same shared screen – Teamviewer and Chrome remote desktop share similar features 1 -38 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -38

***Augmented and Virtual Reality 1 -39 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -39

***Virtual Reality • Virtual reality breaks the physical limitations of space and allow users to act as though they were somewhere else 1 -40 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -40

***Virtual Reality Pros and Cons 1 -41 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -41

***Virtual Reality Developments 1 -42 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -42

Virtual Reality (example) • Image of a virtual meditative world that users can use for engaging in meditation activities – The virtual world has sounds – They change with each chakra (stage) of the meditation process – This is an application of positive computing 1 -43 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -43

***Virtual Reality (another example) 1 -44 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -44

***Augmented Reality • Augmented reality shows the real world with an overlay of additional overlay • Situational awareness shows information about the real world that surrounds you by tracking your movements in a computer model • Augmented reality is an important variant – Enables users to see the real world with an overlay of additional interaction. 1 -45 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -45

***Augmented Reality Pros and Cons 1 -46 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -46

Augmented Reality (example) • Augmented reality might be used to help surgeons or their assistants during surgery, by showing pertinent information superimposed on a view of the real world. http: //augmentarium. umiacs. umd. edu 1 -47 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -47

Augmented (another example) • • • Using augmented reality overlays, various points of interest can be shown on a mobile phone Icons represent the type of place (food, shopping, etc. ) and distances from the current location Links are provided to user reviews 1 -48 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -48

Augmented (another example) • Customers can use their personal mobile devices to pull up objects from the IKEA Catalog and see how the various items would look in their own house 1 -49 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -49

***Augmented (another example) 1 -50 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -50

***Augmented (another example) 1 -51 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -51

***Mixed Reality • Mixed reality is the reality that has some aspects of augmented reality within a virtual environment. 1 -52 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -52

***Augmented vs. Virtual Reality 1 -53 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -53

***Questions? 1 -54 © 2017 Pearson Education, Inc. , Hoboken, NJ. All rights reserved. 7 -54