- Slides: 46
Interaction Devices Human Computer Interaction CIS 6930/4930 Section 4188/4186
Interaction Performance ► 60 s vs. Today § Performance ► Hz -> GHz § Memory ►k -> GB § Storage ►k -> TB § Input ► punch cards -> ► Keyboards, Pens, tablets, mobile phones, mice, digital cameras, web cams § Output ► 10 character/sec ► Megapixel displays, color laser, surround sound, force feedback, VR ► Substantial bandwidth increase!
Interaction Performance ► Future? Gestural input Two-handed input 3 D I/O Others: voice, wearable, whole body, eye trackers, data gloves, haptics, force feedback § Engineering research! § Entire companies created around one single technology § § ► Current trend: § Multimodal (using car navigation via buttons or voice) § Helps disabled (esp. those w/ different levels of disability)
Keyboard and Keypads ► QWERTY keyboards been around for a long time § § (1870 s – Christopher Sholes) Cons: Not easy to learn Pros: Familiarity Stats: ► Beginners: 1 keystroke per sec ► Average office worker: 5 keystrokes (50 wpm) ► Experts: 15 keystrokes per sec (150 wpm) ► Is it possible to do better? Suggestions?
Keyboard and Keypads Look at the piano for possible inspiration ► Court reporter keyboards (one keypress = multiple letters or a word) ► § 300 wpm, requires extensive training and use ► Keyboard properties that matter § Size large - imposing for novices, appears more complex ► mobile devices ► § Adjustable ► Reduces RSI, better performance and comfort § Mobile phone keyboards, blackberry devices, etc.
► QWERTY § § § ► Keyboard Layouts Frequently used pairs far apart Fewer typewriter jams Electronic approaches don’t jam. . why use it? DVOARK (1920 s) § § 150 wpm->200 wpm Reducing errors Takes about one week to switch Stops most from trying ► ABCDE – style ► Number pads ► Those for disabled § Easier for non-typists § Studies show no improvement vs. QWERTY § What’s in the top row? § Look at phones (slight faster), then look at calculators, keypads § § Split keyboards Key. Bowl’s orbi. Touch (screenshot) Eyetrackers, mice Dasher - 2 d motion with word prediction
Keys ► Current keyboards have been extensively tested § § Size Shape Required force Spacing Speed vs. error rates for majority of users ► Distinctive click gives audio feedback ► § Why membrane keyboards are slow (Atari 400? ) Environment hazards might necessitate ► Usually speed is not a factor ►
Keys Guidelines ► Special keys should be denoted State keys (such as caps, etc. ) should have easily noted states Special curves or dots for home keys for touch typists Inverted T Cursor movement keys are important (though cross is easier for novices) Auto-repeat feature ► Two thinking points: ► This is called Fitt’s Law ► ► § Improves performance, but only if repeat is customizable (motor impaired, young, old) § Why are home keys fastest to type? § Why are certain keys larger? (Enter, Shift, Space bar)
Keypads for small devices ► ► ► PDAs, Cellphones, Game consoles Fold out keyboards Virtual keyboard Cloth keyboards (Elek. Sen) Haptic feedback? Mobile phones § Combine static keys with dynamic soft keys § Multi-tap a key to get to a character § Study: Predictive techniques greatly improve performance § Ex. Letter. Wise = 20 wpm vs 15 wpm multitap ► Draw keyboard on screen and tap w/ pen § Speed: 20 to 30 wpm (Sears ’ 93) ► Handwriting recognition (still hard) § Subset: Graffiti 2 (uses unistrokes)
Pointing Devices ► ► Direct manipulation needs some pointing device Factors: ► Interaction Tasks: § Size of device § Accuracy § Dimensionality § Select – menu selection, from a list § Position – 1 D, 2 D, 3 D (ex. paint) § Orientation – Control orientation or provide direct 3 D orientation input § Path – Multiple poses are recorded ► ex. to draw a line § Quantify – control widgets that affect variables § Text – move text ► ► Faster w/ less error than keyboard Two types (Box 9. 1) § Direct control – device is on the screen surface (touchscreen, stylus) § Indirect control – mouse, trackball, joystick, touchpad
Direct-control pointing ► First device – lightpen § Point to a place on screen and press a button § Pros: ► ► Easy to understand use Very fast for some operations (e. g. drawing) § Cons: ► ► Hand gets tired fast! Hand pen blocks view of screen Fragile Evolved into the touchscreen § Pros: Very robust, no moving parts § Cons: Depending on app, accuracy could be an issue ► 1600 x 1600 res with acoustic wave ► If you don’t show a cursor of where you are selecting, users get confused § Must be careful about software design for selection (land-on strategy). § User confidence is improved with a good lift-off strategy
Direct-control pointing ► Primarily for novice users or large user base ► Case study: Disney World ► Need to consider those who are: disabled, illiterate, hard of hearing, errors in usage (two touch points), etc.
Indirect-Control Pointing ► Pros: ► Cons: ► Mouse § Reduces hand-fatigue § Reduces obscuration problems § Increases cognitive load § Spatial ability comes more into play § Pros: ► ► ► Familiarity Wide availability Low cost Easy to use Accurate ► ► Time to grab mouse Desk space Encumbrance (wire), dirt Long motions aren’t easy or obvious (pick up and replace) § Cons: § Consider, weight, size, style, # of buttons, force feedback
Indirect-Control Pointing ► Trackball § Pros: ► Small physical footprint ► Good for kiosks ► Joystick ► Touchpoint § Easy to use, lots of buttons § Good for tracking (guide or follow an on screen object) § Does it map well to your app? § Pressure-sensitive ‘nubbin’ on laptops § Keep fingers on the home position
Indirect-Control Pointing ► Touchpad § Laptop mouse device § Lack of moving parts, and low profile § Accuracy, esp. those w/ motor disabilities ► Graphics Tablet § Screen shot § comfort § good for cad, artists § Limited data entry
Comparing pointing devices ► Direct pointing § Study: Faster but less accurate than indirect (Haller ’ 84) Lots of studies confirm mouse is best for most tasks for speed and accuracy ► Trackpoint < Trackballs & Touchpads < Mouse ► Short distances – cursor keys are better ► Disabled prefer joysticks and trackballs ► § § ► ► If force application is a problem, then touch sensitive is preferred Vision impaired have problems with most pointing devices ► Use multimodal approach or customizable ► Read Vanderheiden ’ 04 for a case study cursors Designers should smooth out trajectories Large targets reduce time and frustration
Example ► Five fastest places to click on for a righthanded user?
Example ► What affects time?
Fitts’s Law Paul Fitts (1954) developed a model of human hand movement ► Used to predict time to point at an object ► What are the factors to determine the time to point to an object? ► ► § § D – distance to target W – size of target § No, since if Target A is D distance and Target B is 2 D distance, it doesn’t take twice as long What about target size? Not linear there either Just from your own experience, is this function linear? § ► MT = a + b log 2(D/W + 1) § a = time to start/stop in seconds (empirically measured per device) § b = inherent speed of the device (empirically measured per device) § Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, W = 2 cm ► Ans: 300 + 200 log 2(14/2 + 1) = 900 ms § Really a slope-intercept model
Fitts’s Law ► MT = a + b log 2(D/W + 1) § a = time to start/stop in seconds (empirically measured per device) § b = inherent speed of the device (empirically measured per device) § Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, W = 2 cm ► Ans: 300 + 200 log 2(14/2 + 1) = 900 ms § Question: If I wanted to half the pointing time (on average), how much do I change the size? ► ► Proven to provide good timings for most age groups Newer versions taken into account § § § Direction (we are faster horizontally than vertically) Device weight Target shape Arm position (resting or midair) 2 D and 3 D (Zhai ’ 96)
Very Successfully Studied ► ► Applies to § § § Feet, eye gaze, head mounted sights Many types of input devices Physical environments (underwater!) User populations (even retarded and drugged) Drag & Drop and Point & Click § § § Dimensionality Software accelerated pointer motion Training Trajectory Tasks (Accot-Zhai Steering Law) Decision Making (Hick’s Law) § § § Buttons and widget size? Edges? Popup vs. pull-down menus Pie vs. Linear menus i. Phone/web pages (real borders) vs. monitor+mouse (virtual borders) § § § http: //particletree. com/features/visualizing-fittss-law/ http: //www. asktog. com/columns/022 Designed. To. Give. Fitts. html http: //www. yorku. ca/mack/GI 92. html Limitations Results (what does it say about) Interesting readings:
Precision Pointing Movement Time ► Study: Sears and Shneiderman ’ 91 § Broke down task into gross and fine components for small targets § PPMT = a + b log 2(D/W+1) + c log 2(d/W) ►c – speed for short distance movement ► d – minor distance § Notice how the overall time changes with a smaller target. ► Other factors § Age (Pg. 369) ► Research: How can we design devices that produce smaller constants for the predictive equation § Two handed § Zooming
Novel Devices ► Themes: § Make device more diverse Users ► Task ► § Improve match between task and device § Improve affordance § Refine input § Feedback strategies ► Foot controls § Already used in music where hands might be busy § Cars § Foot mouse was twice as slow as hand mouse § Could specify ‘modes’
Novel Devices ► Eye-tracking ► Multiple degree of freedom devices § Accuracy 1 -2 degrees § selections are by constant stare for 200 -600 ms § How do you distinguish w/ a selection and a gaze? § Combine w/ manual input § Logitech Spaceball and Space. Mouse § Ascension Bird § Polhemus Liberty and Iso. Track
Novel Devices ► Boom Chameleon § Pros: Natural, good spatial understanding § Cons: limited applications, hard to interact (very passive) ► Data. Glove Pinch glove Gesture recognition American Sign Language, musical director § Pros: Natural § Cons: Size, hygiene, accuracy, durability § § §
Novel Devices ► Haptic Feedback § § Why is resistance useful? Sens. Able Technology’s Phantom Cons: limited applications Sound and vibration are easier and can be a good approximation ► Rumble pack ► Two-Handed input ► Ubiquitous Computing and Tangible User Interface § Different hands have different precision § Non-dominant hand selects fill, the other selects objects § Active Badges allows you to move about the house w/ your profile § Which sensors could you use? § Elderly, disabled § Research: Smart House § Myron Kruger – novel user participation in art (Lots of exhibit art at siggraph)
Novel Devices ► Paper/Whiteboards § Video capture of annotations § Record notes (special tracked pens Logitech digital pen) ► Handheld Devices § § § PDA Universal remote Help disabled ► Read LCD screens ► Rooms in building ► Maps § Interesting body-context-sensitive. ► Ex. hold PDA by ear = phone call answer.
Novel Devices ► Miscellaneous § Shapetape – reports 3 D shape. ► Tracks limbs ► Engineer for specific app (like a gun trigger connected to serial port) § Pros: good affordance § Cons: Limited general use, time
Speech and Auditory Interfaces ► There’s the dream ► Then there’s reality ► Practical apps don’t really require freeform discussions with a computer § Goals: ► Low cognitive load error rates ► Smaller goals: § Speech Store and Forward (voice mail) § Speech Generation § Currently not too bad, low cost, available
Speech and Auditory Interfaces Bandwidth is much lower than visual displays ► Ephemeral nature of speech (tone, etc. ) ► Difficulty in parsing/searching (Box 9. 2) ► Types ► § § § ► Discrete-word recognition Continuous speech Voice information Speech generation Non-speech auditory If you want to do research here, lots of research in the audio, audio psychology, and DSP field you should understand
Discrete-Word Recognition Individual words spoken by a specific person ► Command control ► 90 -98% for 100 -10000 word vocabularies ► Training ► § Speaker speaks the vocabulary § Speaker-independent ► Still requires § § § Low noise operating environment Microphones Vocabulary choice Clear voice (language disabled are hampered, stressed) Reduce most questions to very distinct answers (yes/no)
Discrete-Word Recognition ► Helps: § § § ► Disabled Elderly Cognitive challenged User is visually distracted Mobility or space restrictions Apps: § Telephone-based info Study: much slower for cursor movement than mouse or keyboard (Christian ’ 00) ► Study: choosing actions (such as drawing actions) improved performance by 21% (Pausch ’ 91) and word processing (Karl ’ 93) ► § However acoustic memory requires high cognitive load (> than hand/eye) ► ► Toys are successful (dolls, robots). Accuracy isn’t as important Feedback is difficult
Continuous Speech Recognition ► ► ► ► Dictation Error rates and error repair are still poor Higher cognitive load, could lower overall quality Why is it hard? § § § Recognize boundaries (normal speech blurs them) Context sensitivity “How to wreck a nice beach” § § Dictate reports, notes, letters Communication skills practice (virtual patient) Automatic retrieval/transcription of audio content (like radio, CC) Security/user ID Much training Specialized vocabularies (like medical or legal) Apps:
Voice Information Systems ► ► Use human voice as a source of info Apps: § § § Tourist info Museum audio tours Voice menus (Interactive Voice Response IVR systems) ► Use speech recognition to also cut through menus ► Voice mail systems ► Get email in your car ► Potentially aides with § If menus are too long, users get frustrated § Cheaper than hiring 24 hr/day reps § Interface isn’t the best § Also helps with non-tech savvy like the elderly § Learning (engage more senses) § Cognitive load (hypothesize each sense has a limited ‘bandwidth’) ► Think ER, or fighter jets
Speech Generation ► Play back speech (games) ► Combine text (navigation systems) ► Careful evaluation! § Speech isn’t always great ► Door is ajar – now just a tone ► Use flash ► Supermarket scanners § § Often times a simple tone is better Why? Cognitive load ► Thus cockpits and control rooms need speech ► Competes w/ human-human communication
Speech Generation ► ► Ex: Text-to-Speech (TTS) Latest TTS uses multiple syllabi to make generated speech sound better § Robotic speech could be desirable to get attention § All depends on app § Thus don’t assume one way is the best, you should user test ► ► Apps: TTS for blind, JAWS Web-based voice apps: Voice. XML and SALT (tagged web pages). ► Use if ► Good when visual displays aren’t that useful. When? § Good for disabled, and also for mobile devices § Message is short § Requires dynamic responses § Events in time § Bad lighting, vibrations (say liftoff)
Non-speech Auditory Interface ► Audio tones that provide information ► Major Research Area § Sonification – converting information into audio § Audiolization § Auditory Interfaces ► Browsers produced a click when you clicked link § § Increases confidence Can do tasks without visual cognitive load Helps figure out when things are wrong Greatly helps visually impaired on a
Non-speech Auditory Interface ► Terms: ► Role in video games is huge ► To create 3 D sound § Auditory icons – familiar sounds (record real world sound and play it in your app) § Earcons – new learned sounds (door ajar) § Emotions, Tension, set mood § Need to do more than stereo § Take into account Head-related transfer function (HRTF) ► Ear and head shape ► New musical instruments ► New ways to arrange music § Theremin
Displays ► ► Primary Source of feedback Properties: § § § § § Physical Dimension Resolution Color Depth and correctness Brightness, contrast, glare Power Refresh rate Cost Reliability # of users
Display Technology ► Monochrome displays (single color) § Low cost § Greater intensity range (medical) ► Color Raster Scan CRT LCD – thin, bright Plasma – very bright, thin LED – large public displays Electronic Ink – new product w/ tiny capsules of negative black particles and positive white § Braille – refreshable cells with dots that rise up § § §
Large Displays ► Wall displays § Informational ► Control rooms, military, flight control rooms, emergency response ► Provides § System overview § Increases situational awareness § Effective team review ► Old: Array of CRTs § Interactive ► Require new interaction methods (freehand sketch, PDAs) ► Local and remote collaboration ► Art, engineering
Large Displays ► Multiple Desktop Displays § Multiple CRTs or Flat panels for large desktops § Cheap § Familiar § Spatial divide up tasks § Comparison tasks are easier § Too much info? ► HMD ► Eventually pixel -> Every surface a
Mobile device displays ► Applications § Personal ► Reprogrammable frames picture § Digital family portrait (Ga. Tech) § Business ► PDAs, § Medical cellphones ► Monitor patients § Research: Modality Translation Services (Trace Center – University of Wisconsin) ► As you move about it auto converts data, info, etc. for you
Mobile device displays ► Actions on mobile devices § Monitor information and alert (calendar) § Gather then spread out information (phone) § Participate in groups and relate to individual (networked devices) § Locate services and identify objects (GPS car system) § Capture and then share info (phone)
Mobile device displays ► Guidelines for design § Bergman ’ 00, Weiss, ’ 02 § Industry led research and design case studies (Lindholm ’ 03) § Typically short in time usage (except handheld games) § Optimize for repetitive tasks (rank functions by frequency) § Research: new ways to organize large amounts of info on a small screen § Study: Rapid Serial Visual Presentation (RSVP) presents text at a constant speed (33% improvement Oquist ’ 03) § Searching and web browsing still very poor performance § Promising: Hierarchical representation (show full document and allow user to select where to zoom into)
Animation, Image, and Video Content quality has also greatly increased ► 3 D rendering is near life-like ► Digital Photography is common ► Scanned documents ► Video compression ► Multimedia considerations for the disabled ► Printers ► § 3 D Printers create custom objects from 3 D models