Human Computer Interaction Institute School of Computer Science

  • Slides: 64
Download presentation
Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Handheld Devices for

Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Handheld Devices for Control Brad A. Myers bam@cs. cmu. edu http: //www. cs. cmu. edu/~pebbles

Handheld Devices l Handheld Devices for: l l Communication As a “Personal Information Manager”

Handheld Devices l Handheld Devices for: l l Communication As a “Personal Information Manager” As a “Personal Digital Assistant” As a PC replacement l l “Pocket. PC” Handheld Devices for Remote Control of other devices l l Remote Control of PCs Remote Control of appliances Brad Myers Carnegie Mellon 2

Problem l Appliances are too complex Brad Myers Carnegie Mellon 3

Problem l Appliances are too complex Brad Myers Carnegie Mellon 3

Problem l Too many remotes Brad Myers Carnegie Mellon 4

Problem l Too many remotes Brad Myers Carnegie Mellon 4

Problem l April 8, 1991 Brad Myers Carnegie Mellon 5

Problem l April 8, 1991 Brad Myers Carnegie Mellon 5

Approach l Use a personal handheld device as an Interface to the PC and

Approach l Use a personal handheld device as an Interface to the PC and to the Appliances Brad Myers Carnegie Mellon 6

History of Communication l Parc. Tab – continuous communication l Early “organizers” – no

History of Communication l Parc. Tab – continuous communication l Early “organizers” – no communication l Newton – extra cost for synchronization l Palm – Hot. Sync once a day l Windows. CE/Pocket. PC – Active. Sync once a day Brad Myers Carnegie Mellon 7

Now, Handhelds will be Communicating l 802. 11 Toshiba e 740 HP i. Paq

Now, Handhelds will be Communicating l 802. 11 Toshiba e 740 HP i. Paq 3870 l Blue. Tooth l Cell-phone network (G 3) l (Infrared) l (Wires or cradle) Microsoft Smart. Phone Brad Myers Carnegie Mellon 8

Premises of our Research “With the coming wireless technologies, connecting the PCs and PDAs

Premises of our Research “With the coming wireless technologies, connecting the PCs and PDAs together will no longer be an occasional event for synchronization. Instead, the devices will frequently be in close, interactive communication. ” — Brad Myers, “Using Hand-Held Devices and PCs Together, ” Comm. ACM, Vol. 44, No. 11. Nov. , 2001. pp. 34 - 41. Carnegie Mellon 9

Pebbles Project l l Performed as part of the Pebbles Project Overall goal: Use

Pebbles Project l l Performed as part of the Pebbles Project Overall goal: Use of multiple devices at the same time l l l Palm and desktop PC Mobile Phone and “Smart Home” Multiple handhelds in a meeting Pocket PC and appliances Multiple users with their devices Single user with multiple devices Brad Myers Carnegie Mellon 10

Pebbles is: P DAs for E ntry of B oth B ytes and L

Pebbles is: P DAs for E ntry of B oth B ytes and L ocations from E xternal S ources. http: //www. pebbles. hcii. edu/ Brad Myers Carnegie Mellon 11

Controlling a PC Remote Commander l Slide Show Commander l “Semantic Snarfing” l Shortcutter

Controlling a PC Remote Commander l Slide Show Commander l “Semantic Snarfing” l Shortcutter l http: //www. pebbles. hcii. edu/

Remote Commander l Allow PDAs to control a PC Can be used with any

Remote Commander l Allow PDAs to control a PC Can be used with any application l Uses the standard (single) cursor l Don’t have to jump up and grab mouse l All mouse and keyboard functions l Use PDA like touchpad l Graffiti or our own pop-up keyboard l l Word prediction and completion Brad Myers Carnegie Mellon 13

Pocket. PC version l l Get PC’s screen onto Pocket. PC Full view, or

Pocket. PC version l l Get PC’s screen onto Pocket. PC Full view, or one-to-one zooming l l Scroll with i. Paq’s buttons Control or scribble Brad Myers Carnegie Mellon 14

For People with Muscular Disabilities l Using handhelds as interface to PCs l People

For People with Muscular Disabilities l Using handhelds as interface to PCs l People with Muscular Dystrophy have fine-motor control but lose gross motor control l l Difficulties with mouse and keyboard, but stylus OK So use Remote Commander as PC’s keyboard and mouse Brad Myers Carnegie Mellon 15

NEW! l Alphabet for Accurate Gestures Use a square hole in a template to

NEW! l Alphabet for Accurate Gestures Use a square hole in a template to increase accuracy l l Just as easy to learn as Graffiti™ 18% more accurate for able novices (p<. 02) Patent Pending Also works with joystick Brad Myers Carnegie Mellon 16

Shared Clipboard l Transfer information among your personal devices l l l E. g.

Shared Clipboard l Transfer information among your personal devices l l l E. g. , between PDA and PC Also between multiple computers Connect all clipboards together Transfer content or reference Works with all applications Brad Myers Carnegie Mellon 17

Power Point Control l l Use PC to give the presentation Use hand-held to

Power Point Control l l Use PC to give the presentation Use hand-held to control the PC l l Two-way communication Hand-held shows picture of slide, notes, timer Brad Myers Carnegie Mellon 18

Slide. Show Commander, cont. l l l See list of slide titles Scribble Tap

Slide. Show Commander, cont. l l l See list of slide titles Scribble Tap on on-screen buttons and links Preview other slides Control demonstrations Brad Myers Carnegie Mellon 19

Control at a Distance l Controlling a PC from across the room l l

Control at a Distance l Controlling a PC from across the room l l Use a laser pointer? l l Meetings, etc. Studies “Semantic Snarfing” l Grab contents to handheld Brad Myers Carnegie Mellon 20

Shortcutter l l User-created panels of controls Create custom interfaces and extensions to PC

Shortcutter l l User-created panels of controls Create custom interfaces and extensions to PC applications l l l And then take them with you Direct manipulation for edit, then set properties Pocket. PC or Palm Brad Myers Carnegie Mellon 21

Shortcutter Widgets l Buttons l Virtual Joy stick l Virtual Knob l Sliders l

Shortcutter Widgets l Buttons l Virtual Joy stick l Virtual Knob l Sliders l Mouse pad l Gesture panel Brad Myers Carnegie Mellon 22

Scenarios of Use l Lean-back mail reading l Controlling media players l … and

Scenarios of Use l Lean-back mail reading l Controlling media players l … and many others Brad Myers Carnegie Mellon 23

Controlling Appliances l l Ph. D research of Jeffrey Nichols Assisted by undergrads and

Controlling Appliances l l Ph. D research of Jeffrey Nichols Assisted by undergrads and MS students http: //www. pebbles. hcii. edu/puc/

Problem l Too many complex devices, each with its own idiosyncratic interface l l

Problem l Too many complex devices, each with its own idiosyncratic interface l l l l Stereo system Telephones ATM Fax machine Photocopier Hotel alarm clock Increasingly computerized Low usability

Existing “universal” controls Pre-programmed at the factory with a subset, l Or, laboriously hand-programmed

Existing “universal” controls Pre-programmed at the factory with a subset, l Or, laboriously hand-programmed by the user l Brad Myers Carnegie Mellon 26

Our Approach l l Handheld is a “Personal Universal Controller” (PUC) Two-way communication l

Our Approach l l Handheld is a “Personal Universal Controller” (PUC) Two-way communication l l Appliance describes its functions Handheld PUC l Automatically creates interface l Controls the appliance l Displays feedback about appliance status Specifications Control Feedback Brad Myers Carnegie Mellon 27

Automatic Generation of UIs l Benefits l All interfaces consistent for a user With

Automatic Generation of UIs l Benefits l All interfaces consistent for a user With conventions of the handheld l Even from multiple manufacturers l l Addresses hotel alarm clock problem Can take into account user preferences l Multiple modalities (GUI + Speech UI) l l A Hard Problem l Previous automatic systems have not generated high quality interfaces Brad Myers Carnegie Mellon 28

Research Challenges 1. Automatic Design of Graphical User Interfaces 2. Automatic Design of Speech

Research Challenges 1. Automatic Design of Graphical User Interfaces 2. Automatic Design of Speech User Interfaces 3. Connection with real devices l l l Through various protocols X-10, AV/C, HAVi, UPn. P, etc. Also, custom hardware and software Brad Myers Carnegie Mellon 29

Hand-Generated Graphical Interfaces l First, Hand-Designed Pocket. PC interfaces: l l l AIWA Shelf

Hand-Generated Graphical Interfaces l First, Hand-Designed Pocket. PC interfaces: l l l AIWA Shelf Stereo (Tape, CD, Tuner) AT&T Telephone/Answering Machine Used Embedded Visual Basic Ensured quality with heuristic analysis and think-aloud studies Compared with manufacturer’s interfaces Brad Myers Carnegie Mellon 30

Results of Comparison l l Using PUC, users took 50% less time & made

Results of Comparison l l Using PUC, users took 50% less time & made 50% fewer errors All differences are significant (p < 0. 05) Brad Myers Carnegie Mellon 31

Current PUC Specification Language l l XML Full documentation for the specification language and

Current PUC Specification Language l l XML Full documentation for the specification language and protocol: http: //www. pebbles. hcii. cmu. edu/puc/ l Contains sample specification for a stereo Brad Myers Carnegie Mellon 32

Properties of PUC Language l State variables & commands l Each can have multiple

Properties of PUC Language l State variables & commands l Each can have multiple labels l l Typed variables l l l Useful when not enough room Base types: Boolean, string, enumerated, integers, fixed-point, floating-point, etc. Optional labels for values Hierarchical Structure l Groups Brad Myers Carnegie Mellon 33

Dependency Information l l Crucial for high-quality interfaces Expressed as <active-if> clauses l Operations:

Dependency Information l l Crucial for high-quality interfaces Expressed as <active-if> clauses l Operations: l l Combined Logically l l Equals, Less-Than, Greater-Than AND, OR Used for: l l l Dynamic graying out Layout Widget selection Brad Myers Carnegie Mellon 34

Specifications l Have working specifications for: l l l l l Audiophase stereo X-10

Specifications l Have working specifications for: l l l l l Audiophase stereo X-10 lights control Sony Cam. Corder Windows Media Player Audio Re. Quest hardware MP 3 player Win. Amp Media Player Elevator Parts of GMC Yukon Denali SUV Etc. Brad Myers Carnegie Mellon 35

Controller Generators l i. Paq Pocket. PC l Smart. Phone l No touchscreen l

Controller Generators l i. Paq Pocket. PC l Smart. Phone l No touchscreen l Desktop (Tablet. PC) l Speech Brad Myers Carnegie Mellon 36

Examples of Generated GUIs Shelf stereo Brad Myers Windows Media Player Carnegie Mellon 37

Examples of Generated GUIs Shelf stereo Brad Myers Windows Media Player Carnegie Mellon 37

Generating Speech Interfaces l “Universal Speech Interface” (USI) project l l l Creates grammar,

Generating Speech Interfaces l “Universal Speech Interface” (USI) project l l l Creates grammar, language model and pronunciation dictionary from PUC specification l l Prof. Roni Rosenfeld of CMU http: //www. cs. cmu. edu/~usi Pronunciation from labels using phonetic rules Can provide other pronunciations as labels for fine-tuning Will use dependency information to help with disambiguation and explanation Supports queries and spoken feedback l Paraphrases as confirmation Brad Myers Carnegie Mellon 38

Adaptors l “Adaptors” provide the interface to existing (and future) appliances If do not

Adaptors l “Adaptors” provide the interface to existing (and future) appliances If do not support specification language directly Custom hardware l AV/C (standard protocol) l Sony Cam. Corder Custom software l Lutron l HAVi l Windows Media Player l UPn. P X-10 l Axis Camera l l l Light switches, etc. Brad Myers Carnegie Mellon 39

Demonstration l l Sony Cam. Corder Windows Media Player Simulated automobile Two way communication:

Demonstration l l Sony Cam. Corder Windows Media Player Simulated automobile Two way communication: l l When state changes from application or appliance, GUI is updated Can query state with speech Brad Myers Carnegie Mellon 40

Future Work on PUC l Controlling User “Experiences” l l Consistency for Users l

Future Work on PUC l Controlling User “Experiences” l l Consistency for Users l l One command for multiple devices Play a DVD sets DVD, cable, TV Setting alarm clock sets heat, coffee, car And user preferences More devices l l UPn. P Etc. Brad Myers Carnegie Mellon 41

For More Information l Many papers on the Pebbles web site l l On

For More Information l Many papers on the Pebbles web site l l On PUC and other work Most programs available for free downloading: http: //www. cs. cmu. edu/~pebbles l Slide. Show Commander was licensed for commercial sale Brad Myers Carnegie Mellon 42

Conclusions l l As more and more electronics are computerized and communicating, mobile devices

Conclusions l l As more and more electronics are computerized and communicating, mobile devices can help control them Handheld devices can improve the user interfaces of everything else l Not just be another gadget to be learned Brad Myers Carnegie Mellon 43

Thanks to Pebbles Students l This research performed by a large number of students

Thanks to Pebbles Students l This research performed by a large number of students since 1997: Htet Aung Rishi Bhatnagar Ben Bostwick Franklin Chen Yu Shan Chuang Karen Cross Carl Evankovich Marc Khadpe Dave Kong Chun-Kwok Lee Joonhwan Lee Brad Myers Jennifer Li Yuhua Li Leo Lie Jack Lin Kevin Litwack A. Chris Long Rob Miller Jeff Nichols Choon Hong Peck Mathilde Pignol Suporn Pongnumkul Carnegie Mellon Rajesh Seenichamy Pegeen Shen Herbert Stiel Jeff Stylos Claire Tokar Marsha Tjandra Adrienne Warmack Jacob O. Wobbrock Jerry Yang Sunny Yang Brian Yeung 44

Thanks to Our Sponsors! The Pebbles research is supported by grants from: DARPA Microsoft

Thanks to Our Sponsors! The Pebbles research is supported by grants from: DARPA Microsoft NSF General Motors NEC Foundation Pittsburgh Digital Greenhouse And equipment grants from: Hewlett Packard Lucent Technologies Lantronix, Inc. Lutron, Inc. Mitsubishi Palm Computing Brad Myers Symbol Technologies IBM SMART Technologies Vivid. Logic Synergy Solutions Handango Carnegie Mellon 45

Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Handheld Devices for

Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Handheld Devices for Control Brad A. Myers bam@cs. cmu. edu http: //www. cs. cmu. edu/~pebbles

Black, followed by HAVi

Black, followed by HAVi

Pebbles PUC Project use of the Home Audio Visual Interface

Pebbles PUC Project use of the Home Audio Visual Interface

About HAVi l Home Audio Visual Interface l l Protocol for controlling appliances over

About HAVi l Home Audio Visual Interface l l Protocol for controlling appliances over IEEE 1394 (i. LINK® or Fire. Wire®) cables l l http: //www. havi. org Mostly used for video equipment Supported by various Mitsubishi and RCA high-end products Brad Myers Carnegie Mellon 49

Our Goals for HAVi l Want to: l l Detect HAVi devices on Firewire

Our Goals for HAVi l Want to: l l Detect HAVi devices on Firewire bus Automatically create PUC specification from device capabilities Control all of the appliance’s features using a PC as a server (proxy) HAVi seems to have appropriate features for these Brad Myers Carnegie Mellon 50

HAVi Experience l MERL donated l l l A 65 -inch high-definition TV (WS-65909)

HAVi Experience l MERL donated l l l A 65 -inch high-definition TV (WS-65909) A high definition digital VCR (HS-HD 2000 U) Vivid. Logic Fire. Bus HAVi SDK for Linux, v. 1. 0. 8 Brad Myers Carnegie Mellon 51

What We’ve Done l Investigated the (several) different ways of controlling HAVi devices l

What We’ve Done l Investigated the (several) different ways of controlling HAVi devices l l Made simple applications to control HAVi devices But don’t have a working PUC interface l Have put efforts on hold Limitations of HAVi l Problems with SDK l Brad Myers Carnegie Mellon 52

Television l l l Under the HAVi system, acts as the “master controller” Refuses

Television l l l Under the HAVi system, acts as the “master controller” Refuses to be controlled or to respond to our requests Even having the TV and PC on same bus causes problems Brad Myers Carnegie Mellon 53

VCR l Several different ways to control VCR FCM “Functional Control Module” l Static

VCR l Several different ways to control VCR FCM “Functional Control Module” l Static list of capabilities l l Havlet l l A Java program uploaded and run on TV (or PC) DDI “Data-Driven Interaction” l Panel-based interface for arbitrary control l Brad Myers Carnegie Mellon 54

VCR FCM l A few more features than AV/C: l l l Play, Record,

VCR FCM l A few more features than AV/C: l l l Play, Record, Fast Forward, Fast Reverse, Variable Forward, Variable Reverse, Stop, Record-Pause, Skip, Eject Media, Set Recording Mode, Clear. RTC Get State, Get Recording Mode, Get Format, Get Position, Get capability, Get Reject Info No querying for capabilities (Apparent) Bugs in SDK on getting status Have created simple programs to control VCR using the FCM Brad Myers Carnegie Mellon 55

Hav. Lets l l l Java Applets downloaded to a Java interpreter No hooks

Hav. Lets l l l Java Applets downloaded to a Java interpreter No hooks for programmatic control Very limited functions provided in the VCR’s Hav. Let l l Rewind, Play, Fast Forward, Stop, Pause, Record Status, Counter, Media Type, Recording Rate of Tape, Rate of Data Source Brad Myers Carnegie Mellon 56

Data-Driven Interaction (DDI) l l We hoped this would provide device querying and control,

Data-Driven Interaction (DDI) l l We hoped this would provide device querying and control, but… Only limited features provided in DDI l l l Same as Hav. Let interface Rewind, Play, Fast Forward, Stop, Pause, Record, and some status requests Provides each UI element as a separate item l l E. g. , “Time: ” and “ 54: 35: 33” may be two separate labels placed next to each other Need to infer relationships and types by layout Brad Myers Carnegie Mellon 57

Issues with Vivid. Logic’s SDK l l Their engineers have been very helpful SDK

Issues with Vivid. Logic’s SDK l l Their engineers have been very helpful SDK incomplete, poorly documented and quite buggy Only runs with a specific desktop Firewire card, only on Linux (for now) Slow to come out with new versions l Needed to be “FAV” (Full AV) to use DDI but FAV support not available until Dec’ 02 Brad Myers Carnegie Mellon 58

Demo l l Show other devices we have gotten to work demo Brad Myers

Demo l l Show other devices we have gotten to work demo Brad Myers Carnegie Mellon 59

Black, followed by OLD

Black, followed by OLD

Adaptor for AV/C PUC Java Proxy Hand-written spec of AV/C devices PUC Java Library

Adaptor for AV/C PUC Java Proxy Hand-written spec of AV/C devices PUC Java Library Our Java device code Multiple device controllers Automatic GUI generation Automatic Speech UI generation Our C++ code Microsoft Direct. Show library AV/C protocol Firewire cable (IEEE 1394) Brad Myers Digital Video Camera or VCR Carnegie Mellon 61

Adaptor using Custom Hardware for Stereo l l l Pretends to send IR codes

Adaptor using Custom Hardware for Stereo l l l Pretends to send IR codes Reads LED panel signals to decode state Created by Pittsburgh company: Maya Design Brad Myers Carnegie Mellon 62

Driver Information Console l Simulator for the Driver Information Console of Yukon Denali SUV

Driver Information Console l Simulator for the Driver Information Console of Yukon Denali SUV l l All the modes and messages Full emulation of steering wheel buttons Brad Myers Carnegie Mellon 63

Simulator for HVAC Brad Myers Carnegie Mellon 64

Simulator for HVAC Brad Myers Carnegie Mellon 64