Describing Appliance User Interfaces Abstractly with XML Jeffrey
Describing Appliance User Interfaces Abstractly with XML Jeffrey Nichols Workshop on Developing User Interfaces with XML Advanced Visual Interfaces May 25, 2004 Gallipoli, Italy
Problem Appliances are complex and their user interfaces are often hard to use! 1
Solution Separate the user interface from the appliance Specifications Control Feedback Appliances Interface Devices We call this approach the “Personal Universal Controller” (PUC) 2
Language Goals Inform with hand-designed interfaces • What functional information is needed to create interfaces? Additional Requirements • • Support complete functionality of appliance No specific presentation information Short and concise Easy to author “only one way to specify anything” • Sufficient detail for generating high quality interfaces without assistance Full documentation and XML Schema available at: http: //www. cs. cmu. edu/~pebbles/puc/ 3
Language Elements • • • State variables & commands Labels Group tree Dependency information Smart Templates Example media player specification Play, stop, pause, next track, previous track • Play list • 4
Smart Templates A means to specify high-level information Solution • Mark groups with tags that identify highlevel information media-controls, phone-dialpad, time, date, etc. • Restrict the contents of groups so that interface generators can interpret the highlevel meaning • Standardize the tags and restrictions in advance, so that designers know what interface generators expect • Parameterize the tags so one Smart Template can be used for multiple appliances and situations 5
Smart Templates A means to specify high-level information Solution • Mark groups with tags that identify highlevel information media-controls, phone-dialpad, time, date, etc. • Restrict the contents of groups so that interface generators can interpret the highlevel meaning • Standardize the tags and restrictions in advance, so that designers know what interface generators expect • Parameterize the tags so one Smart Template can be used for multiple appliances and situations 6
Actual Appliances We have written specifications for many actual appliances • • • • Sony Digital Camcorder Windows Media Player Axis UPn. P Pan & Tilt Camera Lutron Lighting X 10 Lighting Audiophase Shelf Stereo Audio. Re. Quest MP 3 player GM Vehicle Information System GM Vehicle Climate Control Elevator Telephone/Answering Machine Several Alarm Clocks Task Manager (To Do List) GM Navigation System (in progress) Mitsubishi DVCR (in progress) Used to build apps for Intel’s “Personal Server” Used by group at TU Vienna for controlling AR apps 7
Interface Generators Four Interface Generators for Two Modalities Graphical • Desktop, Pocket. PC, and Microsoft Smartphone Speech Ph. D. work of Thomas Harris • Built on top of the PUC framework • Implemented using Universal Speech Interface (USI) techniques [Rosenfeld 2001] • 8
Language Analysis Pros • Useful for generating both graphical and speech interfaces • Concise • Easily learned by multiple undergraduates • Capable of describing many existing appliances • Has enough detail to generate usable interfaces without designer intervention Cons • No task information Though some “flow” can be specified using dependencies • Describes appliance functions, not appliance user interface • Designed for appliance descriptions, not desktop applications 9
Acknowledgements Funding National Science Foundation • Microsoft • General Motors • Pittsburgh Digital Greenhouse • Equipment Grants • • Mitsubishi (MERL) Vivid. Logic Symbol Technologies Hewlett-Packard Lucent Lutron Lantronix PUC Project Members • • • • Brad A. Myers Kevin Litwack Thomas K. Harris Roni Rosenfeld Michael Higgins Joseph Hughes Rajesh Seenichamy Mathilde Pignol Stefanie Shriver Pegeen Shen Jeffrey Stylos Suporn Pongnumkul Peter Lucas 10
Thanks for listening! For more information… http: //www. cs. cmu. edu/~pebbles/puc/ http: //www. cs. cmu. edu/~jeffreyn/
Describing Appliance User Interfaces Abstractly with XML Jeffrey Nichols Workshop on Developing User Interfaces with XML Advanced Visual Interfaces May 25, 2004 Gallipoli, Italy
More Screenshots 13
Future Work Further develop Smart Templates idea Add task information Focus on “distributed task information” among multiple appliances • Generate one combined user interface for multiple connected appliances (e. g. home theaters) • Ensure interface consistency for similar appliances Evaluate generated interfaces and specification language • User studies, etc. 14
Research Approach 1. Hand-design remote control interfaces 2. Determine functional information needed from appliances to design user interfaces 3. Design language for describing appliance functions 4. Build interface generators for multiple platforms 5. Perform user studies to evaluate the interface generators 15
Benefits of Automatic Generation • Devices not pre-programmed with appliance details • Multiple modalities (GUI + Speech UI) • All interfaces consistent for a user With conventions of the handheld • Even from multiple manufacturers • • Create user interfaces that control multiple connected appliances e. g. a home theater 16
Initial Approach What information is needed about the appliance to automatically generate remote control interfaces? Investigate via a design process Create interfaces by hand AIWA Shelf Stereo • AT&T Telephone/Answering Machine • Improve quality with heuristic analysis and think-aloud studies with several users Compare interfaces with actual appliance interfaces to validate PUC concept Analyze interfaces for functional information 17
Palm Interfaces Initially designed paper-prototype interfaces for Palm telephone stereo 18
Hand-Designed Interfaces for Microsoft’s Pocket. PC (simulated remote control) telephone stereo 19
Comparison Study Compared performance of first-time users (not experts) Appliance Hand-design Phone Stereo Procedure • Each subject worked two sets of tasks both stereo and phone controlled for order and interface Performance Metrics • Time to complete all tasks • Number of times a user manual was needed • Number of missteps 20
Comparison Study Results Using our interfaces, users were twice as fast and made half as many errors All differences are significant (p < 0. 05) 21
Architecture APPLIANCES (Stereo, Alarm Clock, etc. ) PUC DEVICES (automatic interface generation) ADAPTOR (publishes description + appliance state + controls appliance) PROTOCOL (two-way communication of specification & state) device specification & state feedback COMMUNICATION (802. 11, Bluetooth, Zigbee, etc. ) PROTOCOL (two-way communication of specification & state) COMMUNICATION (802. 11, Bluetooth, Zigbee, etc. ) control 22
Language Elements, cont. State Variables and Commands • Represent functions of appliance • State variables have types • • Boolean, Enumeration, Integer, String, etc. Variables sufficient for most functions but not all • e. g. “seek” button on a Radio 23
Language Elements, cont. Label Information One label not suitable everywhere The optimal label length changes with screen size • Speech interfaces may benefit from pronunciation and text-to-speech information • “Label Dictionary” A group of semantically similar labels • Different lengths • Information for different modalities • 24
Language Elements, cont. Label Information One label not suitable everywhere The optimal label length changes with screen size • Speech interfaces may benefit from pronunciation and text-to-speech information • “Label Dictionary” A group of semantically similar labels • Different lengths • Information for different modalities • 25
Language Elements, cont. Group Tree • Specify organization of functions • We use n-ary tree with variables or commands at leaves • Also used for specifying complex types Lists Unions 26
Language Elements, cont. Group Tree • Specify organization of functions • We use n-ary tree with variables or commands at leaves • Also used for specifying complex types Lists Unions 27
Language Elements, cont. Dependency Information • Formulas that specify when a variable or command is active in terms of other state variables Equals, Greater Than, Less Than, Is Defined Linked with logical operators (AND, OR) • Allows feedback to user when a function is not available 28
Smart Templates Example 29
Graphical Interface Generator Rule-based approach • Multiple phases that iteratively transform a specification into a user interface Focuses on panel structure of user interface Small groups of controls have basic layouts • Complexity comes from structure of groups • Structure can be inferred from dependency info! • 30
Generation Process 1. Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc. ) 2. Allocate space • Examine panel contents and choose sizes 3. Instantiate and place controls 4. Fix layout problems 31
Generation Process 1. Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc. ) 2. Allocate space • Examine panel contents and choose sizes 3. Instantiate and place controls 4. Fix layout problems 32
Generation Process 1. Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc. ) 2. Allocate space • Examine panel contents and choose sizes 3. Instantiate and place controls 4. Fix layout problems 33
Generation Process 1. Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc. ) 2. Allocate space • Examine panel contents and choose sizes 3. Instantiate and place controls 4. Fix layout problems 34
Generation Process 1. Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc. ) 2. Allocate space • Examine panel contents and choose sizes 3. Instantiate and place controls 4. Fix layout problems 35
Generation Process 1. Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc. ) Without layout fixing rules 2. Allocate space • Examine panel contents and choose sizes 3. Instantiate and place controls 4. Fix layout problems With layout fixing rules 36
High-Level Conventions Problem • Human designers rely partly on conventions when making an interface • Users expect their appliances to use conventions they know about 37
Smart Templates Need a way to specify high-level information to interface generators Solution • Mark groups with tags that identify highlevel information media-controls, phone-dialpad, time, date, etc. • Restrict the contents of groups so that interface generators can interpret the high-level meaning • Standardize the tags and restrictions in advance, so that designers know what interface generators expect 38
Smart Templates, cont. Features Parameterized • Specified using primitive elements of specification language • 39
Continuing Work Define and implement Smart Templates date, mute, power, time-absolute, volume, etc. • Develop more as more appliances are specified • Combinations of templates Less implementation cost than a new template • e. g. date and time-absolute • Use of templates with data already on controller device e. g. calendar and address information • Might allow user to enter address from contact list into navigation system • 40
Smart Templates, cont. One template can be used across multiple appliances and can be generated on multiple platforms 41
- Slides: 42