Lecture 14 Modelbased tools Creating the UI Automatically
Lecture 14: Model-based tools: Creating the UI Automatically Brad Myers 05 -830 Advanced User Interface Software Spring 2020 © 2020 - Brad Myers 1
Homeworks l Homework 2 graded, on Canvas with test program l l l Fix drawing, refresh, clipping, layout-groups If grade <=80, can get up to 10 points back if fix lots of bugs Homework 3? © 2020 - Brad Myers 2
Model-Based Tools l Overview l Programmer describes the operation of the system or the user interface in a specification language = the "model". l l l System automatically creates the interface l l model is a high-level description usually declarative (listing parts and relationships) Uses a low-level toolkit for the widgets Note: different use of word “model” than in lecture 8 © 2020 - Brad Myers 3
Goals: l l High-level description of an interface is easier to write than low-level toolkit code Automatic generation may produce better UIs than programmers Allow separation of UI design (embodied in rules) from UI contents (supplied by the programmer) Support dynamic creation of objects l l l define templates or prototypes Increase re-use since design rules shared by multiple applications Tools can reason over the specification to produce extra stuff: l l Automatic generation of help, undo, etc. Transform interface into different but functionally equivalent interface Enabling and disabling of widgets Enforcement or checking of design guidelines- consistency, completeness l l l Enforces consistency since rules will pick similar objects for similar situations Automatic adjustment to different screen sizes, etc. , since rules can take this into account Automatic analysis for quality based on the specification © 2020 - Brad Myers 4
Overview, cont. l Related to the "Declarative" approach discussed in the “Historical Perspectives” lecture 10 l l but here system has some intelligence or knowledge so less has to be specified by the programmer. Different types: l l l Dialog box creators: Mickey, DON, Jade (lots of others) Representations of the full UI: ITS, UIDE, Humanoid, Master. Mind, Later: (XML-based) Concur. Task. Trees, Xweb, ICrafter, Supple, PUC © 2020 - Brad Myers 5
Dialog Box Creators l l Easiest part of the UI to create Given a list of the contents, automatically: 1. choose widgets: l specify type of desired input: string = text input field number = slider one-of-many = radio buttons or pop-up options many-of-many = check boxes or checks in a menu commands = menu © 2020 - Brad Myers 6
Dialog Box Creators, cont. 2. arrange widgets l based on look-and-feel guidelines § § l where OK goes which commands go in which menus based on good graphic design principles. 3. set variables l to reduce the number of callbacks necessary © 2020 - Brad Myers 7
Example: Mickey l l l D. R. Olsen, Jr. . 1989. A programming language basis for user interface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '89), K. Bice and C. Lewis (Eds. ). ACM, New York, NY, USA, 171 -176. DOI=http: //dx. doi. org/10. 1145/67449. 67485 Programmer specifies UI by putting special comments in a Pascal file. Uses the Apple Macintosh guidelines Pre-processor to parse the Pascal code and generate the Macintosh resources. Maps Procedures into Menu items. l If parameter is one of a standard set, pops up appropriate dialog box or waits for input l l File to be read, file to be written New point, line or rectangle 8 © 2020 - Brad Myers
Mickey, cont. l Variables: l l l Enumerated types mapped to check lists separated by lines. Sets the variables when changed. Enumerated types with 2 choices mapped to name changes Booleans: single checked items © 2020 - Brad Myers 9
Jade l Brad Vander Zanden and Brad A. Myers, "Automatic, Look-and-Feel Independent Dialog Creation for Graphical User Interfaces, " Proceedings SIGCHI'90: Human Factors in Computing Systems. Seattle, WA, April 1 -5, 1990. pp. 27 -34. ACM DL Reference l "Judgment-based Automatic Dialog Editor" Given a textual specification of just the contents and their types, creates a dialog box l l Separately specify which look-and-feel (not part of the specification) Defines mapping from types to widget selection Graphic design rules for "nice" layout © 2020 - Brad Myers 10
DON l (Won Chul Kim & Foley, Inter. CHI'93, pp. 430 -437) l l l Ultimate in dialog box layout Kim's Ph. D thesis Works with Open. Look and dev. Guide Allows interactive designer guidance (preferences) on sizes, layout, widget choice, etc. l l ACM DL Reference Can also choose among proposed layouts Sophisticated 2 -D layout l l Tries to balance dialog box Groupings of related items Effective use of white space (even margins, minimize wasted space) Generates multiple designs and uses an evaluation metric to choose. © 2020 - Brad Myers 11
Don, pictures l Paper, with other pictures © 2020 - Brad Myers 12
Generating Full UI l l l These next tools require a specification of the full UI Usually have rule-based components Specifications are in a special language © 2020 - Brad Myers 13
Interactive Transaction System (ITS) l l Bennett, et. al. , UIST'89 pp. 67 -75 Wiecha, et. al. CHI'89, pp. 277 -282 Wiecha, et. al. , ACM TOIS, 8(3), Jul'90, pp. 204 -236 Goal: capture designers knowledge as style rules l l So unlike other systems, designer is required to edit the rules, not just the specification All of UI must be created by editing the rules l l no interactive editing of generated interface (since then the knowledge about why the generated interface wasn't good enough would be lost) Like dialog-box systems, separate specification of content and style l l l Style-independent tags associated with content "Style expert" programs the style for each tag Styles include both output (display) and input (interaction techniques) 14 specifications © 2020 - Brad Myers
ITS, cont. l l l Can handle dialog boxes, forms, node-link diagrams, kiosk frames, etc. Used for a number of internal IBM applications Used for all the information services at Expo'90 in Spain l l l Evaluation l l Information, maps, restaurant reservations, etc. IBM researchers and content experts were in Spain for months + Full representation of design may increase re-use - Design specification ends up containing many specific "hacks" used to achieve specific effects in single interfaces - Complex formal language for specification and rules Pictures from ITS, TOIS, 8(3), pp. 213, 215, 217 © 2020 - Brad Myers 15
ITS, pictures l Paper n © 2020 - Brad Myers n 16
The User Interface Design Environment (UIDE) l l Foley, et. al. CHI'88, pp. 67 -72 Foley, et. al. IEEE Software, Jan'89, 25 -32; Sukaviriya, et. al. Inter. CHI'93, pp. 375 -382 Long-term project of Jim Foley's at George Washington and Georgia Tech l l Ended about 1994 when Foley left A loose collection of separate implementations: l l l IDL's transformations DON dialog boxes (described above) Sukaviriya's animated help Martin Frank's work (EET in Event-Based lecture) © 2020 - Brad Myers - etc. 17
UIDE l Programmer defines Knowledge-base "schemas" describing parts of the interface: l l l Pre-conditions and post-conditions are in a very limited language l l l Objects: in a class, sub-class inheritance hierarchy Actions: what can be done in the interface counting, Booleans, simple tests used for testing enabled and explaining why Transformations change among equivalent UIs: l l e. g. Currently-selected obj <=> currently-selected cmd performed based on pre-, post-conditions © 2020 - Brad Myers 18
Pictures from UIDE © 2020 - Brad Myers 19
UIDE, cont. l Automatic generate help for why commands are not available l l Sukaviriya, et. al. AVI’ 94, Pages: 44 - 52 Animated help provides animations as a tutorial l Determines what needs to be done to demonstrate action Sequence of actions E. g. bring windows to the front, create an object, etc. © 2020 - Brad Myers 20
UIDE l Evaluation l l + Support for more than dialog boxes - Pre and post condition language is weak l l can't express the test "if the selected object is a polygon. . . " - Model language is a different, difficult language to learn © 2020 - Brad Myers 21
Humanoid l l l l Szekely, et. al. UIST'90, pp. 1 -9 Szekely, et. al. CHI'92, pp. 507 -514 Szekely, et. al. Inter. CHI'93, pp. 383 -390 High-level UIMS for Manufacturing Applications Needing Organized Iterative Development Model application data and interaction similar to UIDE Model whole application: semantics + interface Used Garnet! © 2020 - Brad Myers 22
Humanoid, cont. l l System picks generic interaction techniques immediately using "templates" Designer can refine interface iteratively by creating more specific sub-classes: l l Single-Command-Input-With-Alternatives Single-Command-Input-With-Few-Alternatives Allows exploration with incomplete designs Evaluation l l + Much richer specification language than UIDE - More complex to define interfaces (more to learn) l but interactive tools help © 2020 - Brad Myers 23
Humanoid Pictures from Humanoid, CHI'93 pp. 384 © 2020 - Brad Myers 24
Master. Mind l l Neches, et. al. ACM 1993 Intelligent User Interfaces Workshop, pp. 63 -70 Models Allowing Shared Tools and Explicit Representations to Make Interfaces Natural to Develop Idea: combine UIDE and Humanoid Support entire life-cycle: early conceptual design through maintenance l l l Knowledge base is shared among all tools Knowledge base serves as an integrating framework for various tools at design time and run time. Spent a lot of time negotiating on how to combine models Lots of different parts to the model Personnel and coordination problems in doing the research Used Amulet! © 2020 - Brad Myers 25
Master. Mind l One of the first system to integrate multiple models together © 2020 - Brad Myers 26
Mastermind Pictures © 2020 - Brad Myers 27
Research continued… Fragmented into two different approaches l Software engineering approach (early 90’s-) l l Very detailed models to improve overall design process Intelligent design assistants instead of automatic generation Significant use of task models Ubiquitous computing approach (2000 -) l l l Tons of “invisible” processors that perform tasks for us UIs for these processors are presented on other devices (mobile phone, PDA, speech, etc. ) Impossible to manually build user interfaces for every combination © 2020 - Brad Myers 28
What are task models, anyway? l Key part of many current HCI approaches Description of the process a user takes to reach a goal in a specific domain Typically have hierarchical structure l Number of different task modeling languages l l Ø GOMS, Cog. Tool UAN Concur. Task. Trees © 2020 - Brad Myers 29
Hierarchical Task Model l H. Rex Hartson and Pardha S. Pyla. The UX Book: Process and Guidelines for Ensuring a Quality User Experience. Morgan Kaufmann / Elsevier. 2012. http: //www. theuxbook. net/. p. 218 © 2020 - Brad Myers 30
Software Engineering Approach l l l Commercial work “Model-based design” Example: BPMN “business process modeling notation” l l l Business experts should be able to author models Converted into code to support the process (requires people) Keynote at ICSE’ 08: Herbert Hanselmann: Challenges in Automotive Software Engineering l “Model-based design (MBD) of functional behaviour has been a big help in the recent past” © 2020 - Brad Myers 31
Concur. Task. Trees l Developed by Fabio Paterno et al. for the design of user interfaces l Goals l l l Graphical for easy interpretation Concurrent model for representing UI tasks Different task types l Represent all tasks, including those performed by the system © 2020 - Brad Myers 32
Building/Editing Task Models l Tools are available l Concur. Task. Trees Environment http: //giove. cnuce. cnr. it/ctte. html or search for “Concur. Task. Trees” n © 2020 - Brad Myers 33
Mobi-D l l l Angel Puerta, IUI’ 97 Set of tools to support a clearly defined development cycle Uses a series of different models Explicit relationships that specify how the models are related to each other Explicit interaction between end users and developers © 2020 - Brad Myers 34
XIML e. Xtensible Interface Markup Language l XIML. org l Based on Mobi-D work l Supports full development lifecycle l Used by Red. Whale Software to drive their interface consultant business l They developed many tools l l Move interaction data to/from XIML Leverage data in XIML to better understand various interfaces Automate parts of the interface design process Apparently still in business! © 2020 - Brad Myers 35
Other Systems l UIML l l l Originally a research project at Virginia Tech, then developed commercially by Harmonia Goal is platform independent language for describing UI Early versions were not very platform independent Later versions using task models to automatically generate parts of the old language that were not platform independent Teresa l l l (http: //www. harmonia. com/) Transformation Environment for inte. Racti Tool for taking Concur. Task. Trees models, building an abstract interface, and then building a concrete interface on multiple platforms. USIXML l l (http: //giove. cnuce. cnr. it/teresa. html) (http: //www. usixml. org) Many of the same features of XIML Novel aspect is the use of graph structure for modeling relations (seems very complex) © 2020 - Brad Myers 36
Ubiquitous Computing Approach “Pervasive computing cannot succeed if every device must be accompanied by its own interactive software and hardware…What is needed is a universal interactive service protocol to which any compliant interactive client can connect and access any service. ” -Dan Olsen (Xweb paper) The web comes close to solving this problem, but is interactively insufficient. © 2020 - Brad Myers 37
Systems addressing UI issues l XWeb l l ICrafter l l A system for integrating user interfaces from multiple devices Supple l l Later renamed ICE – Interactive Computing Everywhere A system for automatically generating interfaces with a focus on customization/personalization. Personal Universal Controller l Jeff Nichol’s research © 2020 - Brad Myers 38
XWeb l Work by Dan Olsen and group at BYU E. g. UIST’ 2000, pp. 191 - 200 l Premise: l l l Apply the web metaphor to services in general Support higher levels of interactivity © 2020 - Brad Myers 39
XWeb Example Interface © 2020 - Brad Myers 40
ICrafter l l l Part of the Interactive Workspaces research project at Stanford Ponnekanti, et. al. Ubicomp’ 2001 Main objective: l l “to allow users of interactive workspaces to flexibly interact with services” Contribution l l An intelligent infrastructure to find services, aggregate them into a single interface, and generate an interface for the aggregate service. In practice, much of the interface generation is done by hand though automatic generation is supported. © 2020 - Brad Myers 41
i. Crafter UI l l Top: automatic; bottom: custom http: //radlab. cs. berkeley. edu/ people/fox/static/pubs/pdf/c 08. pdf © 2020 - Brad Myers 42
Supple l l Support automatic personalization of user interfaces Treats generation of interfaces as an optimization problem Can take into account usage patterns in generation Krzysztof Gajos and Daniel S. Weld, “SUPPLE: Automatically Generating User Interfaces” in Proceedings of Intelligent User Interfaces 2004, Funchal, Portugal. © 2020 - Brad Myers 43
Modeling Users with Traces l Supple uses traces to keep a usage model l Sequences of events: l l l <interface element, old value, new value> Interfaces are rendered taking the traces into account (though traces are not required) Trails are segmented at interface close or reset © 2020 - Brad Myers 44
Generating with Optimization l Uses a branch-andbound search to explore space of alternatives l Guaranteed to find an optimal solution © 2020 - Brad Myers 45
Screenshots © 2020 - Brad Myers 46
Personal Universal Controller l Jeff Nichol’s Ph. D work l l Problem: l l ~2002 - 2006 Appliance interfaces are too complex and too idiosyncratic. Solution: l Separate the interface from the appliance and use a device with a richer interface to control the appliance: l l PDA, mobile phone, etc. Goal: Generate high-quality UIs © 2020 - Brad Myers 47
Approach Use mobile devices to control all appliances in the environment Specifications Control n Feedback Mobile Devices Appliances Key Features Two-way communication, Abstract Descriptions, Multiple Platforms, Automatic Interface Generation © 2020 - Brad Myers 48
Properties of PUC Language l State variables & commands l Each can have multiple labels l l Typed variables l l l Useful when not enough room Base types: Boolean, string, enumerated, integers, fixed-point, floating-point, etc. Optional labels for values Hierarchical Structure l Groups © 2020 - Brad Myers 49
Dependency Information l l Crucial for high-quality interfaces Expressed as <active-if> clauses l Operations: l l Combined Logically l l Equals, Less-Than, Greater-Than AND, OR Used for: l l l Dynamic graying out Layout Widget selection © 2020 - Brad Myers 50
Generating Consistent UIs l l l Personally consistent Reduce learning time Add unique functions © 2020 - Brad Myers 51
Generating Combined UIs l l l For multiple appliances, such as home theaters Specify content flow Combined controls © 2020 - Brad Myers 52
Summative Study l l l Compared PUC to manufacturer’s interfaces for HP and Canon printer/fax/copiers PUC twice as fast, 1/3 the errors Consistent: another factor of 2 faster © 2020 - Brad Myers 53
Model-based systems advantages/disadvantages l l l + Separate specification of UI from content + Automatic reformatting, retargeting for different platforms, customization to users + May allow programmers (non-experts) to write specification and have a good UI automatically created l l l But this didn’t really work out – Result is often unpredictable – Often UI can be worse UI than hand-drawn – Sometimes model is larger than the code it would replace – Model often in a different language that must be learned © 2020 - Brad Myers 54
- Slides: 54