A Methodology for Developing Multimodal User Interfaces of

A Methodology for Developing Multimodal User Interfaces of Information Systems Adrian Stanciulescu Université catholique de Louvain (UCL) Louvain School of Management (LSM) Information Systems Unit (ISYS) Belgian Laboratory of Computer-Human Interaction (BCHI) http: //www. isys. ucl. ac. be/bchi 1

Outline • • 2 Context & Demonstration State of the art Thesis statement & Focus Methodology: – Models & Language – Method – Tool support Validation – External – Internal Conclusion Future work Ph. D Public Defense, 25 June 2008

Context • 5 senses: sight, hearing, touch, smell, taste • Different user-system interaction types graphical vocal • Prevailing interaction: graphical • Shortcomings: – restrictive (graphical) – less flexible than human-tohuman interaction – difficult to use in mobile environments with new emerging devices 3 Ph. D Public Defense, 25 June 2008 tactile olfactory gustatory

Multimodal systems • New interaction paradigm: – Combines two or more individual interaction – Generic benefits: naturalness, flexibility, robustness, error avoidance • Specific benefits: – wide range of users (disabled persons) – environment (mobile) – platform (heterogenous devices) 4 Ph. D Public Defense, 25 June 2008

Input: graphical (A) Input: vocal (A) Input: multimodal (E) Demonstration 5 Ph. D Public Defense, 25 June 2008

Outline • • 6 Context State of the art Thesis statement & Focus Methodology: – Models & Language – Method – Tool support Validation – External – Internal Conclusion Future work Ph. D Public Defense, 25 June 2008

State of the art: Concerns of multimodal UIs (1) define Thesis statement: 1. 2. 3. Models Method Tool support Methodology (2) analyse Real world (5) guide State of the Art (3) identify Shortcomings (6) validate 12 Requirements Chapter 1 Chapter 2 (4) elicit • Concerns – lack of: – structuring framework for MM UI development, support for MM I/O, separation of modalities, combination of modalities, modalityindependent models, extendibility for new modalities, method extendibility 7 Ph. D Public Defense, 25 June 2008

State of the art: Shortcomings • Survey of 8 languages and 11 development tools • Multimodal Teresa [Paterno, 2004] – Design space with unexplicit options and limited alternatives – Mix of modality dependent and independent elements – CARE: no support for R and C – Pre-computed and hard-coded transformations • 8 Shortcomings: lack of – MM application deployment, fast interaction, error recovery, platform mobility, usable MM UIs, robust systems, device effectiveness, MM experience Ph. D Public Defense, 25 June 2008

State of art: Requirements 9 • Modeling: 1. Support for MM I/O 2. Separation of modalities 3. Support for CARE properties in I/O 4. Ability of modeling a UI independent of any modality 5. Extendibility to new modalities 6. Ontology homogeneity 7. Human readability • Method: 8. Approach based on design space 9. Method explicitness 10. Method extendibility • Tool: 11. Machine processability of involved models 12. Support for tool interoperability Ph. D Public Defense, 25 June 2008

Outline • • 10 Context State of the art Thesis statement & Focus Methodology: – Models & Language – Method – Tool support Validation – External – Internal Conclusion Future work Ph. D Public Defense, 25 June 2008

Thesis statement & Focus Define a design space-based method that is supported by model-to-model colored transformations in order to obtain multimodal user interfaces of information systems from a task and a domain models. • Working hypotheses: – Model-based approach: a set of models specifying different abstractions of the final UI – Semi-automated transformational approach: rules manually selected and automatically applied • Focus: – – – 11 Information systems Predefined and constant contexts of use Graphical, vocal and MM interactions MM UIs familiar to vast majority of users and available on most platforms Target audience: HCI reasearch comunity, professionals in the field of MM interaction Ph. D Public Defense, 25 June 2008

Outline • • 12 Context State of the art Thesis statement & Focus Methodology: – Models & Language – Method – Tool support Validation – External – Internal Conclusion Future work Ph. D Public Defense, 25 June 2008

Models & Language • Cameleon Reference Framework: – – – Task Model Domain Model Abstract User Interface Model Concrete User Interface Model Mapping Model Transformation Model inter-model relationship • • conceptualizes extended modality toolkit UML class independent CTT diagram transformation + extensions mappings rules vocabulary vocablary + extensions ++extensions Task & Concepts Abstract User Interface Concrete User Interface Final User Interface • Usi. XML v 1. 8: – integrates our extensions – 3 dimensions of linguistics: semantics, syntax, stylistics 13 Ph. D Public Defense, 25 June 2008

Semantics Vocal Concrete expansion • Vocal Containers: – – vocal. Group vocal. Form vocal. Menu vocal. Confirmation • Relationships: – – vocal. Transition vocal. Adjacency vocal. Containment synchronization • Event. Types: – error/help/no. Input/ no. Match 14 Ph. D Public Defense, 25 June 2008 • Vocal Individual Components: – vocal. Output: • vocal. Feedback • vocal. Prompt • vocal. Menu. Item • audio – vocal. Input – grammar/part/item – vocal. Navigation/submit – connect – record – vocal. Var/set. Var/reset. Var – if/elseif – break/exit

Syntax Usi. XML language • USer Interface e. Xtensible Markup Language • Particular motivations: – Structured according to Cameleon Reference Framework – Ensures the independence of modality (Req. 4. Ability to model modalityindependent UI ) – Logically structures the transformation steps (Req. 9. Method explicitness) – Flexibility for adding/deleting sub-steps (Req. 10. Method extendibility) – Transformational approach with transformations expressed in the same formalism – Supported by tools processing its format (Req. 11. Machine processability of involved models) 15 Ph. D Public Defense, 25 June 2008

Language: Stylistics A dashed rectangle to suggest the containment purposes and a callout symbol to indicate the vocal character. vocal. Group A dashed rectagle to suggest the containment intentions, a user and a system icon next to a callout symbol. vocal. Form As it is a container that enables to select among different options and by analogy with the menu provided by the graphical toolkits the representation is composed of a blue dashed oval to suggest the containment purpouses, an overlaying callout symbol to indicate the vocal aspects and yellow ovals to indicate the vocal options. vocal. Menu A system icon and a callout symbol containing a question sign. vocal. Prompt ? A user icon next to a callout symbol and a system icon. vocal. Input 16 Ph. D Public Defense, 25 June 2008

Outline • • 17 Context State of the art Thesis statement & Focus Methodology: – Models & Language – Method – Tool support Validation – External – Internal Conclusion Future work Ph. D Public Defense, 25 June 2008

Design option 1 Design space Value 14 Design option n Design option 2 Value 13 Value n 3 Values 12 Value 22 Value n 2 Value 11 Value n 1 Value 21 Design option 7 Design option 3 Value 72 Value 71 Value 32 Value 41 Value 61 Value 51 Value 42 Value 62 Value 52 Design option 4 Value 63 Design option 6 18 Ph. D Public Defense, 25 June 2008 Design option 5

Design space (cont’d) • Advantages – Clarifies the development process: structured, less design effort, consistent results – Every piece of development is reflected in a design option – Abstractions covered by a software that do not require any further interpretation effort • Rationale – Intrinsic: descriptive, comparative, generative – Extrinsic: • implementation and tool-independent: useful for any designer • explicit guidance: flexibility to set the CARE properties • simplifies the design decision: reduces the complexity of the design space 19 Ph. D Public Defense, 25 June 2008

Design space (cont’d) A structured combination of modality-independent design options having assigned a finite set of design option values that support the stakeholder’s design decisions during the development life cycle of multimodal user interfaces Definition, Rationale, Values Sub-task guidance Support for default value and unit Answer cardinality Confirmation answer Answer order Prompt Immediate feedback Input Design options for UIs Guidance Simple output Sub-task triggering 20 Sub-task presentation Sub-task navigation Ph. D Public Defense, 25 June 2008 Navigation type Control type Navigation and control type

Sub-task presentation • Specifies the way in which the system is presenting the sub-tasks to the user. Sub-task presentation Combined separated One at once extended task list 21 reduced task list Many at once tabbed list Ph. D Public Defense, 25 June 2008 single expansion list multiple expansion list All at once separated list grouped list bulleted list ordered list

Sub-task 2 “Which are my options? ” Sub-task 1 Sub-task 2 Sub-task 3 Audio ? Sub-task 1 Sub-task 2 Sub-task 3 Sub-task 2 ? Audio (beep) ? ? ? “Sub-task 2” ? “Sub-task 3” “Sub-task 1” Audio (“ 2 ”) ? Audio (beep) “Sub-task 3” Audio “Sub-task 1” Audio (“ 1 ”) Audio (beep) “Sub-task 2” Audio Sub-task 1 Sub-task 3 “Sub-task 1” “Sub-task 2” Audio(“ 3 ”) ? “Sub-task 3” Sub-task 1 Sub-task 3 Sub-task 2 Sub-task 3 separated extended task list reduced task list tabbed list Sub-task 1 Sub-task 2 22 Sub-task 3 Defense, 25 June 2008 Ph. D Public Sub-task 1 Sub-task 2 Sub-task 3 single expansion list multiple expansion list separated list grouped list bulleted list Concretization in vocal objects Sub-task 1 “Select an option” ordered list Concretizatioin in graphical objects ? ?

Design options for multimodal text field • • Prompting: multimodal (R) Input: multimodal (E) Immediate feedback: multimodal (R) Guidance: – Input: iconic (A) – Immediate feedback: iconic (A) 1 Please specify your name Prompt: multimodal (vocal +graphical) Guidance for feedback: iconic Juan 2 Gonzale z Juan Gonzalez Input: multimodal (vocal +graphical) 23 Ph. D Public Defense, 25 June 2008 Guidance for input: iconic 3 Your answer was: Juan Gonzalez Feedback: multimodal (graphical + vocal)

Design Rationale Approach • Supports different alternative solutions for a design issue by providing a justification and enabling its evaluation • Advantages: – detects the concistency and completeness issues in the early design phases – clarifies the reasons for a particular design solution and enables to argue it • Example: Dream/Team ([Laca 05 ]) – Team notation – Dream tool 24 Ph. D Public Defense, 25 June 2008

Colored ontology • Existing shortcomings: – repeated common parts – huge set of rules • Contributions: – colored graph (rules) + operations • Benefits: – reduced number of rules (250 -> 130) – high scalability 25 Ph. D Public Defense, 25 June 2008

Rule for GUI MMUI Rule for VUI Operations over colored rules: • merging NAC • splitting is. Reified. By LHS abstract. Containment is. Composed. Of is. Reified. By RHS abstract. Containment is. Reified. By is. Composed. Of graphical. Containment 26 is. Composed. Of Ph. D Public Defense, 25 June 2008 is. Reified. By vocal. Containment is. Composed. Of vocal. Containments

Transformation rule catalog • Complete and systematic arrangement of rules (130) into a structured catalog: – Transformation rules supporting design options – Additional transformation rules 27 Ph. D Public Defense, 25 June 2008

Sub-task 1 Sub-task 2 Sub-task 3 “Sub-task 1” Audio (beep) ? Audio “Which are my options? ” ? Sub-task 1 Sub-task 2 Sub-task 3 ? ? ? “Sub-task 1” Audio (“ 2 ”) ? “Sub-task 2” Audio (beep) “Sub-task 3” Audio Sub-task 2 Audio (“ 1 ”) Audio (beep) “Sub-task 2” Audio Sub-task 1 Sub-task 3 “Select an option” “Sub-task 2” Audio(“ 3 ”) ? “Sub-task 3” Sub-task 1 Sub-task 2 Sub-task 3 R 2 separated R 5, R 6 R 7, R 8 R 9, R 10 extended task list reduced task list R 5, R 6 Sub-task 1 Sub-task 2 Sub-task 3 R 11, R 12 tabbed list R 7, R 8 R 2 Sub-task 1 single expansion list R 15, R 16 multiple expansion list R 11, R 12 R 9, R 10 R 17, R 18 separated list grouped list R 15, R 16 Sub-task 2 28 Sub-task 3 Defense, 25 June 2008 Ph. D Public R 21, R 22 bulleted list ordered list R 19, R 20 R 21, R 22 R 13, R 14 R 17, R 18 Sub-task 1 R 19, R 20 Concretizatioin in graphical objects Sub-task 3 Concretization in vocal objects ? ?

Steps and sub-steps of the transformational approach • Step 1: Task & Domain Models • Step 2: From Task & Domain to AUI • Step 3: From AUI to CUI • Step 4: From CUI to FUI 29 Ph. D Public Defense, 25 June 2008 Øsub-task presentation Ønavigation type Øcontrol type

Outline • • 30 Context State of the art Thesis statement & Focus Methodology: – Models & Language – Method – Tool support Validation – External – Internal Conclusion Future work Ph. D Public Defense, 25 June 2008

1 Step 1 Ideal. XML tool Tool support Multimodali. XML Task & Domain Models 2 Transformi. XML tool Step 2 Abstract UI Model 2 Transformi. XML tool Step 3 Concrete UI Model (vocal) Concrete UI Model (graphical) 3 4 Grafi. XML tool Voice. XML Generator tool XHTML code Voice. XML code Concrete UI Model (multimodal) 5 XHTML+Voice Generator tool Step 4 31 Web Ph. Dbrowser Public Defense, 25 June 2008 Voice. XML Browser XHTML+Voice code Multimodal Browser

Input: Multimodal Graphical Vocal (A) 32 Ph. D Public Defense, 25 June 2008

Outline • • 33 Context State of the art Thesis statement & Focus Methodology: – Models & Language – Method – Tool support Validation – External – Internal Conclusion Future work Ph. D Public Defense, 25 June 2008

External validation: case studies • Feasability of the approach - 3 case studies ( ≠ levels of complexity and ≠ design options): – 2 web-form: virtual polling application (low complexity), car rental application (medium complexity) – 1 non web-form: map browsing application Task and Domain 34 Abstract User Interface Ph. D Public Defense, 25 June 2008 Concrete User Interface (graphical) Final User Interface (graphical) Concrete User Interface (vocal) Final User Interface (vocal) Concrete User Interface (multimodal) Final User Interface (multimodal)

Validation: Empirical assessment • Assesment of 3 applications: – Session I (2 web-forms), Session II (1 non web-form) – Relative usability level of interaction modalities • Input design option: G, V, MM (equivalence) • Evaluation measures: – – – 35 task completion time error rate interaction preference task procentage completion learning time n° of mouse clicks Ph. D Public Defense, 25 June 2008

Results: Task completion mean time • 36 Web-form: – Efficiency (G > MM > V) – Experienced faster then non-experienced • experience has a significant influence for G (t-Test analysis p=0. 0339) • higher dispersion for V and MM Ph. D Public Defense, 25 June 2008

Results: Error rate • • 37 Higher error rate for V and MM (synchronization and pronunciation errors ) Correlation (Pearson correlation analysis): – task completion time (web form): V(Pearson= 0. 75), true MM (Pearson= 0. 73) – number of mouse clicks (web form): V(Pearson= 0. 77), true MM (Pearson= 0. 63) – task percentage completion (non-web form): V(Pearson= -0. 72), MM (Pearson= 0. 48) Ph. D Public Defense, 25 June 2008

Results: Interaction modality preference • Satisfaction: strong preference for MM all commands conveyed MM [Oviatt, 1997] – web-form: G ≈ MM > V – non web-form: G > V > MM – mix of monomodal (radio buttons, list box) and multimodal (check boxes with numerous items / combo boxes with items to select at the end) [Oviatt, 1999] 38 Ph. D Public Defense, 25 June 2008

Outline • • 39 Context State of the art Thesis statement & Focus Methodology: – Models & Language – Method – Tool support Validation – External – Internal Conclusion Future work Ph. D Public Defense, 25 June 2008

Internal validation: Requirements level of support Req 1. Support for MM I/O Modeling Req 2. Separation of modalities Req 3. Support for CARE properties in I/O Req 4. Ability of modeling a UI independent of any modality Req 5. Extendibility to new modalities Req 6. Ontology homogeneity Method Req 7. Human readability Req 8. Approach based on design space Req 9. Method explicitness Tool Req 10. Method extendibility 40 Req 11. Machine processability of involved models Req 12. Defense, Support 25 for tool interoperability Ph. D Public June 2008

Outline • • • Context State of the art Thesis statement & Focus Methodology: – Models & Language – Method – Tool support Validation – External – Internal • • 41 Conclusion Future work Ph. D Public Defense, 25 June 2008

Conclusion Methodology barriers • Ontological – Lack of model expressiveness • Method – Difficulty in defining an orthogonal design space – Difficulty in finding an appropriate level of generalization when defining rules – Difficulty in managing large sets of transformation rules: • High reusability • Low reusability • Tool – Low interoperability: high number of software modules 42 Ph. D Public Defense, 25 June 2008

Conclusion (cont’d): Main Contribution • Ontological contribution – Expanded task model to support the requirements of MM UIs – Expanded vocal ontology – Multimodal instruction structure – Synchronization between the modalities – Formalization according to Usi. XML syntax – Stylistics of vocal ontology 43 Ph. D Public Defense, 25 June 2008

Conclusion (cont’d): Main Contribution • Method contribution – Design space • 16 design options: low treshold, high ceiling, wide walls – Expanded model-to-model transformation • colored concepts → colored graph → colored rules • operations over colored transformation rules – Transformation rule catalog: identify rules supporting the design options – 4 transformational development steps: identify the involved design options – Introduction of a new methodogical sub-step: synchronization of CICs 44 Ph. D Public Defense, 25 June 2008

Conclusion (cont’d): Main Contribution • Tool contribution – 5 interoperable software modules – Address specific steps of the development process – Proof of concept reasons 45 Ph. D Public Defense, 25 June 2008

Outline • • 46 Context State of the art Thesis statement & Focus Methodology: – Models & Language – Method – Tool support Validation – External – Internal Conclusion Future work Ph. D Public Defense, 25 June 2008

Future work • Extend the methodology to support development of context-aware systems: dynamic migration from one modality to another based on a set of rules • Design space improvement: reduce dependent dimensions, introduce new dimensions and new values • Knowledge base of inference rules: recommandation of suitable design options • New interaction modalities: – ontological extension – method extension – tool support extension 47 Ph. D Public Defense, 25 June 2008

Future work (cont’d) • Design space-based tool 48 Ph. D Public Defense, 25 June 2008

Thank you ! Questions ? 49 Ph. D Public Defense, 25 June 2008