Multimodal research in Linkping Arne Jnsson NLPLABHCSIDA Arne
Multimodal research in Linköping Arne Jönsson NLPLAB/HCS/IDA Arne Jönsson
NLPLAB Upper row: Pernilla, Sonia, Lena, Annika, Micke, Magnus Bottom row: Håkan, Lars A, Robert, Mustapha, Arne, Lise, Anders, Lars D Missing: Nils, Bertil, Pontus, Gen Arne Jönsson 2
Current research • (Multimodal) information access - Natural language interaction - Text and document analysis • Usability of multimodal interfaces • Open-Source and development methods Arne Jönsson 3
Dialogue system Interpreter Structured Information Dialogue manager Generator Arne Jönsson 4
Ex TV programme guide • • • Search for TV programs Mediaterminal Structured database Spoken input Developed from framework and method for iterative development • With Nokia Home Communication Arne Jönsson 5
Demo Arne Jönsson 6
Text and document analysis Unstructured documents Q&A Language analysis Structured Information MT Arne Jönsson 11
Ex. Ask. RSV • Information extraction from text document • Simple Q&A-system • With Ida Infront Arne Jönsson 12
Ex. taxation brochure 24 pages unstructured text Arne Jönsson 13
Domain ontology in ASK-RSV • Taxonomy of a limited set of objects, events, properties and relations in the tax domain • Only IS-A and synonymy relations THING OBJECT PAPER TAX-RETURN Arne Jönsson DATE EVENT TAXATION ASSET DEBT INCOME ACTOR TAX EVENT PEOPLE TRANSACTION ORG. COMP. PUBL 14
Combined system Unstructured documents Structured Information Dialogue system Language analysis Lexicon Arne Jönsson Grammar Ontology 18
Shared knowledge sources • Ontology, lexicon and grammar - Different demands from IE and DS • Design • Development - Corpus analysis, user - Background system, database • Automatic update Arne Jönsson 19
Bird. Quest • Bird encyclopaedia - Tagged, templates • Corpus with user questions - Analysed • Based on dialogue systems framework Arne Jönsson 20
Ötraf • Time tables from local bus company • Multimodal system - Dialogue - Handling of spatial and temporal expressions Arne Jönsson 26
Studies of multimodal interaction • Wizard of Oz-simulation • Feedback study - User’s experience - Interviews, questionnaires - Various dimensions: control, cooperation, ease of use, efficiency, anxiety, hability etc - Three conditions - No spoken feedback - Limited spoken feedback - Full spoken feedback - 30 subjects Arne Jönsson 28
Results • All or nothing • More like a human in the full feedback condition - Before the study subjects wanted to have a timetable for the task, after they preferred a human • No difference in experienced control and cooperation • The speech synthesis better in full feedback mode - Same voice Arne Jönsson 29
Open source • Share research results between research groups - Stimulate exchange and co-operation • Language technology readily available for industry - Increased use and insights - The companies do their own prototype development • Robust code • nlp. Farm http: //sourceforge. net/projects/nlpfarm/ Arne Jönsson 30
http: //sourceforge. net/projects/nlpfarm/ Arne Jönsson 31
- Slides: 24