ISLE and NITE Laila Dybkjr NISLab Odense Denmark
ISLE and NITE Laila Dybkjær NISLab, Odense, Denmark MUMIN meeting, Helsinki, 15 -16 November 2002
This presentation ISLE NITE • Background • Approach • Objectives • NIMM data resources • Software strands • NIMM annotation schemes • NWB • NIMM annotation tools Conclusions and ongoing work MUMIN meeting, Helsinki, 15 -16 November 2002
The ISLE project • International Standards for Language Engineering (ISLE) 1. 1. 2000 – 31. 12. 2002 • Successor to Eagles I and II • Three working groups: lexicons, evaluation, NIMM • NIMM focuses on data resources, coding schemes, annotation tools and meta-data • US NIMM loosely connected with EU NIMM • isle. nis. sdu. dk MUMIN meeting, Helsinki, 15 -16 November 2002
European ISLE NIMM participants • NISLab, Odense, Denmark • IMS, Stuttgart, Germany • CNRS-LIMSI, Paris, France • ILC, Pisa, Italy • UROME, Rome, Italy • DFE, Barcelona, Spain • MPI, Nijmegen, The Netherlands • ELRA, Paris, France • HCRC, Edinburgh, UK • DFKI, Saarbrücken, Germany MUMIN meeting, Helsinki, 15 -16 November 2002
Approach • Selection criteria • Common description template • Interaction with, and verification by, developers whenever possible • For data resources and coding schemes division into facial expression (plus speech) and gesture (plus speech) MUMIN meeting, Helsinki, 15 -16 November 2002
Data resources (1) • Information on 64 data resources world-wide • Survey of market and user needs done by ELRA • 28 filled questionnaires collected at Dagstuhl workshop • Resources primarily found via web and via proceedings MUMIN meeting, Helsinki, 15 -16 November 2002
Data resources (2) 36 facial and 28 gesture resources: • Dynamic facial resources - with audio - without audio • Static facial resources • Lesser known/used facial resources • Gesture resources • Lesser know/used gesture resources MUMIN meeting, Helsinki, 15 -16 November 2002
Data resources (3) • Static images: – image analysis, specialised databases, special conditions (light, size, face orientation. . ) • Dynamic video (mostly with audio): – e. g. lip movements during speech, audio-visual speech recognition, correlation between speech and gesture MUMIN meeting, Helsinki, 15 -16 November 2002
Needs and purposes (1) The collected data resources reflect among others the following needs and purposes: • Automatic analysis and recognition of facial expressions, including lip movements • Audio-visual speech recognition • Study of emotions, communicative facial expressions, phonetics, multimodal behaviour, etc. MUMIN meeting, Helsinki, 15 -16 November 2002
Needs and purposes (2) • Creation of synthetic graphical interface characters, including, e. g. , talking heads • Automatic person identification • Training of speech, gesture and emotion recognisers • Multimodal system specification and development MUMIN meeting, Helsinki, 15 -16 November 2002
Market study • Questionnaire sent to more than 150 people • 25 answers received • Questions on – types of data resources needed/used/offered – kinds of tasks for which data resources are well-suited – areas in which data resources are being used MUMIN meeting, Helsinki, 15 -16 November 2002
Types of data resources • Audio most popular (84%) • Video (52%) • Image (28%) • Advantage if a resource is annotated • Many users produce their own resources MUMIN meeting, Helsinki, 15 -16 November 2002
Tasks resources are used for Six general categories in questionnaire with several possibilities subsumed • Authentication: Speech verification • Recognition: Speech recognition, face recognition • Analysis: Speech/lips correlation • Synthesis: Multimedia development • Control: Voice control • Other: Information retrieval MUMIN meeting, Helsinki, 15 -16 November 2002
Application areas Possibilities listed in questionnaire • Research (21) • Information systems development (14) • Web applications development (10) • Education/training (9) • Edutainment (6) MUMIN meeting, Helsinki, 15 -16 November 2002
Coding schemes Information on 21 coding schemes • 7 facial • 14 gesture Nearly all schemes meant for video markup A couple of them can also be used for static image markup Schemes for single modality and modality combination markup MUMIN meeting, Helsinki, 15 -16 November 2002
Purposes and practices Many coding schemes tailored to particular task and only used by creator or creator’s site Few coding schemes fairly general and frequently used MUMIN meeting, Helsinki, 15 -16 November 2002
Facial coding schemes • MPEG-4 is considered a standard • FACS used by many (not good for lip movements) • Toon. Face used by some (2 D caricature) MUMIN meeting, Helsinki, 15 -16 November 2002
Gesture coding schemes • Very often combined with other modalities • Only sign language focuses on gesture alone • No real standardisation for gesture markup – Ham. No. Sys most frequently used among schemes looked at • Many gesture schemes created to support application development – used by one or a few people • Picture supported by answers to resource questionnaires at Dagstuhl workshop MUMIN meeting, Helsinki, 15 -16 November 2002
Data and schemes • Focus on accessible data resources and coding schemes • People mostly create own data and schemes • Often tailored to specific purpose, e. g. focus on a particular kind of interaction, modality combination or lighting condition • No strong guidance by best practice and standards and no particular focus on sharing data and schemes • On the other hand, ELRA and LDC exist MUMIN meeting, Helsinki, 15 -16 November 2002
Coding tools • 12 coding tool projects reviewed • All available tools tested with focus on functionality and interface • Most tools run on Windows, but java-based tools are cross-platform • XML is very frequently used for data representation • Some tools built for a very specific purpose while other tools aim at more general use MUMIN meeting, Helsinki, 15 -16 November 2002
Requirements for a general tool • Flexible and open architecture • Multi-level, cross-level, and multimodal annotation • Query, retrieval and extraction of data; tools for data analysis, including statistics • Support for viewing and listening to raw data • Adequate visual presentation of annotated data • Easy addition of schemes and definition of visualisations • Conversion tools (import, export) • Robust, stable, work in real time, and easy to use MUMIN meeting, Helsinki, 15 -16 November 2002
The NITE project • NITE = Natural Interactivity Tools Engineering • Tools for multilevel and cross-level natural interactivity corpus annotation and analysis • Based on NI data, coding scheme and tool overviews, and early tool specification done in the European ISLE NIMM Working Group • Duration: 1 April 2001 - 31 March 2003 • Eight partners: DFE, DFKI, HCRC, ILC, IMS, NISLab (coordinator), Noldus, TUB • Six countries: Denmark, Germany, Italy, Spain, The Netherlands, UK MUMIN meeting, Helsinki, 15 -16 November 2002
Objectives • Facilitate use and re-use of NI resources Theory • Develop standard framework for full NI corpus annotation at multiple levels and across modalities • Identify best practice NI coding schemes Practice • Build tools in support of standard framework, including some best practice coding schemes MUMIN meeting, Helsinki, 15 -16 November 2002
Software One overall requirements specification Three strands in tools development 1. NITE Workbench (NWB) for ordinary users OS 2 a. NITE XML Toolset (NXT) for XML and stylesheet writers. Based on MATE Workbench OS 2 b. Joint development of plug-ins with Anvil OS 3. Future product releases of The Noldus Observer will embody some NITE requirements • OS = Open source software MUMIN meeting, Helsinki, 15 -16 November 2002
NITE Workbench • For ordinary users who want an easy-to-use interface for annotating full natural interactive behaviour • Easy addition of new coding schemes through built-in pre-defined framework • Windows platform (at least initially), familiar interface • Database (DBMS) approach using Access • Easy query, information extraction and analysis • Multiple video windows • XML import-export MUMIN meeting, Helsinki, 15 -16 November 2002
MUMIN meeting, Helsinki, 15 -16 November 2002
MUMIN meeting, Helsinki, 15 -16 November 2002
MUMIN meeting, Helsinki, 15 -16 November 2002
Ongoing work in ISLE • Guidelines for the creation of NIMM data resources • Guidelines for the creation of NIMM annotation schemes • We are collecting information on ongoing standardisation activities • We would greatly appreciate any information you may have on such activities • We would also greatly appreciate input on draft versions of our work • Please contact: laila@nis. sdu. dk MUMIN meeting, Helsinki, 15 -16 November 2002
We need your help ISLE cannot update and extend the collected information If you have a data resource, coding scheme or coding tool you are willing to share and want others to know about GO TO isle. nis. sdu. dk AND FILL IN INFORMATION ABOUT YOUR CORPUS, CODING SCHEME, OR TOOL MUMIN meeting, Helsinki, 15 -16 November 2002
NITE Advisory Panel • For colleagues who want to follow, comment and receive NITE reports and software early on • Next NITE AP meeting in Stuttgart 25 -26 November 2002 • You are kindly invited to join the NITE AP • Send an email to kiil@nis. sdu. dk saying that you would like to join the NITE AP • See the list of the current NITE AP members at the NITE web site nite. nis. sdu. dk MUMIN meeting, Helsinki, 15 -16 November 2002
THANK YOU MUMIN meeting, Helsinki, 15 -16 November 2002
- Slides: 32