A Virtual Character based Italian Sign Language Dictionary

A Virtual Character based Italian Sign Language Dictionary Instructor: Dr. Nadeem Ahmad Chauhdry Dual Ph. D (Politecnico di Torino, Italy & University of Potsdam, Germany) HOD, CS & IT Department The University of Lahore Defence Road Campus, Lahore. Email: nadeem. ahmad@cs. uol. edu. pk https: //sites. google. com/site/pctresearchgroup/

Outline • • Introduction ATLAS Project ATLAS corpus and lexicon The LIS Dictionary Architecture The ATLAS Visualization system Web and Mobile Application Conclusion and Future Work

Introduction • • • Sign language is a gestural visual language with signs as lexical units instead of words. Sign languages have their own grammars, which are quite different from oral languages. [1] Sign Language is not a universal language; each country has its own Sign Language • • In some countries more than one Sign Language • 1. 2. American Sign Language (ASL) British Sign Language (BSL) Italian Sign Language (LIS) In Switzerland, Swiss-German, Swiss-French and Swiss. Italian Sign Language(s) are used. [2] Liddell, S. K. Grammar, gesture and meaning in American Sign Language, Cambridge Univ Pr (2003) Timmermans, N. The status of sign languages in Europe, Council of Europe, 2005

Introduction • We proposed Platform Independent Lingua dei Segni Italiana (LIS) Dictionary to support learning of Sign Language and written language • LIS Dictionary provides full set of lexemes and videos data set of Italian sign language with an online interface attached to the Multi. Word. Net. • By linking LIS lemmas to the corresponding closest concepts in the Multi. Word. Net lexical database enhances the possibility to understand the new words from existing signs.

ATLAS Project • • • Targets automatic translation of written contents Defines a web platform and a service Requires the collection of LIS resources Output provided by means of a virtual character Provides several software applications • • • Translator On- line dictionaries An editor for the manual translation from Italian to LIS • The signed content is provided on several platforms • • • Digital Terrestrial Television Word Wide Web Mobile

ATLAS Partners • The project collaborates with companies and academic institutions selected according to their characteristics of excellence in specific areas • Polytechnic of Turin, Italy • University of Illinois at Chicago • Italian RAI Sp. A • Microsoft Innovation Center • BEPS Engineering • Lumiq Studios Ltd • Virtual Reality and Multimedia Park and some others

ATLAS Objective • The main objective of ATLAS project is to develop tools that allow naturally challenged people to follow and understand the contents of television, multimedia and entertainment. • More specifically, ATLAS aims at developing a virtual interpreter, which automatically translates Italian into LIS.

Usage • To load LIS visualization service on the cell phone or PDA for the deaf traveller to understand announcements on Railway stations and airports.

ATLAS Work Flow TEXT VIDEO VIRTUAL ACTOR COMMANDS SOURCES Teletext Subtitles Audio/Video TEXT Transformations Virtual Actor Commands AEWLIS Sentences Transformations ATLAS CORE Transformations

ATLAS corpus and lexicon • • The Platform Independent LIS Dictionary is part of the Automatic Translation into Sign Languages (ATLAS) Project. As a significant amount of parallel data is not available yet, we have started building a new corpus from scratch. Our corpus contain weather forecast bulletins broadcasted by Italian National TV Channel (RAI TV) for a total of about 15 K Italian running words and about 1. 5 hours of LIS videos We have adopted mixed approach for the generation of signs, using an animation language to compose and parameterize pre-captured and hand animated signs.

ATLAS corpus and lexicon • • • Our Sign Language dictionary provides more than three thousand sign videos. These sign videos are set of Avatar animations which are rendered through a graphic engine. Each sign is linked to a corresponding Italian lemma existing in the Multiword. Net. About twenty seven hundred video animations were created using standard Lingua dei Segni Italiana dictionary. More than four hundred signs are custom created by a team of expert signers and deaf.

The LIS Dictionary Architecture • • The Sign Language Dictionary takes Italian lemmas as input and provides as output their representation in the Italian Sign Language. The process for the generation of avatar based signs in the LIS Dictionary is as follows.

The ATLAS Visualization system The module used for the signs generation is system that generates real-time animations, display them in a player and export them as picture frames. It takes as input the symbolic form of LIS called ATLAS Written Extended LIS (AEWLIS) [3] , a sequence of lemmata and a description of the meaning, syntactic number and other semantic, pragmatic and articulatory information. Signs are described by a default animation (stored in a repository, the “signary”), which has to be parameterized with respect to the context given by the specific sentence. • • • 3. N. Bertoldi, G. Tiotto, P. Prinetto, E. Piccolo, F. Nunnari, V. Lombardo, R. Damiano, L. Lesmo, and A. D. Principe, “On the creation and annotation of a large-scale italian-Lis parallel corpus, ” in 4 th Workshop on the Representation and Processing of Sign Languages: Corpora and Sign language Technologies (CSLT 2010), 2010.

The LIS Dictionary Architecture

The Italian Sign Language Dictionary • • • Provides more than 3 k signs Signs stored as set of animations Provides link with Italian written lemmas Meanings in written Italian Links with Multi. Word. Net

Italian Sign Language Dictionary

Italian Sign Language Dictionary Coverage Italian Lemmas Grammatical Category Italian Synsets Multi. Word. Net LIS Dictionary % Nouns 49, 003 9321 19. 0 28, 429 3910 13. 8 Verbs 9861 4534 46. 0 4975 1814 36. 5 Adjectives 6443 2417 37. 5 3898 998 25. 6 Adverbs 2243 942 42. 0 1239 342 27. 6 67, 550 17, 214 25. 5 38, 541 7064 18. 3 Total

Sign Language Mobile Dictionary

Conclusion and Future Work • • • We discussed a novel platform independent Italian Sign Language dictionary, which is capable to run on several devices and is linked with Multi. Word. Net. It is available in the form Web application online and as a mobile application for Windows Phone 7, i. Phone and Android. Our dictionary contains a large video data set and its structure is flexible enough to include further signs in future. We plan to develop web applications that can help in learning of new words. The objective is to propose web based interface as a tool to enhance deaf learning of written Italian Language in schools.

Conclusion and Future Work • • • Different scenarios of LIS vocabulary will be created to check the coherence and understanding and hence get feedback on lexical resources by deaf users with help of some expert signers. The subjects would be set of deaf users, normal people having different literacy level and aptitude. The community feedback will be helpful to check the originality of the material and comprehensiveness of lexical resources. Since Multi. Word. Net has multilingual lexicons, such as Spanish, German and Portuguese, it could be helpful for written multi-language learning. ATLAS LIS WEB APPS

Questions?