Welcome Technology Development for Indian Languages Presentation at

����� Welcome Technology Development for Indian Languages Presentation at LREC 2010 Conf , Malta Swaran Lata & Somnath Chandra Human Centred Computing Division Department of Information Technology slata@mit. gov. in, schandra@mit. gov. ind Presented by : Shyam S Agrawal Executive Director KIIT, Advisor CDAC TDIL 1

Technology Development for Indian Languages) TDIL) Programme ² Promotes Research & Development of Technology, Software Tools and Applications for Indian Languages ² Catalyzes proliferation of Language Technology products and solutions ² Promotes Standardization TDIL 2

Complexity for Language Technology Development ²Complexity: § Very challenging area for computer scientists due to voluminous, informal and ambiguous nature of human languages. § Involves interdisciplinary research in advanced and sophisticated computer processing involving Artificial Intelligence and Machine Learning in one hand ; linguistic knowledge for incorporating human communication techniques on the other hand. § Still in research stage in many areas despite huge efforts by academia and scientists in India as well as abroad ²More intense challenge for Indian Languages: § Large linguistic diversity with 22 officially recognized languages and 12 scripts. § One-language Many Scripts ; Many Languages – One Script § Specificity for each language and script is unique in nature and can not be easily replicated. § Difference in perceptions of usage among various user groups, e. g. State Governments , Academia and industry TDIL 3

Official Indian Languages & Scripts Sl. No. Language Script 1. Hindi Devanagari 2. Sanskrit Devanagari 3. Marathi Devanagari 4. Konkani Devanagari 5. Nepali Devanagari 6. Maithili Devanagari 7. Sindhi Devanagari 8. Bodo Devenagari 9. Dogri Devanagari, Sharda 10. Bengali 11. Assamese Bengali 12. Manipuri Bengali, Meitei (Mayak) 13. Gujarati 14. Kannada 15. Malayalam 16. Oriya 17. Punjabi Gurmukhi 18. Tamil 19. Telugu 20. Urdu Arabic 21. Santhali Ol-Chiki, Devanagai, 22. Kashmiri Arabic, Sharda TDIL 4

Genesis of Language Technology Development in India-Early Initiatives ² Pioneering Effort by DIT in collaboration with IIT Kanpur in 1983: ² Department of Electronics (Now DIT) entrusted a sponsored project to IIT Kanpur to build an integrated Devanagari Terminal (GIST). ² A standalone system with a computer keyboard was used for inputting the character in devanagari, a monitor for display and a Dot Matrix printer for printing and a serial communication for sending character to another terminal was developed. ² Developed Indian Script Code for Information Interchange (ISCII standard) ² C-DAC Pune adopted GIST technology to develop products and licensed it to manufacturers Technology Development for Indian Languages (TDIL) Programme started in the year 1991 as a separate entity. TDIL 5

Phases of Language Technology developments ² Seeding Phase : 1991 -1995 § TDIL programme established in the year 1991 § Some linguistic resources such as corpora developed § NLP training programme for Computer Scientists and linguists § Some stand-alone language learning tools have also been developed § Exploratory Work in the area of NLP ² Exploratory Phase : 1995 -2000 § Development of Proof –concept Machine Translation System for English to Indian Languages and Indian Languages (Angla-Bharti) to Indian Languages (Anusaraka) systems have been developed. § Laboratory model of font dependent Optical Character Recognition in Hindi § Text-to-Speech for Hindi TDIL 6

² Catch-up Phase : 2000 -2004 The TDIL programme gathered momentum by establishing 13 Resource Centres for Indian Languages Technology Solutions (RCILTS)and 10 Co. ILNet Centres. Resource Centres for Indian Languages Technology Solutions (RCILTS) § The objective was to proliferate this activity to a large number of institutions across the country with the specific mandate for a language or a group of languages. § Under this project, these centres have developed several important tools , linguistic resources and technologies for Indian language support § Many of these tools are now being modified and upgraded to be released in public domain under National Roll-Out Project. § Some of the important language technology tools and resources developed under Resource centres Project are: TDIL 7

§ Bi-lingual Dictionaries: between Indian Languages with over 30, 000 words [Resource Centres] § Spell-Checkers in Indian Languages [Resource Centres] § Ontology & Word-Net: 9000 syn-sets with morphological analyzer and front end for Hindi Word-net with 1100 lexical entries with X-window interface for Oriya. § Proof-of –concept technologies for Optical Character Recognition system (OCR) in other Indian languages § Proof-of-concept TTS in other Indian languages § In addition , several other tools , Operating Systems and resources have been developed under various sponsored projects. Some of the notables are: § INDIX-2 (Localized LINUX in 12 Indian languages) § Phrasal Dictionaries: in Tamil and Kananda [IIIT, Hyderabad] § Online Vishwa. Kosha: with 9162 topics [CDAC] § Parallel Corpora: One Million pages Parallel Corpora in 11 languages [CDAC] TDIL 8

COIL-Net Centres: ² The objective was to develop Localized Content in Hindi Speaking states for enhancement of IT proliferation ² E- content of approximately 16000 HTML & Dynamic pages in the domains of health, education, tourism and agri-business have been developed. Content on the eminent personalities, tourist places, classical work, and cultural heritage information on these regions have been developed. ² The developed content is uploaded on the internet at the website http: //tdil. mit. gov. in. ² National Train Enquiry website http: //www. trainenquiry. com TDIL localized in Hindi by CDAC. 9

Product Development and Proliferation Phase : 2005 -onwards § A ‘Roadmap for Language Technology Development in India’ was evolved-to formulate short-term & long-term mission plan and strategy for development of Language Technologies in India. § The Focus is to synergize development efforts and Develop deployable products § National Roll-Out Programme and Six Mission Mode Projects have been initiated to facilitate Speedy Development & Availability of the Language Technologies. TDIL 10

Proliferation of Indian Language Technology Products : National Roll-Out Plan Objectives of the initiative To facilitate Speedy Development & Availability of the Language Technologies. Broad contents of the CD • Common user’s Toolkit – Content Creation Tools, DTP, Office Automation, Code Converters • Productivity Tools – Spellchecker, Domain based Dictionaries, Transliteration. • Power user – OCR, Text to Speech, MAT, etc Distribution channel for the CD • Registered users of www. ildc. in web site of TDIL, DIT – through postal department. • IT magazines, publications, etc. • Schools, Government departments, etc. Software tools and fonts for 12 Indian languages namely Hindi, Tamil, Telugu, Assamese, Kannada, Malayalam, Marathi, Oriya, Punjabi and Urdu and Gujarati and Sanskrit languages have been released in public domain CDs containing 4 Indian Languages namely Bodo , Dogri , Maithili and Nepali languages are being released on Feb 21, 2009 – UNESCO International Mother Language Day. TDIL 11

Software tools and fonts CD contents 1 Language True Type Fonts with Keyboard Driver - more than 200 Supporting INSCRIPT, Typewriter, Phonetic Keyboard layouts Allows content creation in Indian languages using applications running under Microsoft windows 2 Language Multi-font Keyboard Engine for True Type Fonts Allows content creation in Indian languages using applications running under Microsoft windows in variety of font encoding. 3 Language Unicode Compliant Open Type Fonts - more than 200 Allows to render the Indian language Unicode data. 4 Unicode Compliant Keyboard Driver Supporting INSCRIPT, Typewriter, Phonetic Keyboard layouts. Allows Unicode complaint data inputting 5 Generic fonts and storage code converter Allows user to convert the existing data in different encoding to ISCII / UNICODE 6 Localized version of Bharateeya OO (Office Suite) This consists of word processor, presentation tool, spreadsheet & drawing tool 7 Fire fox browser Localized version of Fire fox browser TDIL 12

Software tools and fonts CD contents 8 Colombo - Email client for Windows and Linux Operating systems. Using this user can send / receive emails in Indian languages. The menus are also in local language. 9 GAIM - Multiprotocol Messenger. This enables the user to user various messenger clients for communications 10 Optical Character Recognition With the help of OCR one can scan the printed text matter and convert it into editable form for further processing. 11 Typing Tutor This application teaches the user to type in Indian languages. 12 Spellchecker Allows the end user to rectify spelling mistakes in the document 13 Dictionaries English to Indian language and vice versa dictionaries in general, administrative, technical domains. 14 Transliteration Tool Transliterates a given Indian language text into Roman & vice versa. Useful for user who is not familiar with the script. 15 Text to Speech system Readouts the text TDIL 13

TDIL 14

Product Development Efforts…. Mission Mode Projects Phase -I In the consortium mode 26 premier Institutes and R&D organizations are working together on six projects to develop the advanced technologies & applications. ² Development of English to Indian Languages Machine Translation (MT) System: § § 10 institutions are participating to build deployable MT System. Consortium Leader: CDAC, Pune Domains: Tourism and Health Six Languages pairs: English to Hindi/ Marathi/ Bengali/ Oriya/ Tamil/ Urdu. ² Development of English to Indian Languages Machine Translation (MT) System with Angla-Bharti Technology: 4 institutions are participating to build deployable MT System. Consortium Leader: IIT Kanpur Domains: Tourism and Health Six Languages pairs: English to Hindi/ Marathi/ Bengali/ Oriya/ Tamil/ Urdu. § § ² Development of Indian Language to Indian Language Machine Translation System: § 11 institutions are participating to build deployable Bi-directional MT System. § Consortium Leader: IIIT, Hyderabad § Domains: Tourism and Health § Nine Language pairs: Tamil-Hindi, Telugu-Hindi, Urdu-Hindi, Kannada-Hindi, Punjabi. Hindi, Marathi-Hindi, Bengali-Hindi, Tamil-Telugu, Malayalam-Tamil TDIL 15

Development Efforts…. Mission Mode Projects Phase -I ² Development of Robust Document Analysis & Recognition System for Indian Languages: § 11 institutions participating to build OCR System with improved accuracy, font and point-size independent recognition capability. § Consortium Leader: IIT, Delhi § 10 Scripts: Bengali, Devanagari, Malayalam, Gujarati, Telugu, Tamil, Oriya, Tibetan/Nepali, Gurmukhi, Kannada ² Development of On-line handwriting recognition system: § Seven institutions are participating to build On-Line Handwriting Recognition System. § Consortium Leader: IISc, Bangalore § 6 Scripts: Devanagari, Bengali, Tamil, Telugu, Kannada and Malayalam ² Development of Cross-lingual Information Access § 11 institutions participating to develop a portal where, a user will be able to give a query in one Indian Language and the user will be able to access documents available in (a) The language of the query and (b) Hindi (If the query language is not Hindi) and (c) English. § Consortium Leader: IIT, Bombay § Domains: Tourism and Health § Six Languages: Bengali, Hindi, Marathi, Punjabi, Tamil and Telugu. 16 TDIL

Status of the readiness of the consortium mode projects Sl No Name of the product /system 1 English to Indian Languages Machine Translation System (E-IL) Tourism α 2 English to Indian Languages Machine Translation System (E-IL) with Angla-Bharti approach Tourism α 3 Indian Language to Indian Language Machine Translation (IL-IL) Tourism α March 31, 2009 4 Cross-lingual Information Access (CLIA) Tourism α March 31, 2009 5 Printed Text OCR -- α March 31, 2009 6 On-line Handwriting recognition system (OHWR) -- α March 31, 2009 TDIL Language Pairs Marathi , Tamil and Bengali Domains Version Possible Date 17

Development Efforts…. Speech Processing Speech Corpora: • Annotated Speech Corpora of approximately 50 hours developed for Hindi, Marathi, Punjabi, Bengali, Assamese and Manipuri. [CDAC] • Speech Corpora for Tamil, Malayalam, Telugu and Kannada under development. [CDAC] Speech Recognition: Phonetic Engine for Speech recognition system for Hindi and Telugu languages are being developed [IIIT Hyderabad] Text-to-Speech Languages: (TTS) and Automatic Speech Recognition in Indian § Consortium Mode project for development of Text-to-Speech system for visually challenged persons in six Indian languages namely Hindi, Tamil , Telugu , Marathi , Malayalam and Bengali languages has been initiated. § Development for Automatic Speech Processing in Indian languages is also being initaited. TDIL 18

Development Efforts…. Sanskrit Computing : Consortium Mode project has been initiated for development of Sanskrit Computational tool kit and Sanskrit-Hindi Machine Translation System [ Univ. of Hyderabad] Corpora Consortium Mode project is being initiated for development of annotated corpora in 11 Indian languages. The project will evolve the standards for natural language processing TDIL 19

Development Efforts for North –Eastern Languages § Consortium Mode Projects to develop linguistic resources and basic information processing tool for North-Eastern languages namely Assamese, Bodo Manipuri and Nepali languages have been initiated. [ C-DAC Pune] § Consortium Mode project has also been initiated for development of Word-net in North-Eastern Languages [ IIT Bombay] § Speech Corpora and standardization of International Phonetic Alphabet (IPA) for Bodo language has been initiated [Univ. of Guwahati] TDIL 20

Standardization TDIL 21

ISCII – Indian Script Code for Information Interchange Since the 1970 s, efforts were made to evolve different codes for characters and symbols of the 10 Brahmi based Indian scripts due to their common phonetic structure. These efforts culminated in bringing out Indian standards for Indian Script Code for Information Interchange (ISCII) in December, 1991. The ISCII code standard specifies a 7 - bit code table which can be used in 7 or 8 -bit ISO compatible environment. It allows English and Indian script alphabets to be used simultaneously. TDIL 22

INSCRIPT Keyboard Layout § Standardized by Bureau of Indian Standards - 13194 : 1991 § Key placement is such a way that a user well versed with one language can type in another without efforts. § This is overlaid on the existing QWERTY keyboard. § Language selection is done with help of either Caps lock, scroll lock or Num lock key § Since it is based on phonetic nature of Indian languages it is very easy to learn. § Efforts have been initiated to incorporate the additional characters as per latest UNICODE 5. 1 standards in the modified layout TDIL 23

UNICODE § § § § § TDIL Unicode uses a 16 bit encoding that provides code point for more than 65000 characters (65536). Corresponds to ISO/IEC 10646 -1 Universal Multiple Octet Coded Character Set (UCS) In Sync with other International Standards such as W 3 C Unicode Standards assigns each character a unique numeric value and name. Encodes all of the characters used for the written languages of the world. Unicode is increasing being accepted as a standard for Information Interchange worldwide as most of the major IT Companies have declared their support for it Department of Information Technology is the voting member of the Unicode Consortium since the year 2000 to ensure the adequate representation of Indic scripts in the Unicode Standards. http: //tdil. mit. gov. in/pchangeuni. htm DIT finalized the changes in the Unicode Standard and majority of changes have been accepted and incorporated in UNICODE Standards version 5. 0. Initiatives have been taken to incorporate additional languages/ scripts and additional characters and symbols of Vedic Sanskrit in UNICODE. http: //tdil. mit. gov. in/prop_uni/Vedic. pdf 24

UNICODE. . Examples Indicates proposed characters/symbols/signs shape change in the existing standard Indicates the change in the annotation/explanation of that particular code point. Indicates proposed characters/symbols/signs addition in the existing standard TDIL 25

W 3 C § Project “Web Internationalization Initiative” has been initiated with the objective of adequate representation of Indic scripts in the Web Technology Standards being evolved by World Wide Web Consortium (W 3 C). § Initiative has been taken to incorporate key findings of WII projects in the W 3 C standards /guidelines TDIL 26

W 3 C - Large amount of works need to be done § In the phase-I of WII projects only few exploratory work has been carried out § W 3 C Internationalization has 115 recommendations covering web internationalization , XML, Cascaded Style Sheet and Speech Synthesis § These recommendations needs to carefully studied in the Indian Language perspective and specific recommendations need to projected to W 3 C. § A few specific Interest of them are Internationalization Tag Set (ITS) Version 1. 0 , Voice Extensible Mark-up Language (Voice. XML) 2. 1 , Web Content Accessibility Guidelines 1. 0 , Cascading Style Sheets (CSS 1) Level 1 Specification , Speech Synthesis Mark-up Language 1. 0 § Need for consultation with all stake holders such as academia , industry and various state governments. § Sensitization to industry and web service providers to adopt W 3 C standards. W 3 C India Office has been established at DIT under the aegis of the TDIL programme. TDIL 27

International Phonetic Alphabet (IPA) § Since phonetic representation of symbols is the required for present-day speech mark-up language like W 3 C Speech synthesis mark-up language (SSML), standardization of IPA symbols is necessary. § India being a multilingual country a standardized phonetic alphabet has to be developed for scientific study of phonetics and SSML for Indian languages. § The IPA standardization for all Indian language and acceptance of it by International Phonetic Association is thus required for development of speech technology and associated products. § Efforts initiated to standardize IPA symbols in Indian languages TDIL 28

Common Locale Data Repository § Common Locale Data Repository (CLDR) is an initiative of UNICODE consortium to develop locale data for World languages. § The Unicode CLDR provides key building blocks for software to support the world's languages. CLDR is by far the largest and most extensive standard repository of locale data. § This data is used by a wide spectrum of companies for their software internationalization and localization § Department of Information Technology has already become TC (Team Coordinator) to incorporate / modify Indian languages in CLDR. § Modifications/ Development of Common Locale data repository in Indian languages have been initiated in consultations with state governments and other stake holders. CLDR data for 6 Indian Languages have been incorporated in UNICODE CLDR. TDIL 29

§ § § TDIL Language Tags are being used in most of the multilingual applications such as web development, Multilingual Internet Data Exchange, Language Negotiation and web services. The nomenclatures of the Language Tags are being standardized under ISO 639 standard. The Language Tag Standard ISO 639 -x (x stands for different versions) are being used in many other international Standards and Best Practices such as IETF (Internet Engineering Task Force) RFC 4646, RFC 4647 and W 3 C web standards. They are also related to ISO 3166 (for region codes) and ISO 15924 (script codes). The present forms of ISO 639 -2, ISO 639 -3 and the futuristic ISO 639 -5 and ISO 639 -6 have many ambiguous entries for Indian languages, which need to be corrected urgently in order to prevent propagation of incorrect nomenclatures for Language sets. Modification / Additions of Language Tags in Indian languages have been taken up in consultations with the state governments and all stake holders. 30

Information Dissemination § TDIL portal: http: //tdil. mit. gov. in § ILDC Portal: http: //ildc. gov. in, http: //ildc. in On ILDC Portal, a user can: • Request for a Language Tools CD • Register on ILDC website • Provide Feedback and Access FAQ (Frequently asked Questions) • Free Downloads and Software for Indian Language Tools § TDIL Half-yearly Journal: Vishwa. Bharat@tdil: 16 Issues published; accessible through TDIL web-site. 31

Future Activities: All the on-going consortium mode projects, National Roll-Out Plan project and Specialized Manpower Development in Language Technology project would be continued. Phase-II of Consortium Mode Projects: Consortium Mode Projects –Phase –II in the areas of Machine Translation , Cross-lingual Information Access , Optical Character Recognition and OHWR. The systems developed in Phase-I would be improvised and expended for other domains Technology Development: (a) Speech Technology: Development of Automatic Speech Processing engines to be initiated for major Indian languages. (b) Basic Research : Basic Research in the areas of semantic Web technology would be initiated. Web Internationalization Initiative: The phase –II of Project “Web Internationalization Initiative (WII) ” to be initiated with the objective of adequate representation of Indic scripts in the Web Technology Standards being evolved by World Wide Web Consortium (W 3 C). 32 TDIL

Future Activities: Establishment of Data Centre at TDIL Setting up of Indian Language Data Centre , Language Technology Demonstration Facility at DIT , Up-gradation of ILDC and TDIL websites , Up-gradation of Language CDs and their distribution support would be undertaken. Web Internationalization Initiative: Phase-II of Web Internationalization Initiative (WII) programme would be initiated for finalization of Indian Language specific inputs / recommendations in W 3 C web technology standards. National Localization Research and Resource Centre (NLRRC) Seeding Activity for National Localization Research and Resource Centre would be initiated. 33 TDIL

Government, Academia, Industry together to play globally and to serve locally for making India a Global Multilingual Computing Hub ������� Thank You