Unicode for Under Resourced Languages Daniel Yacob Geez

Unicode for Under Resourced Languages Daniel Yacob Ge’ez Frontier Foundation SALTMIL 5: Genoa, Italy 2006

Overview • What is “Unicode”? – More than Just Encoded Letters! • Working with Unicode – How Unicode can help you. – Resources and how to apply them. • Working for Unicode – How you can help Unicode. – How Unicode can help your U-RL.

My Background • Started Ethiopic software work in 1993 – transliterator, keyboard, fonts • Amharic Computational Linguistics in 1994 • “Extended Ethiopic” Unicode Standardization 1995 -2004 • Corpus Collection 1997 – Present • Began Using Unicode in 1995 for Ethiopic – but no Unicode standard existed until 2000!

My Background • Little or no Unicode based resources in 1993 -1997 – Today there is almost always an Open. Source project that you can start with and extend. – Minimize the time and labour you put into developing basic resources. – Avoid the maintenance trap. • We will assume the worst case scenario – You work on a language, using a script, with no pre-existing software resources at all.

What Unicode is Unicode … – is a consortium – is a process – is a community – is a conference – is a database – is a standard – is a collection of standards

What Unicode is not Unicode … – is not a font – is not a keyboard system – is not a transliteration system – is not the ISO – is not perfect – is not complete

Over 80 Scripts not Encoded! India, Nepal, Bangladesh: Southeast Asia • Chakma • Methei / Manipuri • Newari • Sorang Sompeng • Varang Kshiti • Batak • Cham • Javanese • Pahawh Hmong • Viet Thai (excluding China): China: • Lanna • Naxi Geba • Naxi Tomba • Pollard Africa: • Bamum • Bassa • Mende Courtesy of Michael Everson: http: //evertype. com

Over 80 Scripts not Encoded! • Ahom • Alpine • Aramaic • Avestan • Aztec Pictograms • Balti • Brahmi • Büthakukye • Byblos • Chalukya • Chola • Cypro-Minoan • Egyptian Hieroglyphs • Elbasan • Elymaic • Grantha • Hatran • Iberian • Indus Valley • Jurchin • Kaithi • Kawi • Khotanese • Kitan Large Script • Kitan Small Script • Landa • Linear A • Luwian • Mandaic • Manichaean • Mayan Hieroglyphs • Meroitic • Modi • Nabataean • North Arabic • Numidian • Old Hungarian • Old Permic • Orkhon • Pahlavi • Palmyrene • Proto-Elamite • Pyu • Rongorongo • Samaritan • Satavahana • Sharada • Siddham • South Arabian • Soyombo • Takri • Tangut Ideograms • Uighur • Vedic accents Courtesy of Michael Everson: http: //evertype. com

Current State of the Unicode Standard: New Script Additions For Unicode 5. 0 (2006): For Unicode 5. 1 (2008): N’Ko (West Africa) Lepcha (India) Ol Chiki (India) Vai (Liberia) Saurashtra (India) Myanmar minorities (Myanmar) Kayah Li (Myanmar) Rejang (Indonesia) Sundanese (Indonesia) Carian, Lycian, Lydian (historical) Balinese (Indonesia) Phags-pa (historical) Phoenician (historical) Cuneiform (historical) Courtesy of Michael Everson: http: //evertype. com

Working with Unicode is all About Text • Most applicable to problems where language is represented by text. • Unicode addresses some vocabulary but under the scope of localization (CLDR). • May not be the solution if you are not working with text represented in written form – Although, Unicode can be used for symbol processing

Working with Unicode Operating Systems • Most anything from this millennia. • Apple Mac. OS Version ≥ 9. 2 • Microsoft Windows CE, NT, XP, 2000 • Solaris ≥ 2. 8 • Any GNU/Linux (for console use) – GNOME 2. 0 or KDE 2. 0 and Later

Working with Unicode The International Phonetic Alphabet (IPA)

Working with Unicode The International Phonetic Alphabet (IPA) • SIL Charis, Doulos, Gentium – free and most complete – matches “New Times Roman” style – http: //scripts. sil. org/IPAhome

Working with Unicode If you need more letters… • Create Your own Fonts! • Use the Unicode Private Use Area (PUA) – this is Unicode’s extension mechanism. – does not break compatibility with Unicode software. – you must send your fonts with your work. – encode non-letter symbols, no need for fonts.

Working with Unicode The PUA • 6, 400 code points in the range E 000 -F 8 FF • 218 additional available in “planes” 15 & 16 • Work in Plane 0 first (0000 – FFFF) • Intended for company logos, ligatures used by typesetting software, etc.

Working with Unicode Creating Your Own Fonts • Bitmap (BDF) – Faster to create – One size per font, not so scalable – Works best with X-Windows (Unix) • Outline (True. Type, Post. Scipt, Open. Type) – Takes more time – Scalable – MS Windows, Mac, Modern Unixes

Working with Unicode Bitmap Editors • Each letter is a matrix of pixels, like tiles • You toggle them on or off to shape your letters • GBDFED for recent GNOME/Linux • XMBDFED for general Unix • Or search for “BDF Editor”

Working with Unicode

Working with Unicode Bitmap Editors Zoom View Within Edit Window

Working with Unicode Outline Editors • Create Bezier curves to outline scalable shapes • Here traced around a scanned image • Font. Forge http: //fontforge. sf. net

Working with Unicode Creating Your Own Keyboards • No standard formats • Different on every operating system • May require some painful programming – transliteration may be a better alternative. • For small amounts of typing try: Ctrl+Shift+X 1 X 2 X 3 X 4 Ctrl+Shift+1234

Working with Unicode Creating Your Own Keyboards Linux • Migration Toward Smart Common Input Method (SCIM) – simple table based – more complex as needed – http: //scim. sf. net - or Yudit, Emacs for older Unixes, but you can only type in these applications.

Working with Unicode Creating Your Own Keyboards Windows • Keyman, most mature & robust • Keyboards created with Keyman. Developer – $59 academic and developing world license – worth every cent – compiled keyboards also run under Linux with a SCIM module – http: //tavultesoft. com

Working with Unicode Text Processing • International Components for Unicode (ICU) – http: //icu. sf. net – Java, C/C++ – Bindings in: Python, Ruby, C#, Perl 6 (some Perl 5) – started by IBM, is Open. Source – managed by the Unicode president – check with ICU before • 700+ Encoding Conversions – convert legacy systems to and from Unicode – migrate corpora to Unicode

Working with Unicode Text Processing ICU: Normalization • Equate letters and diacritical symbols n + ˜ 006 E u 0303 + 0065 e 00 EA ^ + 0302 + 0065 ê = 030 A + . 0323 ü 00 FC ° 0031 e = 0308 + ñ 00 F 1 ¨ 0075 A = Å 212 B . 0323 + ^ 0302 = ê 1 EC 7

Working with Unicode Text Processing ICU: Regular Expressions • Applies the Unicode Character Database • Categorize every character as one of – – – – Letter Number Separator Punctuation Marks Symbols Others • Subcategories within each. Examples – Letter, Uppercase, lowercase, Other, … – Symbols, Math, Currency, Modifiers, … – Mark, spacing, non-spacing, enclosing • Defines 80 character property types

Working with Unicode Text Processing ICU: Regular Expressions Set Operations • [^p{Letter}] • [p{Letter}p{Number}] • [p{Letter}&p{script=Cyrllic}] • [p{Letter}-p{Latin}] Negation Union Intersection Difference • Important for a character set the size of Unicode.

Working with Unicode Text Processing ICU: Regular Expressions • Enhanced Word Boundaries: Hello There. G’day 123. 456 Classic RE Hello There. G’day 123. 456 Unicode Word Boundaries

Working with Unicode Text Processing ICU: Regular Expressions • Equivalence Classes – [=e=] matches all “e” [eèéêëēĕėęě] – not yet implemented – use Perl instead

Working with Unicode Overloading Perl Regex with Regexp: : Ethiopic Simple Plurals: [#7#]ች vs [������������ ]�

Working with Unicode Text Processing ICU: Transliteration • Defined by “transform rules” – One to one mappings: • α <> a; • β <> b; – Context Rules: • β } [aeiou] > b; • β } [^aeiou] > v;

Working with Unicode Text Processing ICU: Transliteration • Defined by “transform rules” – Applying UCD Properties • Θ } [: Lowercase. Letter: ] <> Th; • Θ <> TH; – Reverse Transliteration Context Rules • σ < [: ^Letter: ] { s } [: ^Letter: ] ; • ς < s } [: ^Letter: ] ; • σ<s;

Working with Unicode Text Processing • ICU: Transliteration – Gets much more sophisticated • See also Perl’s Text: : Transliterate

Working for Unicode Taking Your Work a Step Further • You’ve helped create an orthography –now make it official. • You’ve worked with a pre-existing un-encoded script using the PUA –now formalize it. • You’ve created a transliteration system –make it an ISO standard. • You’ve identified a dialect –encode it in ISO 639. • You’ve developed a keyboard –make it a national standard. • etc.

Working for Unicode Why go the extra mile kilometer? • Ethnic pride and identity is promoted. • Literacy efforts can be encouraged. • The study of historic scripts is kept alive. • Communication between and amongst members of the community is promoted. • Government communication in times of emergency (disease, war, natural disaster). • Leads to localization, greater access to ICT. • …and you become the expert!

Working for Unicode What to Consider • The work will be more social than technical. • The work will take years (at least two). • Review Encoding History – Has this been attempted before and failed? Why? – Are there any non-Unicode encodings? • Determine the Stakeholders – The Government –will they support you, oppose you, jail you? – Political Parties, Religious, Education, Cultural Groups • does anyone have something to lose by the encoding? • Communicate, Communicate… – and be transparent. – the perception of being closed breeds suspicion and opposition. • …even 11 years after the fact, trust me on this.

Working for Unicode New Keyboard? • No international standardization working groups • Contribute Keyboard back to main project • Contact Local ICT Professionals Organization • Contact Local University CS Department • Contact Local Standards Body

Working for Unicode New Language or Dialect? • Contact the ICO/DIS 639 -3 Registration Authority – http: //sil. org/iso 639 -3/ – iso 639 -3@sil. org • Contact Language or Cultural Authority • Contact Local University Linguistics Department

Working for Unicode New Orthography? Or Un-encoded? • Contact the ISO 15924 Registration Authority – http: //unicode. org/iso 15924/ • • • Contact Language or Cultural Authority Contact Local ICT Professionals Organization Contact Local University CS Department Contact Local University Linguistics Department Contact Local Standards Body Contact the Script Encoding Initiative

Working for Unicode The Script Encoding Initiative • http: //linguistics. berkeley. edu/sei • Works with users on script proposals. • Helps raise money for script proposals to be written and free fonts to be created. • Works collaboratively with other groups (e. g. SIL) to avoid duplication of effort. • Helps seek experts to review proposals. • Participates at standards meetings on behalf of minority groups and scholars.

~fini~ • Conclusion – – – Use Unicode Now! You can do it! Yes you can do it! There are no excuses anymore… …its 2006 already, I’m telling you can do this! and when you do (remember I have faith in you!) consider feeding back into the system via standardization. – Be a good citizen of earth, always ☺. Thank You for Listening. Are There Any Questions? This presentation: http: //yacob. org/papers/