Unicode WTF is UTF for Secondary School Students

☺ Unicode WTF is UTF? (for Secondary School Students) U+263 A Jan Zidek Tieto Czech s. r. o. jan. j. zidek@tieto. com

Table of contents • Puzzles • • Game Questionnaire What Is It? What Contains More Characters? • Unicode Overview • • • Evolution of Character Encoding Can you read it? Unicode Standard Timeline Unicode Characters Unicode on the Web Unicode Script Blocks 17 Code Point Planes BMP – Basic Multilingual Plane Characters per Plane • Encoding • 2 • UTF-32 • UTF-8 • UTF-16 • • 3 From Unicode 1. 0 to Unicode 2. 0 Surrogates UTF-16 Transformation Surrogates Mapping UTF-16 Encoding Example Endianness in Normal Life Unicode BOM – Byte Order Mark • Encodings Summary • • Properties of Each UTFs Unicode Encodings Length Unicode Encoding – Example 1 Unicode Encoding – Example 2 • Useful Stuff Unicode Encoding Unicode 4 5 6 7 8 9 10 11 12 13 15 17 18 19 20 21 22 2016 -02 -26

Unicode 3 ¿ Puzzles U+00 BF 2012 -02 -15

Game 4 Unicode 2012 -02 -15

Questionnaire • • 5 Who knows what binary code is? Who can convert between decimals and binaries? Who knows what hexadecimal code is? Who can convert between hexadecimals and binaries? Who has heard the word Unicode? Who has heard the word UTF-8, UTF-16, UTF-32? Who creates web pages? Unicode 2012 -02 -15

What Is It? 6 Unicode 2012 -02 -15

What Contains More Characters? • • 7 Unicode? UTF-8? UTF-16? UTF-32? Unicode 2012 -02 -15

�� Unicode Overview U+1 F 026

Evolution of Character Encoding • Pre-standards • ASCII – 1960 s – 7 bits – 128 characters • Extended ASCII – 8 bits – 128 characters more → Kód bratří Kamenických → MS-DOS CP 852, … → ISO 8859 -1, ISO 8859 -2, ISO 8859 -3, … → Microsoft CP 1252, CP 1250, … → …, …, … 9 Unicode 2012 -02 -15

Can you read it? 10 Unicode 2012 -02 -15

Unicode Standard • Character coding system • http: //www. unicode. org/ 11 Unicode 2012 -02 -15

Timeline Year Version Features 1991 Unicode 1. 0 Code space: 16 bits U+0000 – U+FFFF Characters defined Address space 7, 161 65, 536 38, 950 1, 114, 112 (17 * 65536) 120, 737 1, 114, 112 1111 1996 Unicode 2. 0 Code space: 21 bits U+0000 – U+10 FFFF 1 0000 1111 2015 12 Unicode 8. 0 Unicode 2012 -02 -15

Unicode Characters 13 Unicode 2012 -02 -15

Unicode Characters Number of Unicode Characters 140, 000 120, 000 100, 000 80, 000 60, 000 40, 000 20, 000 0 Total Han Others 14 1991 1. 0. 0 7, 161 1992 1. 0. 1 28, 359 21, 215 7, 144 1993 1. 1 34, 233 21, 215 13, 018 Unicode 1996 2. 0 38, 950 21, 215 17, 735 1999 3. 0 49, 259 28, 129 21, 130 2001 3. 1 94, 205 71, 382 22, 823 2003 4. 0 96, 447 71, 442 25, 005 2006 5. 0 99, 089 71, 570 27, 519 2010 2014 2015 6. 0 7. 0 8. 0 109, 449 113, 021 120, 737 75, 960 33, 489 37, 061 44, 777 2012 -02 -15

Unicode on the Web 15 Unicode 2012 -02 -15

Unicode on the Web • Before HTML 5: <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> • HTML 5: <head> <meta charset="UTF-8"> </head> • NCR: &#x 61; 16 Unicode 2012 -02 -15

Unicode Script Blocks 17 Unicode 2012 -02 -15

17 Code Point Planes 18 Unicode 2012 -02 -15

BMP – Basic Multilingual Plane 19 Unicode 2012 -02 -15

Characters per Plane 20 Unicode 2012 -02 -15

Unicode 21 �� Encoding U+1 F 427 2012 -02 -15

Unicode Encoding • CEF: Character Encoding Form • UTF: Unicode Transformation Format 22 Unicode 2012 -02 -15

�� UTF-32 U+1 F 467

UTF-32 • 1: 1 24 Unicode 2012 -02 -15

�� UTF-8 U+1 F 466

UTF-8 Bits 26 Last code point Byte 1 Byte 2 Byte 3 Byte 4 Byte 5 7 0000 007 F 0 xxx xxxx 11 0000 07 FF 110 x xxxx 16 0000 FFFF 1110 xxxx 10 xx xxxx 21 001 F FFFF 1111 0 xxx 10 xx xxxx 26 03 FF FFFF 1111 10 xx xxxx 10 xx xxxx 31 7 FFF FFFF 1111 110 x 10 xx xxxx Unicode Byte 6 10 xx xxxx 2012 -02 -15 10 xx xxxx

UTF-EBCDIC Bits 27 Last code point Byte 1 Byte 2 Byte 3 Byte 4 7 0000 007 F 0 xxx xxxx 11 0000 009 F 100 x xxxx 16 0000 03 FF 110 x xxxx 21 0000 3 FFF 1110 xxxx 101 x xxxx 26 0003 FFFF 1111 00 xx 101 x xxxx 31 0010 FFFF 1111 100 x 101 x xxxx Unicode Byte 5 101 x xxxx 2012 -02 -15 Byte 6

UTF-8 Example 28 Unicode 2012 -02 -15

�� UTF-16 U+1 F 467

From Unicode 1. 0 to Unicode 2. 0 • 65, 536 characters ought to be enough for anybody • Workaround concept for Backward Compatibility • Surrogates • Planes (65, 536 characters) • Original Unicode 1. 0 Basic Multilingual Plane • Added 16 extra planes • Total 17 * 65, 536 = 1, 114, 112 characters 30 Unicode 2012 -02 -15

Surrogates Range High U+D 800 – U+DBFF Low Combinations 31 U+DC 00 – U+DFFF 1, 024 * 1, 024 Unicode Mask D 8 XX 1101 1000 xxxx D 9 XX 1101 1001 xxxx DA XX 1101 1010 xxxx DB XX 1101 1011 xxxx 8*256 1, 024 high surrogates DC XX 1101 1100 xxxx DD XX 1101 xxxx DE XX 1101 1110 xxxx DF XX 1101 1111 xxxx 8*256 1, 024 low surrogates 1, 048, 576 new characters 2012 -02 -15

UTF-16 Transformation 32 Unicode 2012 -02 -15

Surrogates Mapping hi lo 33 DC 00 DC 01 DC 02 DC 03 … DFF 0 DFFF D 800 1 0001 1 0002 1 0003 … 1 03 FE 1 03 FF D 801 1 0400 1 0401 1 0402 1 0403 … 1 07 FE 1 07 FF D 802 1 0800 1 0801 1 0802 1 0803 … 1 0 BFE 1 0 BFF D 803 1 0 C 00 1 0 C 01 1 0 C 02 1 0 C 03 … 1 0 FFE 1 0 FFF ⋮ ⋮ ⋮ ⋱ ⋮ ⋮ DBFB 10 EC 00 10 EC 01 10 EC 02 10 EC 03 … 10 EFFE 10 EFFF DBFC 10 F 000 10 F 001 10 F 002 10 F 003 … 10 F 3 FE 10 F 3 FF DBFD 10 F 400 10 F 401 10 F 402 10 F 403 … 10 F 7 FE 10 F 7 FF DBFE 10 F 800 10 F 801 10 F 802 10 F 803 … 10 FBFE 10 FBFF DBFF 10 FC 00 10 FC 01 10 FC 02 10 FC 03 … 10 FFFE 10 FFFF Unicode 2012 -02 -15

UTF-16 Encoding Example 34 Unicode 2012 -02 -15

Endianness 35 Unicode 2012 -02 -15

Endianness in Normal Life Language 92 Endian ninety-two (90 -2) Big zweiundneunzig (2 -and-90) Little quatre-vingt-douze (4 -20 -12) 36 Usage Form Endian Java package com. tieto. intra Big Domain name intra. tieto. com Little Unicode 2012 -02 -15

Unicode BOM – Byte Order Mark • U+FEFF • BOM use is optional at the start of the text stream 37 Unicode 2012 -02 -15

�� Encodings Summary U+1 F 3 B 8

Properties of Each UTFs Name UTF-16 BE UTF-16 LE UTF-32 BE UTF-32 LE Smallest 0000 code point 0000 0000 Largest 10 FFFF code point 10 FFFF 10 FFFF Code unit 8 bits size 16 bits 32 bits Byte order N/A <BOM> big-endian littleendian Fewest bytes per 1 character 2 2 2 4 4 4 Most bytes per 4 character 4 4 4 39 UTF-8 Unicode 2012 -02 -15

Unicode Encodings Length Code range 00 0000 – 00 007 F UTF-8 UTF-EBCDIC UTF-16 UTF-32 GB 18030 1 1 1 00 0080 – 00 009 F 00 00 A 0 – 00 03 FF 2 2 2 for characters inherited from GB 2312/GBK (e. g. most Chinese characters) 2 00 0400 – 00 07 FF 3 4 00 0800 – 00 3 FFF 4 for everything else 3 00 4000 – 00 FFFF 4 01 0000 – 03 FFFF 4 4 04 0000 – 10 FFFF 40 4 5 Unicode 2012 -02 -15

Unicode Encoding – Example 1 41 Unicode 2012 -02 -15

Unicode Encoding – Example 2 42 Unicode 2012 -02 -15

א U+05 D 0 Useful Stuff

Useful Utilities • Online Character Converter • http: //code. cside. com/3 rdpage/us/java. Unicode/converter. html • Babel. Map 6. 0. 0. 2 • http: //babelstone. co. uk/Software/Babel. Map. html • Unibook 5. 2. 0 • http: //unicode. org/unibook/ • Alan Wood’s Unicode Resources • http: //www. alanwood. net/unicode/index. html • Microsoft True. Type. Property Extension • http: //www. microsoft. com/typography/True. Type. Property 21. mspx • Uniview 6. 1 • http: //rishida. net/scripts/uniview/ 44 Unicode 2012 -02 -15

Useful links • http: //en. wikipedia. org/wiki/Unicode_font#Comparison_of_fo nts 45 Unicode 2012 -02 -15

Good Night! �� U+1 F 4 A 4