Localization and HTML 5 Technical Aspects Felix Sasaki

Localization and HTML 5: Technical Aspects Felix Sasaki DFKI / W 3 C Fellow Sasaki – Feisgiltt 2012 1

Pitch: Why this presentation? • HTML 5 is the upcoming (or existing) format for content on the Web • The Web is becoming multilingual • HTML 5 localization is essential to make this happen • Localization workflows with HTML 5 input / output need to take various aspects of HTML 5 into account – learn more here Sasaki – Feisgiltt 2012 2

Acknowledgement • Thanks to Jirka Kosek for introducing the participants of the W 3 C Multilingual. Web-LT working group to the “do” and “do not” of HTML 5 content creation and processing Sasaki – Feisgiltt 2012 3

Overview • • HTML 5 Serializations + Model Localization Workflow with HTML 5 Metadata for (HTML 5) Localization What Else? Sasaki – Feisgiltt 2012 4

HTML 5 – Serializations + Model • Two serializations <!DOCTYPE html> <head> <meta charset=utf-8> <title>My example</title> </head> <body>. . . </body> </html> Sasaki – Feisgiltt 2012 <html xmlns= "http: //www. w 3. org/1999/xhtml"> <head> <meta charset="utf-8"/> <title>My example</title> </head> <body>. . . </body> </html> 5

HTML 5 – Serializations + Model • Two serializations: HTML 5 vs. XHTML 5 <!DOCTYPE html> <head> <meta charset=utf-8> <title>My example</title> </head> <body>. . . </body> </html> Sasaki – Feisgiltt 2012 <html xmlns= "http: //www. w 3. org/1999/xhtml"> <head> <meta charset="utf-8"/> <title>My example</title> </head> <body>. . . </body> </html> 6

HTML 5 – Serializations + Model • Two serializations: HTML 5 vs. XHTML 5 <!DOCTYPE html> <head> <meta charset=utf-8> <title>My example</title> </head> <body>. . . </body> </html> <html xmlns= "http: //www. w 3. org/1999/xhtml"> <head> <meta charset="utf-8"/> <title>My example</title> </head> <body>. . . </body> </html> One Document Object Model (DOM) document. get. Elements. By. Tag. Name("meta") Sasaki – Feisgiltt 2012 7

Rational • More than 90% of the Web is invalid – See browser “Opera” MAMA report • XHTML was revolution • HTML 5 is evolution – Parsing algorithm for existing Web content – Two serializations as input – Detailed error handling – Ouput: one DOM Sasaki – Feisgiltt 2012 8

Overview • • HTML 5 Serializations + Model Localization Workflow with HTML 5 Metadata for (HTML 5) Localization What Else? Sasaki – Feisgiltt 2012 9

Localization Workflow with HTML 5 as XML HTML 5 as HTML XLIFF-based Localization XHTML 5 as HTML with errors Sasaki – Feisgiltt 2012 10

Localization Workflow with HTML 5 as XML HTML 5 as HTML XLIFF-based Localization XHTML 5 as HTML with errors HTML 5 parsing > DOM creation > (XML serialization) > XLIFF generation Sasaki – Feisgiltt 2012 11

Localization Workflow with HTML 5 as XML HTML 5 as HTML XLIFF-based Localization XHTML 5 as HTML with errors HTML 5 parsing > DOM creation > (XML serialization) > XLIFF generation Sasaki – Feisgiltt 2012 Transformation > XHTML 5 > HTML 5 parsing > HTML 5 or XHTML 5 12

Localization Workflow with HTML 5 Central: HTML 5 parsing library, e. g. validator. nu HTML 5 as XML HTML 5 as HTML XLIFF-based Localization XHTML 5 as HTML with errors HTML 5 parsing > DOM creation > (XML serialization) > XLIFF generation Sasaki – Feisgiltt 2012 Transformation > XHTML 5 > HTML 5 parsing > HTML 5 or XHTML 5 13

Overview • • HTML 5 Serializations + Model Localization Workflow with HTML 5 Metadata for (HTML 5) Localization What Else? Sasaki – Feisgiltt 2012 14

Metadata for (HTML 5) Localization: ITS 2. 0 • “Internationalization Tag Set” 2. 0 • Set of disjoint metadata items (“data categories”) for XML and HTML 5 – Translate, Localization Note, Terminology, Directionality, Ruby, Language Information, Elements Within Text, Domain, Locale Filter, Provenance, Text Analysis Annotation, External Resource, Target Pointer, Id Value, Preserve Space, Localization Quality Issue, Localization Quality Précis, MT Confidence, Allowed Characters, Storage Size Sasaki – Feisgiltt 2012 15

Metadata for (HTML 5) Localization: ITS 2. 0 • “Internationalization Tag Set” 2. 0 • Some items are part of HTML 5 spec – Translate, Localization Note, Terminology, Directionality, Ruby, Language Information, Elements Within Text, Domain, Locale Filter, Provenance, Text Analysis Annotation, External Resource, Target Pointer, Id Value, Preserve Space, Localization Quality Issue, Localization Quality Précis, MT Confidence, Allowed Characters, Storage Size Sasaki – Feisgiltt 2012 16

“Translate” <!DOCTYPE html> <head> <meta charset=utf-8> <title>Translate flag test: Default</title> </head> <body> <p>The <span translate=no>World Wide Web Consortium</span> is making the World Web worldwide!</p> </body> </html> Sasaki – Feisgiltt 2012 17

ITS “global rules” • XPath based metadata approach – Attach metadata to several nodes – Specify metadata for a document format or (HTML) template – Example: map proprietary HTML to ITS “translate” <its: rules. . . > <its: translate. Rule translate="no" selector="//h: *[@class='notranslate']"/> </its: rules> Sasaki – Feisgiltt 2012 18

ITS “inline”, e. g. global rules in HTML 5 • “Work” inside HTML “script” element with proper mime type • Upcoming: application/its+xml • If possible: avoid; use linked rules <!DOCTYPE html> <html>. . . <script type="application/xml“ <its: rules. . . > <its: translate. Rule translate="no" selector="//h: code"/> </its: rules> </script>. . . </html> Sasaki – Feisgiltt 2012 19

“Terminology” <!DOCTYPE html> <html lang=en> <head> <meta charset=utf-8> <title>Terminology test: default</title> </head> <body> <p>We need a new <span its-term=yes>motherboard</span> </p> </body> </html> Sasaki – Feisgiltt 2012 20

“Directionality” <!DOCTYPE html> <html lang=en> <head> <meta charset=utf-8> <title>Dir test: Default</title> </head> <body> <p>In Arabic, the title <quote dir=rtl lang=ar>، ﻧﺸﺎﻁ ﺍﻟﺘﺪﻭﻳﻞ W 3 C</quote> means <quote>Internationalization Activity, W 3 C</quote>. </p> </body> </html> Sasaki – Feisgiltt 2012 21

“Ruby” – XHTML vs. HTML 5 <ruby> <rb>日本</rb> <rt>にっぽん</rt> </ruby> <rb>電</rb> <rt>でん</rt> </ruby> <rb>気</rb> <rt>き</rt> </ruby> Sasaki – Feisgiltt 2012 <ruby> 日本 <rt>にっぽん</rt> 電 <rt>でん</rt> 気 <rt>き</rt> </ruby 22

“Domain” <its: rules xmlns: its="http: //www. w 3. org/2005/11/its" version="2. 0"> <its: domain. Rule selector="/html/body" domain. Pointer="/html/head/meta[@name='keywords']/@content” domain. Mapping="automotive auto, medical medicine, 'criminal law' law, 'property law' law"/>/> </its: rules> Means • Express domain information about content of „body“ element • Domain information is in the „meta“ element • Optional mapping of source content domains, e. g. automotive > auto Purpose: not define a domain vocabulary, but pass domain information to application (MT system, MT training tool) Sasaki – Feisgiltt 2012 23

“Storage Size” <!DOCTYPE html> <html lang=en> <head> <meta charset=utf-8> <title>Example</title> </head> <body> <p>String to translate: </p> <p contenteditable=true id=123 its-storage-size=25>Papua New-Guinea</p> <p contenteditable=true id=139 its-storage-size=25>Dominican Replubic</p> </body> </html> Sasaki – Feisgiltt 2012 24

“Translate” in XML and HTML 5 • ITS namespace vs. HTML 5 native “translate” attribute <article. . . >. . . <para> You need a new <span its: translate="no"> motherboard</span></para>. . . </article> Sasaki – Feisgiltt 2012 <!DOCTYPE html> <html>. . . <p>You need a new <span translate="no"> motherboard</span>. . . </p>. . . </html> 25

“Terminology” in XML and HTML 5 • ITS namespace vs. HTML 5 its-* “term” attribute <article. . . >. . . <para> You need a new <span its: term="yes"> motherboard</span></para>. . . </article> Sasaki – Feisgiltt 2012 <!DOCTYPE html> <html>. . . <p>You need a new <span its-term=yes> motherboard</span>. . . </p>. . . </html> 26

“Quality” metadata in the browser <html>… <script id=its-standoff-1 type=application/xml> <its: loc. Quality. Issues xml: id="lq 1" …> <its: loc. Quality. Issue. Type="misspelling" …/> <its: loc. Quality. Issue. Type="typographical” …/> </its: loc. Quality. Issues> </script>…… <p> <span its-loc-quality-issues-ref=#lq 1>c'es</span> le contenu</p> …</html> • See life demo at http: //tinyurl. com/its 2 -lq-html 5 Sasaki – Feisgiltt 2012 27

Rationale for its-* • HTML attributes are case insensitive; no qualified namespace • ITS 1. 0/2. 0 attributes use – camel case: its: loc. Note, its: term. Info, its: within. Text, … – ITS namespace • Good news: conversion to HTML 5 is straight forward – its-loc-note, its-term-info, its-within-text, … Sasaki – Feisgiltt 2012 28

Effect on Localization Workflow translate, dir, its-loc. Note, its-term. Info, … : „interpretation“ like its: translate, its: term. Info, . . . HTML 5 as XML HTML 5 as HTML XLIFF-based Localization XHTML 5 as HTML with errors HTML 5 parsing > DOM creation > (XML serialization) > XLIFF generation Sasaki – Feisgiltt 2012 Transformation > XHTML 5 > HTML 5 parsing > HTML 5 or XHTML 5 29

Overview • • HTML 5 Serializations + Model Localization Workflow with HTML 5 Metadata for (HTML 5) Localization What Else? Sasaki – Feisgiltt 2012 30

Other HTML versions • “HTML legacy content”: no native supported for its-* • HTML validation tools will complain • Good news: its-* attributes “work” in older versions of HTML (e. g. 3. 2 or 4. 01), e. g. recognized by HTML DOM parser Sasaki – Feisgiltt 2012 31

Tool support • its-* attributes in the pipeline for W 3 C HTML validator • Lot’s of XML+ITS / HTML 5+ITS (partially) sensitive tools being developed in W 3 C Multilingual. Web. LT working group – HTML 5 validation with ITS 2. 0 metadata, XML tool chain, online MT system, translation package creation, simple MT, HTML-to-TMS roundtrip, CMS support (Drupal), quality check, browser based review, named entity annotation, … – *Very raw* details (but further links!) at http: //tinyurl. com/its 2 -use-cases Sasaki – Feisgiltt 2012 32

What’s missing? • ITS 2. 0 localization focuses on HTML markup – Elements, attributes • Server side / client side scripting content not taken into account – Java. Script, PHP, … • Using ITS 2. 0 in HTML 5 with XLIFF: still many bits missing – But: moving forward this week Sasaki – Feisgiltt 2012 33

Overview again … • • HTML 5 Serializations + Model Localization Workflow with HTML 5 Metadata for (HTML 5) Localization What Else? Sasaki – Feisgiltt 2012 34

ありがとうございました。 Localization and HTML 5: Technical Aspects Felix Sasaki DFKI / W 3 C Fellow Sasaki – Feisgiltt 2012 35

LOCALIZATION AND HTML 5: POTENTIAL SLIDES FOR “CHALLENGES AND PROMISES” Sasaki – Feisgiltt 2012 36

What is HTML 5? • DOM specification • Parsing algorithm to cover most of current (and future) Web content • A set of APIs – Part of HTML 5 specification – Defined in separate documents Explanatory and other documents For markup authors, XML tool chains etc. Sasaki – Feisgiltt 2012 37

HTML 5 – Serializations + Model • Two serializations <!DOCTYPE html> <head> <meta charset=utf-8> <title>My example</title> </head> <body>. . . </body> </html> Sasaki – Feisgiltt 2012 <html xmlns= "http: //www. w 3. org/1999/xhtml"> <head> <meta charset="utf-8"/> <title>My example</title> </head> <body>. . . </body> </html> 38

HTML 5 – Serializations + Model • Two serializations: HTML 5 vs. XHTML 5 <!DOCTYPE html> <head> <meta charset=utf-8> <title>My example</title> </head> <body>. . . </body> </html> Sasaki – Feisgiltt 2012 <html xmlns= "http: //www. w 3. org/1999/xhtml"> <head> <meta charset="utf-8"/> <title>My example</title> </head> <body>. . . </body> </html> 39

HTML 5 – Serializations + Model • Two serializations: HTML 5 vs. XHTML 5 <!DOCTYPE html> <head> <meta charset=utf-8> <title>My example</title> </head> <body>. . . </body> </html> <html xmlns= "http: //www. w 3. org/1999/xhtml"> <head> <meta charset="utf-8"/> <title>My example</title> </head> <body>. . . </body> </html> One Document Object Model (DOM) document. get. Elements. By. Tag. Name("meta") Sasaki – Feisgiltt 2012 40

Rational • More than 90% of the Web is invalid – See browser “Opera” MAMA report • XHTML was revolution • HTML 5 is evolution – Parsing algorithm for existing Web content – Two serializations as input – Detailed error handling – Ouput: one DOM Sasaki – Feisgiltt 2012 41

HTML 5: current state • Developed within – W 3 C: HTML 5 to become a standard – WHATWG http: //www. whatwg. org/ - HTML as a “living standard” • High pressure in W 3 C to wrap up – Rationale: “We need one stable version” • At the same time: “We need more features!” – e. g. – ITS 2. 0 – HTML accessibility http: //www. w 3. org/WAI/PF/htmltask-force Sasaki – Feisgiltt 2012 42

Plan: HTML 5 finalized by 2014 • Finish HTML 5 specification in W 3 C by 2014 • Work closely with WHATWG and others on new features, for next version • Don’t try to get everything into HTML 5! – Allow for extension specifications, e. g. ITS 2. 0 – Moving forward at their own pace Sasaki – Feisgiltt 2012 43

HTML 5 time line 2012 2013 2014 2015 2016 ---------- -----HTML 5. 0 CR start. . . CR, LC Rec. . . HTML 5. 1 FPWD --LC + CR. . . CR Rec From http: //dev. w 3. org/html 5/decision-policy/html 5 -2014 plan. html Sasaki – Feisgiltt 2012 44

Challenge: many extensions HTML+RDFa - RDFa WG Web Intents - Web Apps WG / Device APIs WG HTML Editing APIs - HTML Editing APIs CG HTML Media Capture - Device APIs WG Media Capture and Streams - Device APIs WG / Web. RTC WG • Media Fragments URI - Media Fragments WG • Encrypted Media Extensions - HTML WG • Media Source Extensions - HTML WG • . . . many rel value specifications registered at the link type registry – Microformats • • • Sasaki – Feisgiltt 2012 45

Promises: many extensions • See last slide • That also means: Easy of adding localization features to HTML 5 Sasaki – Feisgiltt 2012 46

HTML 5 and Localization Issues • Localization: Mostly covered by ITS 2. 0 • Technical aspects: see presentation from Felix Sasaki on Tuesday • Important: get by-in by browser vendors – Awareness of ITS 2. 0 – Fostering browser based implementations – Easy of adoption for web developers Sasaki – Feisgiltt 2012 47

Metadata for (HTML 5) Localization: ITS 2. 0 • “Internationalization Tag Set” 2. 0 • Set of disjoint metadata items (“data categories”) for XML and HTML 5 • Translate, Localization Note, Terminology, Directionality, Ruby, Language Information, Elements Within Text, Domain, Locale Filter, Provenance, Text Analysis Annotation, External Resource, Target Pointer, Id Value, Preserve Space, Localization Quality Issue, Localization Quality Précis, MT Confidence, Allowed Characters, Storage Size Sasaki – Feisgiltt 2012 48

Metadata for (HTML 5) Localization: ITS 2. 0 • “Internationalization Tag Set” 2. 0 • Some items are part of HTML 5 spec – Translate, Localization Note, Terminology, Directionality, Ruby, Language Information, Elements Within Text, Domain, Locale Filter, Provenance, Text Analysis Annotation, External Resource, Target Pointer, Id Value, Preserve Space, Localization Quality Issue, Localization Quality Précis, MT Confidence, Allowed Characters, Storage Size Sasaki – Feisgiltt 2012 49

HTML 5 and Internationalization Issues • Many things to do – Ruby – International layout (work done mostly via CSS 3 modules) – Here: our most favorite i 18 n core issues Sasaki – Feisgiltt 2012 50