eScience eBusiness eGovernment and their Technologies Core XML
e-Science e-Business e-Government and their Technologies Core XML Bryan Carpenter, Geoffrey Fox, Marlon Pierce Pervasive Technology Laboratories Indiana University Bloomington IN 47404 January 12 2004 dbcarpen@indiana. edu gcf@indiana. edu mpierce@cs. indiana. edu http: //www. grid 2004. org/spring 2004 1
What are we doing n n n n This is a semester-long course on Grids (viewed as technologies and infrastructure) and the application – mainly to science but also to business and government We will assume a basic knowledge of the Java language and then interweave 6 topic areas – first four cover technologies that will be used by students 1) Advanced Java: including networking, Java Server Pages and perhaps servlets 2) XML: Specification, Tools, Linkage to Java 3) Web Services: Basic Ideas, WSDL, Axis and Tomcat 4) Grid Systems: GT 3/Cogkit, Gateway, XSOAP, Portlet 5) Advanced Technology Discussions: CORBA as history, OGSA -DAI, security, Semantic Grid, Workflow 6) Applications: Bioinformatics, Particle Physics, Engineering, Crises, Computing-on-demand Grid, Earth Science 2
Contents of this Lecture Set n n Intro: HTML and XML and Unicode Core XML: • • n XML syntax and well-formedness, DTDs and validity. XML namespaces. The XML DOM with linkage to Java. XPath basics. XML Schema. • Validation for data-centric applications. n Later lectures may include additional information on: • • XHTML, SVG, RDF. XML style languages: XSLT and CSS. XML Databases (Xindice, Sleepycat). Search: advanced XPath, XQuery. 3
Motivations for XML: a Better HTML? n Limitations of HTML: • Extensibility: HTML does not allow users to specify their own tags or attributes in order to parameterize or otherwise semantically qualify their data. • Structure: HTML does not support the specification of deep structures needed to represent database schema or object-oriented hierarchies. • Validation: HTML does not support the kind of language specification that allows applications to check data for structural validity when it is imported. 4
XML in the HTML world n XML = e. Xtensible Markup Language. • XML is a subset of SGML—Standard Generalized Markup Language, but XML is specifically designed for the web. n n Specification by W 3 C: http: //www. w 3. org/XML and lots of links like http: //www. xml. org XML 1. 0 in February 98. • XML 1. 1 became a W 3 C recommendation 4 Feb, 2004! n How XML fits into the new HTML world: • XML describes the logical structure of the document. • CSS (Cascading Style Sheets) and/or XSL describes the visual presentation of the document. • DOM (Document Object Model) allows scripting languages like Java. Script to access and dynamically change document objects. 5
Logical vs. Visual Design n Logical design of a document (content) should be separate from its visual design (presentation). • • n Promotes sound typography. Encourages better writing. Is more flexible. Allows the same “knowledge/information” (defined in XML) to be displayed on PC’s, PDA’s, Braille devices etc. XML used to define the logical design, with XSL (Extensible Style Language) or other mechanism used to define the visual layout (e. g. by mapping XML into HTML). 6
XML Design Goals 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. XML shall be usable over the Internet. XML shall support a variety of applications. XML shall be compatible with SGML. It shall be easy to write programs that process XML documents. Optional features in XML shall be kept to the absolute minimum, ideally zero. XML documents should be human-legible and reasonably clear. Design of XML should be prepared quickly. Design of XML shall be formal and concise. XML documents shall be easy to create. Terseness in XML markup is of minimal importance. 7
Document-Centric or Data-Centric? n n Roots of XML in document markup (HTML-like). In practice use of XML as a data format has become at least as pervasive. Examples: • Use of XML format in configuration and deployment files of EJB, Tomcat, … • Uses of XML as a format for message exchange (e. g. SOAP, BEEP). n There is also an important intermediate case—XML as program text for machine interpretation. E. g. : • XSLT declarative transformation language. • WSDL interface definition language for Web services. • BPEL Web services workflow language. 8
Features of XML n n Documents are stored in plain text and thus can be transferred and processed anywhere. Unifying principles make it easily acceptable: • “Everything is a tree” (DOM). • Unicode for different languages. 9
XML and Unicode n n All XML documents must be written using the Unicode character set. Unicode is also the character set used by Java, C#, ECMAScript, …, so we should know something about it. 10
Special Topic: Unicode 11
Unicode n Unicode (http: //www. unicode. org) is an international standard character set that covers alphabets of all the World’s common written languages. • Eventually it should cover all languages, living and dead. • Unicode helps make the Web truly “worldwide”? ! n n Unlike, say, ASCII, which allows for 128 characters, Unicode has space for over 1, 000, of which around 96, 000 are currently allocated. Unicode itself assigns a unique sequence number (code point) to any character, regardless its alphabet. • Three Unicode encoding forms map these code points to sequences of fixed size units—UTF-8, UTF-16, UTF-32. 12
Unicode Code Points n n A Unicode point is a numeric value between 0 and 10 FFFF 16, commonly denoted in one of the formats: U+XXXXX U+10 XXXX where X is a hexadecimal digit. There a total of 1, 114, 112 (= 17 · 164) code points, but most of the World’s common characters are encoded in the first 65, 536 points—the Basic Multilingual Plane (BMP). • 2048 code points in BMP are disallowed because their values have a special role in UTF-16 encoding. n For each assigned character code, the Unicode standard defines a name, and “semantic” properties like case, directionality, . . . 13
Planes n n The space of 17 · 216 Unicode points is conventionally divided into 17 planes of 216 points each. Currently used planes include: Plane # Plane Name Code Range 0 Basic Multilingual Plane 000016. . FFFF 16 1 Supplementary Multilingual Plane 1000016. . 1 FFFF 16 2 Supplementary Ideographic Plane 2000016. . 2 FFFF 16 • Note early versions of Unicode used a strict 16 -bit encoding, and essentially contained just BMP 14
Unicode Allocation n Layout of planes: 15
Blocks n n n Planes are subdivided into blocks. Blocks have variable size. Each block contains the characters of one alphabet or a group of related alphabets. The following slides are a random sampling of the blocks in BMP. • I have put 128 code points on each slide, but this is just what would fit… no general significance to pages of size 128. n For all blocks in the current Unicode standard see: http: //www. unicode. org/charts/ 16
“Basic Latin” (a. k. a. ASCII) 17
“Latin 1” (supplement) 18
“Greek and Coptic” 19
“Arabic” (1 of 2) 20
“Devanagari” 21
“Hangul Jamo” (1 of 2) 22
“CJK Unified Ideographs” (1 of 164) 23
Unicode Allocation n Layout of Basic Multilingual Plane: 24
Unicode Allocation n Layout of Plane 1: 25
Encoding Forms n n n In electronic documents or computer programs the space of Unicode points is normally broken down into a sequence of units, each unit having a convenient, fixed number of bits. The Unicode standard defines 3 encoding forms. The most straightforward is UTF-32, in which the units have size 32 bits. This unit is easily large enough to hold the integer value of a single code point, so UTF-32 encoding is “obvious”. But for nearly all documents, UTF-32 wastes at least half the available storage space. • Also, most programming languages work with 8 bit or 16 bit character units. 26
UTF-16 n The UTF-16 encoding form breaks Unicode characters into 16 bit units. • Java, for example, uses UTF-16 for chars and Strings. n n One 16 bit unit is not large enough to represent all possible Unicode points. Code points higher than 216 -1 are split over two consecutive units. • These are called surrogate pairs. The leading unit is a highsurrogate unit; trailing is a low-surrogate unit. • There are 1024 code points reserved in the BMP for high surrogates, and 1024 more reserved for low surrogates. n This allows for 1024 · 1024 surrogate pairs representing code points higher than 216 -1, while ensuring a legal BMP code point can always be represented in a single unit, and such a unit can never be confused with a surrogate unit. 27
UTF-16 Bit Distribution 28
UTF-8 n n The UTF-8 encoding form breaks Unicode characters into 8 bit units (i. e. , individual bytes). UTF-8 is a variable-width encoding with the following properties: • Any Unicode point maps to 1, 2, 3, or 4 bytes. • Byte sub-sequences for individual characters can always be recognized by local search in the encoded string. • The Basic Latin block coding points (U+0000. . U+007 F) map to one byte, identical to their ASCII value. • All code points in the BMP map to at most 3 bytes. • For European texts UTF-8 will normally use 8 or 16 bits per character (vs 16 bits for UTF-16). • For East Asian texts UTF-8 will normally use 24 bits per character (vs 16 bits for UTF-16). 29
UTF-8 Bit Distribution 30
Encoding Schemes n n The 3 encoding forms don’t quite complete the encoding schemes of Unicode, because they don’t address the endianness with which the UTF-32, UTF-16 numeric unit values are rendered to bytes (byte-serialization). To allow applications to distinguish the endianness of a given document instance, Unicode allows a Byte Order Mark (BOM) as the first character of a document. • BOM is a code point (U+FEFF) for which the byte-reversed unit value doesn’t correspond to a legal code point, so serves to determine the actual byte order. 31
The Seven Unicode Encoding Schemes 32
Unicode Summary n n Unicode is a large and important standard that is a foundation for XML, HTML, etc. Although you are unlikely to manipulate the encodings yourself, you should be aware of the pros and cons of UTF-16, UTF-8. • UTF-8 is backwards compatible with ASCII—Basic Latin texts can be read by legacy applications. • UTF-16 is better-suited for internationalization. It is the internal representation used by Java, C#, ECMAScript, … 33
Core XML I: The XML Specification 34
Introduction n n In this section we will describe core XML, as defined by the XML specification document from W 3 C. XML is a format for documents—originally documents for the Web—but its scope is wider than that. XML is a subset of SGML—Standard Generalized Markup Language. Some features of XML exist simply for compatibility with SGML. XML can also be viewed as a kind of generalization of HTML—presumably familiar from the Web. 35
XML Parsers and Applications n n For purposes of this section an application is any program that reads data from an XML document. Applications normally do not (and probably should not) read the text of XML documents directly. The XML specification assumes that this text is initially processed by a piece of software called an XML processor. We will also refer to this as an XML parser. The parser exhaustively checks that the text is in a legal XML form, then extracts the essential data from the document, and hands that data to the application. 36
Reading XML Data <? xml version="1. 0"? > <!DOCTYPE svg PUBLIC "-//W 3 C//DTD SVG 1. 0//EN" "http: //www. w 3. org/TR/2001/REC-SVG-20010904/DTD/svg 10. dtd"> <svg width="500" height="500"> XML Parser <g transform='rotate(45)'> <circle cx='150' cy='50' r='25'/> <text x='125' y='100'>A Circle</text> </g> </svg> XML Source svg g Application width 500 height 500 Parsed XML Data transform rotate(45) circle cx 150 cy 50 r 25 text x 125 y 100 A Circle 37
Well-formed Documents n An XML document follows a strict syntax. For example: • An XML document contains regions of text called elements, delimited by matching start-tags and end-tags. Elements must be correctly nested. • Start-tags may include attribute specifications, where attribute values are strings delimited by matching quote marks. n n A document that obeys the full set of these rules is called well-formed. Every legal XML document must be well-formed, otherwise it cannot be parsed. 38
Examples n Well-formed: <html> <body style="font-style: italic"> This is a well-formed document. </body> </html> n Not well-formed: <html> <body style=font-style: italic> This is not a well-formed document. </html> </body> • The style attribute value is not in quote marks, and the html and body tags don’t nest correctly. 39
Install Xerces n n n The Xerces parser is a product of the Apache XML project, http: //xml. apache. org. Follow the “Xerces Java 2” project link and go to the download area, then to the master distribution directory or a mirror directory. Download Xerces-J-bin. 2. 6. 2. zip, and extract it to a suitable place, e. g. C: • When extracting, remember to select “Use folder names”!! • This should create a folder called, e. g. , C: xerces-2_6_2. 40
Put Xerces on your Class Path n Using the menu at Control Panel→System→Advanced→Environment Variables add the jar files xerces. Samples. jar, xerces. Impl. jar, and xml-apis. jar, to you class path. • E. g. append …; C: xerces 2_6_2xerces. Samples. jar; C: xerces 2_6_2xer ces. Impl. jar; C: xerces-2_6_2xml-apis. jar to the value of your CLASSPATH variable. 41
Example Using Xerces n Copy the two HTML examples given above to files called, say, wellformed. html and illformed. html. Then, in a new Command Prompt window, try running the commands: > java dom. Writer wellformed. html … > java dom. Writer illformed. html … • The first command should just echo the document. The second should print a syntax error message. • dom. Writer is one of the sample applications in the Xerces release. It simply uses the Xerces parser to convert the source file to a tree data structure (DOM), then converts the tree back to nicely formatted XML, which it prints. 42
“Rolling Your Own” Parser? n People approaching XML sometimes decide they can write their own “lightweight” parser that handles just the bit of XML their application needs. • In general this is a bad idea! • We will see later that even basic XML is a moderately complex specification; unless you are going to invest a lot of effort it is unlikely you can parse the full specification more efficiently than existing parsers. • If you subset the specification you may be compromising the most crucial advantage that XML brings to your application —interoperability. • Later in these lectures we will see how to use the Xerces parser from your own Java programs, to read XML input. 43
Valid Documents n An XML document may optionally include a Document Type Definition (DTD). • This declares the names of all elements and attributes appearing in the document, and how they may nest. • The DTD also declares and defines entities that may be referenced from within document content. n A well-formed XML document that includes a DTD— and accurately follows the declarations in that DTD—is called valid. 44
Invalid Documents n n n It is quite possible to parse invalid (but well-formed) documents, by using a non-validating parser. Many applications accept XML files without DTDs, which are therefore technically invalid. Applications may exploit “validation” mechanisms other than DTDs. An important one is XML Schema which we will discuss later. • A document validated against an XML Schema usually does not have a DTD, so technically is invalid as far as the base XML specification is concerned. • But of course it is valid relative to the XML Schema specification! 45
Validation Side Effects n n The use of a validating parser certainly affects what documents are treated as legal. In some cases “switching on” validation may also alter the exact data passed from the parser to application. These effects will be considered when we discuss DTDs. 46
Physical Entities n n An XML document is represented by one or more “storage units” (typically files), called “entities”. We can enumerate five kinds: • Document entities—root XML documents. • Parsed external entities, which contain fragmentary XML content. • External DTD subsets, which contain some or all of the DTD declarations needed by a document. • External parameter entities, which also contain fragmentary DTD content. • Unparsed external entities, which are usually complete “binary” files in some native format (not XML). 47
Physical Structure n n n The structure of a non-trivial XML document is illustrated in the following figure. Every XML document must have exactly one document entity. It may also involve zero or more external entities: • The document entity may reference any number of external general entities. These can be parsed external entities or unparsed external entities. A parsed external entity may in turn reference other external general entities. • The document may have at most one external DTD subset. • A DTD subset in the document entity, or an external DTD subset, may reference any number of external parameter entities (which may in turn reference other external parameter entities). 48
A Complex XML Document Entity External DTD Subset DTD Content External Parameter Entity Parsed External Entity Unparsed External Entity Parsed External Entity 49
Syntactic Features n The following two tables summarize the “top-level” syntax of all the constructs in XML. Full details will be given in later slides, as needed. • The first columns give an abbreviated example of the syntax, the second columns (“what? ”) describe the construct, and the third columns (“where? ”) specify the places in an XML document where the construct may appear. • In a “where? ” column, Document means at the top-level of the document entity, and Content means in the kind of content allowed in an element—also called Parsed Character Data. • A Literal is character data in quotes—exactly what can appear in a literal depends strongly on its context. • XML Names will be discussed shortly. 50
Syntax I: Logical Structures Example Syntax What? Where? <Name …>Content</Name> Element Document, Content Name = Literal Attribute specification Element start tag <? xml …> XML declaration/ Text declaration Document/ External entity <? Name …> Processing instruction Document, DTD, Content <!-- … --> Comment Document, DTD, Content <!DOCTYPE …> DTD Document <!ELEMENT …> Element declaration DTD <!ATTLIST …> Attributes declaration DTD <!ENTITY …> Entity declaration DTD <!NOTATION …> Notation declaration DTD 51
Syntax II: References, Sections Example Syntax What? Where? &#Code-point; Character reference Content, Literal &Name; Entity reference Content, Literal %Name; Parameter entity reference DTD <![[ … ]]> CDATA section Content <![IGNORE[ … ]]> Conditional section DTD <![INCLUDE[ … ]]> Conditional section DTD 52
Character Set n Every XML document must be composed using the Unicode character set. • The specification does not stipulate any particular encoding, though defaults are UTF-8 or UTF-16. • ASCII is a subset of Unicode, so you can create XML documents using your favorite, pre-Unicode, text editor. 53
Allowed Character Ranges n The allowed characters are • white space: U+0020 (space), U+0009 (tab), U+000 A (line feed), U+000 D (carriage return), plus all Unicode characters higher than U+0020, excluding: n The surrogate blocks U+D 800. . DFFF. n U+FFFE and U+FFFF (noncharacters in Unicode. Note FFFE 16 is the BOM after byte-reversal in UTF-16) n Because some codes are forbidden, can’t consider including raw binary data in parsed XML (without additional encoding). 54
Names and Name Tokens n In XML, names are used in many places: • An element has a name, an attribute has a name, an entity is referenced by a name, etc. n n As in programming languages, there are rules about what constitutes a valid name (next slide). In XML there is also a concept of name tokens, which are strings similar to names that can be specified as values of certain types of attribute. • They are less restricted. For example a number can be a valid name token. 55
What is a Name? n Well-formed XML name tokens include any sequence of the following characters: • a letter • a digit • a period (“. ”), hyphen (“-”), underscore (“_”), or colon (“: ”) n Well-formed XML names include any name token that starts with a letter, “_” or “: ”. • Names that begin with “XML” (in upper, lower or mixed case) are reserved. n Case is significant in XML names: names are only identified if they consist of identical character sequences. 56
Examples n Some well-formed names: bryan Big-Foot Century_21 rated: PG _. --. _ n Some illegal or reserved names: 2004. com #x. FFFF xml-1. 0 57
What is a Letter? n n n In defining names and name tokens, XML 1. 0 relies on a definition of letter and digit. These are “Westerncentric” notions—see Chapter 4 of the Unicode 4 standard for relevant discussion. XML 1. 0 defines a “letter” as an alphabetic or syllabic base character or an ideograph, and gives a (probably dated) recipe for extracting these from Unicode character databases. XML 1. 1 gives intentionally liberal Unicode ranges for Name. Start. Char and Name. Char, then has a nonnormative appendix suggesting what kind of characters should be preferred. 58
Elements n n The most characteristic markup feature of XML is the element. The basic syntax for an element is either <Name>Content</Name> or <Name/> n n Here Name is an XML name—the type of element. In the first case, Content stands for further text, which may include nested elements. The second case is called an empty element. It is formally equivalent to <Name></Name> 59
Examples from XHTML n n XHTML is an XML-compatible dialect of HTML. Examples: • <h 1>A header</h 1> This is an element of type h 1 with content text “A header”. • <body><h 1>My Page</h 1>Welcome. </body> This is an element of type body. The content is a nested h 1 element plus the text “Welcome. ”. • <hr/> This is an empty element of type hr. • Examples here don’t illustrate, but it is allowed to include white space in the tags, on either side of the element name. 60
Attributes n n An element start tag (or an empty element tag) may include one or more attribute specifications. An attribute specification has the general syntax Name = "Value" or Name = 'Value' where Name is the name of the attribute and Value is some text. • This value text must not include the literal character “<” (directly or through an entity replacement, see later). • It must not include the character (" or ') used to delimit it. • The value text may include line breaks and other white space. • Attribute specifications can have white space around the “=”. 61
Examples from SVG n n SVG (Scalable Vector Graphics) is an XML notation for representing graphical objects. Examples: • <rect x="50" y="50" width="100" height="75"/> An empty element representing a rectangle at position (50, 50), with shape 100 × 75 pixels. • <g transform='rotate(45)'> <circle cx='150' cy='50' r='25'/> <text x='125' y='100'>A Circle</text> </g> A group containing a circle and some text. The group as a whole is rotated 45° about the origin, as specified by the attribute with name transform and value rotate(45). 62
Possible Display of Examples 63
The Document n It is recommended to start a document entity with an XML declaration, although technically this is optional. • An XML declaration must be strictly the first thing in the file—no white space can go before an XML declaration (a single BOM is allowed). n Every XML document entity must contain exactly one top-level element called the root element. • Of course there can be any number of elements nested inside the root element. n n If there is a Document Type Declaration, this must appear before the root element. Miscellaneous other things are allowed at the document level (anywhere between XML declaration and end of file): • White space • Comments • Processing instructions n Anything before the root element is collectively called the prolog. 64
Document Layout XML declaration (optional) …optional comments and processing instructions … Document Type Declaration (optional) Prolog …optional comments and processing instructions … root element (required) …optional comments and processing instructions … 65
XML Declaration n n It is strongly recommended to start any document entity with an XML Declaration. The XML declaration has syntax: <? xml version=Literal …optional declarations… ? > n The version specification and optional declarations have a syntax similar to attribute specifications on elements (declared values are quoted the same way). • The value assigned to version must be 1. 0 or 1. 1, according to which version of the XML specification you are following (1. 0 may be prudent, for now). • The optional declarations are the encoding declaration and the standalone declaration. 66
Example XML Declaration n Here is a complete (and completely pointless) XML document with an XML declaration—including optional parts—and an empty root element: <? xml version="1. 0" encoding="UTF-8" standalone="yes"? > <my-root/> • In general the encoding declaration, if included, specifies the character encoding scheme. n This must be an encoding of Unicode, but it may be some encoding not defined in the Unicode standard. • Informally a standalone declaration says whether this document stands alone. See later for details. Meanwhile, “if in doubt, leave it out”—the default is always safe. 67
On Encoding Declarations n If you are awake, you may wander what is the point of declaring the character encoding inside a document encoded according to that scheme—apparently we can’t read the declaration unless we already know the scheme! • The XML specification gives an argument about autodetection of the encoding scheme allowing us to read far enough into the document to read the contents of the declaration. • Debatable how much point there is giving an explicit encoding declaration—seems more logical to rely on auto-detection of standard encodings alone, or external metadata? 68
Comments n Comments can appear at the top level of a document, in the DTD, or in the document content. • Specification says the parser may or may not pass the text of the comment to the application. • XML comments are not white space—placement is more restricted. n Comments have the syntax <!-- Text --> where Text is any text, except it must not contain two adjacent hyphens, “--”, and it must not end with a hyphen: <!-- This--not--legal --> <!-- Neither is this---> 69
Processing Instructions n Processing instructions can appear anywhere that comments can—they effectively are comments so far as the XML parser is concerned. • But the specification requires the parser to pass the text of a processing instruction to the application. n The syntax is <? Target Text ? > n where Target is an XML name, and Text is any text, except that it must not contain the string “? >”. Processing instructions are a convenience for the application. They allow application-specific annotation of the document (example <? php … ? >). The target name may be declared as a notation (see later). 70
The Document Type Definition 71
Document Types n n In the syntax for the document entity, we saw that the document type declaration was an optional feature. This declaration, if present, contains the document type definition, or DTD. A validating parser will read the DTD, which should contain (among other things) declarations of all the elements and attributes appearing in the body of the document. The DTD is required if the parser is validating, but optional for a non-validating parser • Even a non-validating parser may read the DTD if it is present, to look for entity declarations. These will be discussed later. 72
Document Type Declaration n n The document type declaration, if present, appears before the root element of the document. The most general form of this declaration contains three things: 1. The type (i. e. name) of the following root element 2. An identifier for an External DTD Subset 3. An Internal DTD Subset. n Items 2. and 3. are optional. The DTD itself is either given in an external file (“external subset”), or “in line” in the document (“internal subset”), or divided between the two. 73
General Syntax n General syntax is one of: <!DOCTYPE Name [ Declarations ] > <!DOCTYPE Name External-ID [ Declarations ] > where Name is the type of the root element, External-ID is an identifier for an external entity—a separate file containing the external DTD subset—and Declarations is the internal DTD subset. • Syntactically, the form: <!DOCTYPE Name > is allowed, but can never yield a valid document. (Why? ) 74
External Entity Identifiers n n n External entity identifiers can occur in a couple of places: they will be discussed in the next section when we discuss entity declarations. Meanwhile the simplest form is “SYSTEM Literal”, where Literal contains a URI—a file name or URL or (in principle) a URN. So a valid document with an external DTD might be: <!DOCTYPE my-root SYSTEM "my-type. dtd"> <my-root> … </my-root> where my-type. dtd is a local file name. 75
Example DTD n A DTD (subset) contains a series of DTD Declarations. • Comments and processing instructions can be interleaved amongst the declarations. n A possible pair of declarations in an internal DTD: <!DOCTYPE my-root [ <!ELEMENT my-root (my-leaf)> <!ELEMENT my-leaf EMPTY> ]> • These declare two types of element: my-root and my-leaf. • They also specify a very restricted kind of nesting. The only valid document content would be something equivalent to: <my-root> <my-leaf/> </my-root> 76
Validation Using dom. Writer n If I save the document type declaration and root element in a file called “internaldtd. xml”, I can validate the file with the Xerces parser by using the dom. Writer sample application as follows: > java dom. Writer –v internaldtd. xml • If validation is successful, this simply prints a formatted version of the input file. If validation fails, you will see error messages early in the output. • The –v flag is needed here: validation is “off”, by default. n “v” must be lower case: “-V” forces no validation. 77
Element Type Declarations n The general form of an element type declaration is: <!ELEMENT Name Content-Specification > where • Name is an XML name, the element type, and • Content-Specification describes the content allowed in elements of type Name (i. e. it describes the structure of any text surrounded by Name tags). • No element type may be declared more than once. 78
Empty Content Specification n n The simplest content specification that can appear in a document type definition is “EMPTY”. This means elements of this type never have any content, not even white space! E. g. the declaration <!ELEMENT b EMPTY> allows elements: <b/>, <b></b> but not: <b>Hello</b>, <b><!-- Hello --></b>, <b> </b>, • The form <b/> is preferred when the element type is declared EMPTY; <b></b> is preferred when empty content matches another declaration. 79
Parsed Character Data n The content specification “(#PCDATA)” means elements include only “flat” text—no nested elements are allowed. PCDATA stands for “Parsed Character Data”. • Comments and processing instructions are allowed in parsed character data. n E. g. the declaration <!ELEMENT a (#PCDATA)> allows elements: <a>Hello, hello </a>, <a>Hello <!-- pause --> hello</a>, but not: <a>Hello <b/> hello</a>, <a>Hello</a> hello </a>. 80
Content Models and Mixed Content n Two sorts of content specification allow for nested elements: • If an element may contain only nested elements, with no interspersed character data, one gives a content model. n E. g. typical content like this: <c> <a>Hello</a> <b/> </c> c contains nested a and b elements, but no directly nested character data. • Otherwise, if character data and elements can both appear, the content is specified (less precisely) as mixed content. n E. g. typical content like this : <c> Hello <b/> </c> c can’t have a content model because character data “Hello” directly nested in c. 81
Structure of Content Models n n A content model is a kind of regular expression, in which elements as whole are treated as the tokens. The content model: (a, b, c) n means the content is a sequence of an a element, a b element, and a c element, in exactly that order. The content model: (a | b | c) means the content is a choice of an a element, a b element, or a c element. 82
Composing Content Models n You can compose sequences and choices with appropriate parentheses, e. g. ((a | b), c) means a sequence of a c or a sequence b c, and ((a, b) | c) n means a sequence of a b or a single c. In these expressions you can follow an individual element name or a parenthesized expression by one of the modifiers ? , *, or +. • ? Means the term is optional (occurs zero or one times), • * Means the term can be repeated zero or more times, • + Means the term can be repeated one or more times, e. g. (a | b | c)+ means any combination of one or more as, bs and cs 83
Content Model Example n n Suppose the format of a simple “report” is: a title, followed by any mix of paragraphs and figures, followed by an optional bibliography section. A suitable document type might start: <!DOCTYPE report [ <!ELEMENT report (title, (paragraph | figure)*, bibliography? ) > <!ELEMENT title (#PCDATA)> <!ELEMENT paragraph (#PCDATA)> <!ELEMENT figure EMPTY> <!ELEMENT bibliography (reference)* > … ]> 84
Example report Data <report> <title>Early Use of XML</title> <paragraph>Recently uncovered documents (see figure) prove XML was first used by the Incas of ancient Peru. </paragraph> <figure source="notafake. jpg"/> <paragraph>The author is grateful to W 3 C for making this research possible. </paragraph> <bibliography> … </bibliography> </report> 85
Content Model Miscellany n n n As stated earlier, character data cannot appear in a (valid) element described by a content model. But white space, comments, and processing instructions can be interleaved in an allowed element sequence. The first character in the top level of the content model expression must be a left parenthesis: (a*) and (a)* are allowed as content models, but a* is not. The specification says a content model must be deterministic. This fairly technical requirement (needed for SGML compatibility only) is outlined in the following few slides. 86
Positions in Content Models n For purposes of this discussion, we label every appearance of an element type in a content model with some unique identifier. We call a labeled element type a position. • E. g. if the content model is: (a, (b, c)*, (b | a)) then a labeled version could be: (a 1, (b 2, c 3)*, (b 4 | a 5)) which includes positions a 1, b 2, c 3, b 4, and a 5. 87
“Deterministic” n n Imagine reading some input string of elements from the document, one element at a time, and assume that at the time each individual element is read we have no knowledge of what follows it in the document. A content model is deterministic if, as elements are read from the document and matched against the content model, there is never more than one position in the content model that could match each inputted element. • This must be true for all possible input strings. n To give a full formal definition, need to define a follows relation between positions. Won’t do that here. 88
Determinism Examples n The content model ((b 1, c 2) | (b 3, d 4)) is not deterministic. Suppose the first element read is a b. At the time it is read, the b may match against b 1 or against b 3. • The content model (b, (c | d)) is equivalent and deterministic. n The content model (a 1, (b 2, c 3)*, (b 4 | a 5)) is not deterministic. Suppose one a element has already been read and the next element encountered is a b. At the time b is read, it may match against b 2 or b 4. • In this case DTDs provide no equivalent deterministic model. 89
Mixing Content n Many XML formats allow character data and text to be intermingled in the content of a single element. • For example XHTML allows a mixture of text and element markup in the content of its body element (or, say, its p element). n n Neither the parsed character content specification “(#PCDATA)”, nor a content model specification, will allow this mixture. Instead one must use a mixed content specification. 90
Mixed Content Specifications n A mixed content specification looks like: (#PCDATA | Name 1 | Name 2 | … | Namen)* where Name 1, Name 2, …, Namen are different element types. • This may look like a content model, but it is not! Mixed content specification must appear in the ELEMENT declaration in exactly the form above. • You can’t replace any of the Name fields with more general expressions; the #PCDATA token must appear first in the parentheses; the right parenthesis must be followed by *. n Nevertheless the matching rules are what the syntax suggests: a valid element can contain an unrestricted mix of character data and the listed types of element. 91
Mixed Content Example n Earlier we said that an element c like : <c> Hello <b/> </c> n cannot be described by a content model. It can be described by the declaration: <!ELEMENT c (#PCDATA | b)* > n Note however this allows any sequence of text and b elements. • There is no way to specify there must be exactly one piece of text preceding exactly one b element. 92
ANY Content n n n The last kind of content specification allowed in an element declaration is “ANY”. This is equivalent to a mixed specification naming all elements declared anywhere in this DTD. It does not allow appearance of elements that are undeclared; and all nested elements must be valid according to their own declarations! 93
Attribute-List Declarations n n n For a valid document, besides declaring the elements themselves, you must also declare all attributes that appear on all elements. An ATTLIST declaration declares a list of attributes for a named element type. For each attribute in the list, the declaration defines three things: 1. the name of the attribute, 2. the type of values it can be assigned, and 3. whether the attribute has a default value, and if so what it is. 94
Syntax of ATTLIST n Is fairly unstructured—it contains the name of the element the attributes apply to, then simply a list of triples: <!ATTLIST Element-Name 1 Type 1 Default 1 Name 2 Type 2 Default 2 … > where Name 1, Type 1, Default 1, etc are the attribute properties mentioned on the previous slide. 95
Attribute Types n There is a longish list of allowed types for attributes (not nearly as long as in XML Schema). The specification subdivides them into: • string type, • tokenized types, and • enumerated types. n The simplest and most general is string type, indicated by the keyword CDATA in the attribute declaration. • The value of an attribute is declared with this type can be any literal string. n Other types will be described shortly. 96
Attribute Defaults n The attribute type is followed by a default rule, one of: Literal #FIXED Literal #REQUIRED #IMPLIED • Literal is a default value. The attribute is always logically present (passed to application), but optionally specified. • #FIXED modifier means the attribute can only take its default value (trying to specify something else is invalid). • #REQUIRED means the attribute must be specified (so no default is necessary). • #IMPLIED means the attribute is optional (if unspecified it is absent, so no default is necessary). 97
Attribute Default Examples n Attribute list declaration: <!ATTLIST a val fix req opt n CDATA "nothing" CDATA #FIXED "constant" CDATA #REQUIRED CDATA #IMPLIED> Instances of element a: <a val="something" fix="constant“ req="reading" opt="extra"/> <a req="no experience"/> <!-- OK: val = “nothing”, fix = “constant”, opt absent. --> <a fix="variable"/> <!-- Invalid! fix not “constant” and req unspecified. --> 98
“Tokenized” Attribute Types n In place of “CDATA” we may have: • NMTOKEN: syntax of attribute value is an XML name token (defined earlier), e. g. “ 2004”. • NMTOKENS: …a list of name tokens. • ID: attribute is an identifier for this element. • IDREF: attribute is a reference to another element in this document. • IDREFS: …a list of references to other elements. • ENTITY: attribute is a reference to an unparsed external. Entity (see next section). • ENTITIES: …a list of references to unparsed external entities. n Items in a “list” are separated by white space. 99
Element Identifiers n The value given to an attribute with type ID should follow the syntax of an XML name. This name acts as an identifier for the element instance on which it appears. For validity: • For each element type in the DTD, there should be at most one attribute declared with type ID. • All identifiers on all element instances in the document must be different (regardless element type). n For validity, the value given to an attribute of type IDREF should be an identifier for an element appearing somewhere in the document. 100
Example n Assume the attribute name on element type agent is declared to have type ID, and the attribute boss is declared to have type IDREF. <agent name="Alice" boss="Alice"/> <agent name="Bob" boss="Alice"/> <agent name="Carole" boss="Alice"/> Alice <agent name="Dave" boss="Bob"/> n This document captures the hierarchy illustrated on the right. Can use this technique to represent general graphs. Bob Carole Dave 101
Enumerated Attribute Types n The type field in an ATTLIST declaration may have the form: (Token 1 | Token 2 | … | Tokenn) n where each Token is an XML name token. This says that the specified value of the attribute must be one of these token values. • Example: attribute list declaration: <!ATTLIST a color (red | green | blue | white) "white"> • Instances of element a: <a color="red"/> <!-- OK. --> <a color="black"/> <!-- Invalid! --> 102
Notation Attribute Types n The type field in an ATTLIST declaration may have the form: Notation (Name 1 | Name 2 | … | Namen) n where each Name is declared elsewhere in the DTD as a notation (see next section). The handling of this type is similar to enumeration types. 103
Attribute Declaration Miscellany n n n Attributes for a single element type may all appear in a single ATTLIST declaration, or they may be divided over several ATTLIST declarations. It is allowed to declare the same attribute more than once, but any declarations after the first are ignored. It is allowed (but pointless) to declare an attribute for an undeclared element type. 104
Attribute Order n n Attribute specifications can appear in element tags in any order. The same attribute cannot be specified more than once on a single element. 105
Entities, References, and other Processing Issues 106
Collecting Things Together n n n So far we have described an ideal subset of XML in which a document contains DDT, elements, attributes, and character data, all laid out linearly. In reality fragments of content and DTD may be defined in places other than where they ultimately appear (perhaps in other files), and portions of text may need special processing before they are made available to the application. To complete the discussion we must cover: • • Character references and entity references. Internal and external entity declarations, and notations. CDATA sections. Conditional sections. 107
Character References n n n Character references can be viewed as an “escape” mechanism that allows us to include specially-treated or hard-to-type characters in the XML document. They include the Unicode point for a character, taking either of the forms “&#dd…; ” or “&#x. XX…; ”, where ds are decimal digits and Xs are hexadecimal digits. For example: • & or &#x 26; represents “&” (ampersand). • < or &#x 3 C; represents “<” (left angle bracket). • σ or &#x 3 A 3; represents “Σ” (large Greek sigma). n One application is for including the literal characters “<” or “&” in parsed character data. 108
CDATA Sections n CDATA sections provide a way of including a section of character data in an XML document. • The section can include “<” and “&” characters without escaping—in a CDATA section these characters have no special significance (so “markup” syntax is ignored). n The syntax is <![CDATA[ Text ]]> where Text is any text, except that it must not contain the string “]]>”. • One application is for including scripting in XML—e. g. Java. Script uses < and & for operators. • You cannot include any characters generally forbidden in XML: can’t put raw “binary” data in a CDATA section! 109
Entity References n n A character reference includes a single Unicode character in the document. An entity reference includes the content of some “entity”, which may be the contents of an external file. The simple syntax is: &Name; n where Name is an XML name. The name of the entity, Name, must have been declared in the document DTD. There are just five exceptions to this rule. 110
Predefined Entities n As a convenience the entities amp, lt, gt, apos, and quot are considered predefined. • You may declare them in a DTD, but it isn’t necessary. n They must contain the single-character values: • • • n n & expands to “&” (ampersand). < expands to “<” (left angle bracket, or less than). > expands to “>” (right angle bracket, or greater than). ' expands to “'” (single quote, or apostrophe). " expands to “"” (double quote). Provide a more convenient way of including reserved characters. Note these are entity references, not character references (affects processing in some contexts). 111
A Hofstadteresque XHTML Example <html> <body> The source of this document is: <pre> < html> < body> The source of this document is: < pre> & lt; html& gt; … & lt; html& gt; < /pre> </body> </html> </pre> </body> </html> 112
Declaring Entities n Entities are defined in the DTD by an ENTITY declaration with the syntax: <!ENTITY Name Definition > n Here Name is of course the name of the declared entity. This general form can declare: 1. Internal entities 2. Parsed external entities 3. Unparsed external entities Only 1. and 2. can be included by an entity reference. • Later we will see another form of the ENTITY declaration that defines parameter entities. 113
Internal Entities n n The simplest kind of entity is an internal entity. In this case the Definition is just a literal string, and the entity behaves like a kind of macro. The string can contain character data and markup. E. g. : <!ENTITY <!ENTITY me "Bryan" > lt "& #60; " > Sigma "&#x 3 A 3; " > flag "Stars & Stripes" > icon '<image xlink: href="icon. jpg" />' > These declarations allow the shorthand forms “&me; ”, “< ”, “Σ ”, “&flag; ”, “&icon; ” in the body of the document. 114
Replacement Text for Internal Entities n n Character references (and also parameter entity references, see later) appearing in the entity definition are expanded when the declaration is processed. So the replacement text for lt is “<” and for Sigma is “Σ”. References to other entities appearing in an entity definition are not expanded at the time the declaration is processed (see next slide). The replacement text for flag is “Stars & Stripes”. 115
Expansion of General Entities n n n When the parser encounters an entity reference in the content of a document, the reference is expanded to the replacement text for the entity. The parser then resumes processing, starting at the beginning of the inserted text. If the replacement text contains further entity references, these are replaced in turn, as they are encountered. An entity reference must not expand to fragmentary markup. For example, the replacement text may include complete elements, but not isolated start tags, or isolated “<” characters. 116
External Entities n n For an external entity the Definition appearing in the ENTITY declaration is an external ID—same syntax as in DOCTYPE declarations. There are two forms (for parsed external entities): <!ENTITY Name SYSTEM URI-Literal > <!ENTITY Name PUBLIC Public-ID-Literal URI-Literal > n The first form is fairly self-explanatory: URI-Literal is typically an absolute or relative URL. • If relative, it is relative to the location of the document entity or external entity containing the ENTITY declaration. 117
References to External Entities n External entities may be referenced anywhere markup can appear in document content, except in the literal value of an attribute. • Internal entities can be referenced in attribute specifications. 118
Public Identifiers n Where an external entity is likely to widely used one can give a PUBLIC identifier. • This acts as a logical identifier, something like a URN. • XML standard itself doesn’t specify a syntax for public identifiers, but XML-based standards usually follow the SGML format for Formal Public Identifiers. n For example the DOCTYPE declaration for an XHMTL 1. 1 document should be: <!DOCTYPE html PUBLIC "-//W 3 C//DTD XHTML 1. 1//EN" "http: //www. w 3. org/TR/xhtml 11/DTD/xhtml 11. dtd"> • When an document type declaration or an entity declaration gives a public identifier, this must be followed by a URI, which acts as a fall back for locating the entity. 119
Format of Parsed External Entities n A parsed external entity may optionally start with a text declaration. The rest of the file should be Content, such as may appear in an XML element. • The replacement text for an external entity is the unprocessed content (minus text declaration). n An external DTD subset may optionally start with a text declaration. The rest of the file should be Declarations, such as may appear in an internal DTD subset. • We will see that external DTD subsets allow a couple of processing options beyond those allowed in internal subsets. 120
Text Declaration n n It is recommended to start a parsed external entity with a text declaration. The text declaration has syntax: <? xml …optional version declaration… encoding=Literal ? > i. e. syntax is identical to an XML declaration, except that now it is the encoding declaration that is mandatory (and no standalone declaration is allowed). • It is allowed for the entity character encoding to be different from the document that references it. 121
XHTML Example n A document entity: <!DOCTYPE html PUBLIC "-//W 3 C//DTD XHTML 1. 1//EN" "http: //www. w 3. org/TR/xhtml 11/DTD/xhtml 11. dtd" [ <!ENTITY myfooter SYSTEM "myfooter. xml" > ]> <html> <body> This is one page in a complicated Web site. &myfooter; </body> </html> n An external entity “myfooter. xml”: <hr/>Maintained by <a xlink: href="mailto: dbcarpen@indiana. edu"> Bryan Carpenter</a>. 122
Combining DTD Subsets n n The example above is illustrative—sadly no known Web browser will recognize this kind of DTD. In principle it shows how one can use both an external subset and an internal subset in the same DTD. • We will see shortly that one can go further with this using parameter entities. • Although the internal subset appears after the reference to the external subset, the internal subset is processed first. If there are conflicting declarations of entities, the internal subset takes precedence. • In general if an entity or attribute is declared more than once, declarations after the first are ignored. 123
Unparsed Entities n n n Unparsed entities are declared in a similar way to parsed entities, but used in a completely different way. The XML parser does not include the contents of the entity in the XML data, it just forwards a reference to the application, e. g. as a URI. An unparsed entity must be annotated with information about what type of content it holds (or, equivalently, what kind of application can process it). • This notation is also passed to the application. n We give the syntax on the following slides—there is a quite lot of syntax with modest payoff. 124
Notation Declarations n Before using a notation to label an unparsed entity, its notation is declared. Declaration syntax is quite free, e. g. the following are legal: <!NOTATION myformat > <!NOTATION jpeg SYSTEM "image/jpeg" > <!NOTATION perl SYSTEM "/usr/bin/perl" > <!NOTATION Name PUBLIC Public-ID > • According to the specification, public and system identifiers may “locate a helper application capable of processing data”. • Some authors interpret that they may be, say, MIME types. • A declared notation may also be the target of a processing instruction, or specified in the value of an attribute of type NOTATION or NOTATIONS. 125
Declaring and Using Unparsed Entities n Declaration similar to parsed external entities, except it ends with “NDATA Name” where Name is a notation. E. g. : <!ENTITY notafake SYSTEM "notafake. jpg" NDATA jpeg > n The only way to reference a declared unparsed entity is in an attribute declared of type ENTITY or ENTITIES, e. g. if we have: <!ATTLIST figure source req ENTITY #REQUIRED > then the name notafake can finally be used as follows: <figure source="notafake"/> 126
Parameter Entities n General entity references are expanded when they appear to in the document content. They are not useful for abstracting sections within a DTD. • The only time general entities are expanded while processing a DTD is if they appear in the default value of an attribute. n There is a separate “macro-expansion” mechanism that is designed for use within DTDs. This uses parameter entities. 127
Declaration and Reference n Declaration of parameter entities follows a syntax similar to parsed general entities, except the name is preceded by a “%” sign, e. g. : <!ENTITY % my-dec '<!ENTITY me "Bryan" >' > This declares the internal parameter entity my-dec. Its value is the declaration of the general entity me. n To actually insert the declaration of me, later in the DTD, one would use a parameter entity reference, which looks like an entity reference except it uses “%”, e. g. : %my-dec; 128
External Parameter Entities n Like general entities, parameter entities may be external, e. g. <!DOCTYPE my-root [ <!ENTITY % my-type SYSTEM "my-type. dtd"> %my-type; ]> is for all intents and purposes equivalent to: <!DOCTYPE my-root SYSTEM "my-type. dtd"> 129
Attribute Groups n n Parameter entities can be used to modularize a DTD, for example by splitting it over multiple files which may be selectively included (c. f. conditional sections, described shortly). Another common use is to collect together lists of related attribute definitions that appear in more than one attribute list, e. g. <!ENTITY % image-atts "source CDATA #REQUIRED height NMTOKEN #IMPLIED"> … <!ATTLIST image %image-atts; > <!ATTLIST figure %image-atts; caption CDATA #IMPLIED > 130
Restrictions n Some reasonable restrictions are placed on references to parameter references, to ensure their replacement text is a “complete” fragment of DTD syntax. • This is similar to the requirement that a parsed general entity must expand to a “complete” fragment of markup. n There is a more ad hoc requirement that in an internal DTD subset any parameter references must expand to one or more complete DTD declarations • Example of previous slide would be illegal in internal subset because %image-atts; isn’t a complete declaration. • This restriction does not apply inside external parameter entities referenced from the internal subset. 131
Parameter Entity Expansion n n When a parameter entity is referenced anywhere in the DTD except in the literal text of another entity declaration, extra space characters are added at the ends of the replacement text. E. g. , this will not work: <!ENTITY % my-prefix "MYAPP_" > <!ELEMENT %my-prefix; Root (#PCDATA) > as element “name” expands to “ MYAPP_ Root”, with illegal space. Following is OK: <!ENTITY % my-prefix "MYAPP_" > <!ENTITY % my-root "%my-prefix; Root" > <!ELEMENT %my-root (#PCDATA) > 132
Conditional Sections n Two special kinds of section can appear in an external DTD subset: <![IGNORE[ Text ]]> <![INCLUDE[ Declarations ]]> n These are designed for use in the following the idiom: <![%my-control; [ … conditional Declarations … ]]> where my-control is the name of a parameter entity defined earlier with value IGNORE or INCLUDE. • For example my-control can be declared in the internal subset, affecting what parts of an external DDT are effective. • The IGNORE section has the unusual property that other sections can nest within it. Thus also convenient for “commenting out”. 133
Processing Miscellany n The XML parser will present character data extracted from element bodies and attributes to the application. It does this after suitable processing. E. g. • Character references may be replaced by actual character. • CDATA sections may be replaced by their contents. • Entity references may be expanded. n Various other normalizations take place. • Every CR LF sequence, or CR not followed by LF, is replaced by a single LF. • A validating parser will normalize the values of attributes according to their type (next slide). 134
Normalization of Attribute Values n For every attribute value, the following normalizations are applied by a validating parser: • Convert every literal white space character (Line Feed, etc) to a space character (#x 20). • Expand character references • Expand entity references (and apply normalizations above to replacement text). n For attributes that have type other than CDATA, all leading or trailing spaces are then removed. Also sequences of spaces are replaced by single spaces. • Note that these transformations generally will not be applied by a non-validating parser (so the application will see different data). 135
Standalone Documents n Various markup declarations affect the data passed to the application. • E. g. declarations of entities, default values of attributes, types of attributes affecting normalization, … n n The XML declaration may include standalone="yes" if all those markup declarations are in the internal DTD subset (and moreover not in parameter entities). Note this restricts only external declarations, not external entities: somewhat mysteriously, a document that is “standalone” can have references to external entities, provided they are declared internally. 136
xml: space and xml: lang n If these attributes are used or declared, the specification restricts what values they may take. • xml: space may only be declared with an enumerated type restricted to one or both of “default” and “preserve”, and may only be specified with one or other of these values. • xml: lang may only be specified with a value that is a standard IETF RFC 3066 language tag (see also ISO-639). n Intended uses are to specify handling of white space, and the natural language of the document. This handling is up to the application: a parser will likely forward these the same as other attributes, and they don’t affect its handling of text. 137
Conclusion n n We have covered everything in the basic XML specification, but there is still a long way to go with XML. Coming up: • • The Document Object Model Namespaces in XML Introductory XPath XML Schema 138
The Document Object Model: Programming Interfaces 139
The Document Object Model n n n The Document Object Model or DOM is a set of W 3 C specifications that define standard ways to access parts of an XML or HTML document from within a program. This addresses, for example, how a Java program extracts the data from an XML document, after the document has been processed by an XML parser. But the origins of the DOM lie in Java. Script programming for interactive and dynamic Web pages. 140
Dynamic HTML Example <html> <head> <script language="javascript"> function append. Text() { para. Node = document. get. Elements. By. Tag. Name("p") [0] ; text. Node = para. Node. first. Child ; text. Node. node. Value += " Ouch!" ; } </script> </head> <body> <p>Hello. </p> <form> <input type="button" value="Push Me" onclick="append. Text()"/> </form> </body> </html> 141
The DOM from Java. Script n n The DOM represents an XML (or HTML) document as a series of nodes. In the example, the Java. Script method append. Text() is called when the user clicks on the button labeled “Push Me”. • This method identifies the DOM node representing the <p> element, extracts the nested text node, and modifies the data (the text) associated with that text node. • The function get. Elements. By. Tag. Name() is a method defined in the DOM, and first. Child and node. Value are node properties in the DOM. 142
A Document is a Tree n Here is how an example fragment of HTML and can be thought of as a tree. 143
DOM Nodes n n In the DOM, one builds the document tree out of a set of Node objects Each Node object has a set of capabilities (properties and methods) and implements specific interfaces. Node Node Node Node Node 144
Node Types in Core DOM n n DOM APIs are defined as a series of interfaces (this term meant in a similar way to how Java uses it). Node itself is an interface, and the following interfaces extend Node: • Document: node representing entire document, including prolog. • Document. Fragment: convenience node, representing an incomplete document. • Element: node representing an element. • Attr: node representing an attribute attached to an element. • Text: node representing “character data”. • Comment: node representing a comment. 145
Further Node Types n Nodes on previous slide cover requirements of HTML. Following extra node types are needed to represent general XML: • Entity. Reference: node representing an entity reference. • Entity: node representing content of a parsed external entity. • Processing. Instruction: node representing a processing instruction. • CDATASection: node representing a CDATA section. • Notation: node representing a notation declared in the DTD. 146
Constants The Node Interface Properties Methods 147
Remarks n DOM interfaces are traditionally defined in the OMG’s Interface Definition Language. • IDL interfaces can be mechanically converted to interface definitions for languages like Java, ECMAScript, C++, … n By design, the Node interface is sufficiently generic that any DOM tree can be manipulated using the methods and properties of this interface alone. • Use of the more specialized derived interfaces—Element, Attribute, etc—is at programmer option. Those interfaces provide more type-safety, and various type-specific convenience methods and properties. 148
node. Name, node. Value Properties n n Each node type has different rules for values of some of the properties—most importantly node. Name and node. Value. The attributes property is only relevant for an element node. Node Type 149
Node Children n The following node types may have children, as indicated, in the DOM tree: • Document: Document. Type, Element (maximum of one), Processing. Instruction, Comment. • Document. Fragment: Element, Text, CDATASection, Entity. Reference, Processing. Instruction, Comment. • Element: Element, Text, CDATASection, Entity. Reference, Processing. Instruction, Comment. • Attr: Text, Entity. Reference. • Entity. Reference: Element, Text, CDATASection, Entity. Reference, Processing. Instruction, Comment. • Entity: Element, Text, CDATASection, Entity. Reference, Processing. Instruction, Comment. 150
The Document Interface n The Document interface has some indispensable factory methods for creating other node types: 151
Levels of the DOM n W 3 C recognizes 4 stages of evolution of the DOM, called levels: • Level 0: Legacy HTML “DOM” features from Navigator 3. 0 and IE 3. 0. No W 3 C specification. • Level 1: Specification completed 1998. Basic tree structure for HTML and XML: 152
Level 2 DOM n Specification completed 2000. Notably added support for Events. Also XML Namespaces, DOM representation of Cascading Style Sheets, and new facilities for manipulating the tree. 153
Level 3 DOM n Specification still in progress. It will add, for example, support for XPath search operations on a DOM tree, and formally define the mapping between DOM and XML Infoset. Load/Save standardizes parsing/serialization APIs. 154
Using Xerces n n Earlier we used the dom. Writer sample application from the Xerces project to validate XML documents. In the following slides we illustrate how to use the Xerces parser from your own Java program. • We will use the conventional Java JAXP API to control parsing. • In future the Load/Save features of Level 3 DOM are likely to be the preferred API (Xerces already provides an experimental implementation). • To control certain aspects of parsing, it may be necessary to use the org. apache. xerces. parsers. DOMParser implementation class directly. 155
Checking Well-Formedness import javax. xml. parsers. Document. Builder; import javax. xml. parsers. Document. Builder. Factory; import org. w 3 c. dom. Document; import java. io. File; public class My. Checker { public static void main(String [] args) throws Exception { File source = new File(args [0]) ; Document. Builder. Factory factory = Document. Builder. Factory. new. Instance(); factory. set. Namespace. Aware(true) ; Document. Builder builder = factory. new. Document. Builder(); } } Document document = builder. parse(source) ; 156
Remarks n In JAXP one first obtains a parser factory, then uses the factory to create a parser instance (called a document builder in JAXP). • Behaviors of the parser (e. g. whether namespace-aware, validating/non-validating, …) are controlled by setting properties of the factory before creating the instance. n Actual parsing is done by the parse() method, which returns a DOM instance (Document node). • Example above does nothing if document is well-formed; prints an exception if it is not. 157
Validating public static void main(String [] args) throws Exception { File source = new File(args [0]) ; Document. Builder. Factory factory = Document. Builder. Factory. new. Instance(); factory. set. Namespace. Aware(true) ; factory. set. Validating(true) ; Document. Builder builder = factory. new. Document. Builder(); builder. set. Error. Handler(new My. Error. Handler()) ; } Document document = builder. parse(source) ; 158
Simple Error Recovery import org. xml. sax. * ; class My. Error. Handler implements Error. Handler { public void warning(SAXParse. Exception e) throws SAXException { System. out. println(e. get. Message()); } public void error(SAXParse. Exception e) throws SAXException { System. out. println(e. get. Message()); } } public void fatal. Error(SAXParse. Exception e) throws SAXException { System. out. println(e. get. Message()); System. exit(1); } 159
Processing the DOM n n The two previous examples ended by creating a Document object, then did nothing with it. The following example summarizes several important Java DOM classes and methods in one recursive method that extracts all useful information in the elements, attributes, and text nodes of a document. • Assumes a declaration like import org. w 3 c. dom. * ; is in effect. • After parsing the document, this method could be invoked from main() by: process(document. get. Document. Element()) ; 160
Recursive Processing of DOM static void process(Node node) { System. out. println(node. get. Node. Name()) ; switch(node. get. Node. Type()) { case Node. ELEMENT_NODE: Named. Node. Map attributes = node. get. Attributes() ; for(int i = 0 ; i < attributes. get. Length() ; i++) process(attributes. item(i)) ; Node. List children = node. get. Child. Nodes() ; for(int i = 0 ; i < children. get. Length() ; i++) process(children. item(i)) ; break ; } } case Node. TEXT_NODE: case Node. ATTRIBUTE_NODE: System. out. println(node. get. Node. Value()) ; 161
Interactive XML n n n We have been discussing use of the DOM in Java as a convenient intermediate representation for handling XML data. This is different from the original application of the DOM, illustrated in our introductory dynamic HTML example. In that application, Java. Script event handlers manipulate the DOM embedded in a Web browser, dynamically altering the text displayed by the browser. This technique can also be applied to XML documents that have a visual representation, for example to Scalable Vector Graphics documents. Following screen captures are from www. svgarena. org. 162
XML Chess 163
XML Solitaire 164
Namespaces in XML 165
Motivation n XML DTDs allow us to define a set of customized element, attribute, and entity names to support a particular application. But in a document of any complexity one may want to mix and match markup from different application domains. In particular in a Web page authored using XHTML tags one might want to embed pictures represented with SVG elements; in a technical Web page one might want to embed mathematical formulae represented using Math. ML elements. 166
Namespaces n n n Typically different XML vocabularies are designed by different committees, and it is impractical to avoid name clashes. In general we can’t simply merge the DTDs—unclear this would be desirable anyway: loss of modularity. The Namespaces in XML specification address this issue by allowing each application to define names in a particular namespace. A single XML document can then include markup from more than one namespace. 167
Relation to the XML Spec n n n The namespaces specification was released a year after the XML 1. 0 specification. It is formally 100% compatible with the XML specification—it “merely” puts some restrictions on the document form. Significant practical issues arise. DTDs—set in stone in the XML/SGML standards—are not an ideal match for namespaces: DTDs can be used with namespaces, but it takes some hacks. • The likely intention was to move away from DTD validation towards XML Schema validation, which is fully namespace compatible. So far this transition is incomplete? 168
Qualified Names n n The simple general idea is that names occurring in document instances may be prefixed in a way that identifies their name space. Here is an (imaginary!) example including SVG tags in an XHTML document: <body> <h 1>What are circles really like? </h 1> This is what a circle really looks like: <object> <svg: svg> <svg: circle cx='150' cy='50' r='25'/> </svg: svg> </object> </body> 169
Name Syntax n n n In the namespaces specification use of the colon, “: ”, in XML names is reserved. Namespaces qualified names have at most one colon; if there is a colon it separates the prefix from the local part: Prefix: Local-part In the example above we had qualified SVG names svg: circle Here the prefixes are svg and the local parts are svg and circle respectively. HTML names had no prefix, only local parts. 170
Namespace Names n Each namespace (e. g. the XHTML namespace or the SVG namespace) itself has a “name”. • This is neither the prefix appearing on qualified names, nor in general a legal XML name! n Instead a namespace name is an IRI. • An IRI is an Internationalized Resource Identifier: it is a generalization of a URI allowing non-ASCII characters. n E. g. the name of the XHTML 1. 0 namespace is: http: //www. w 3. org/1999/xhtml and the name of the SVG 1. 1 namespace is: http: //www. w 3. org/2000/svg 171
Use of URIs n n The use of URIs (or IRIs) for namespace names is quite confusing. Though namespace names follow the syntax of resource identifiers, the namespace is not a resource; there need not be any resource at the location—all that is relevant is the sequence of characters in the URI. This string of characters identifies the namespace. One day this odd situation may be resolved to allow applications to automatically find schema at namespace URIs—see for example www. rddl. org. For now namespace names and schema locations are two independent IRIs. 172
Defining Prefixes n In principle an instance document can choose any convenient prefix to use within that document • Though an existing DTD may restrict the choice. n A prefix is declared in an attribute named xmlns: Prefix. The scope is the element the declaration appears on, and any nested elements. Value specified for attribute must be the namespace name. E. g. : <object xmlns: svg='http: //www. w 3. org/2000/svg' > <svg: svg> <svg: circle cx='150' cy='50' r='25'/> </svg: svg> </object> 173
Default Namespaces n n If a “vocabulary” has been defined in a namespace, a document instance must acknowledge this, even if prefixes aren’t needed. Use the attribute xmlns to declare a default namespace. E. g. all XHTML documents should start: <html xmlns="http: //www. w 3. org/1999/xhtml" > <head> … </html> 174
Scope of Declarations n So far as the namespaces specification is concerned: • Declarations of prefixes and default namespaces can appear on any elements in a document. Scope is the declaration is limited to the element it appears on. • Different namespaces can be the default in different elements. • The same prefix could be used to represent different namespaces in different parts of the document, or vice versa (these are bad ideas!) n DTDs may limit these possibilities. Default namespace declarations don’t affect attribute names. Interpretation of an un-prefixed attribute name is determined by the element it is attached to. • Means attribute is in same namespace as element? No! 175
Namespaces and DTDs n n n DTDs are not “namespace aware”. So far as DTDs are concerned a qualified name with a colon is just an atomic XML name. A partial workaround is to make all names introduced in DTD declarations parameter entity references, then factor out the prefix as a single parameter entity that can be set in a document’s internal DTD subset. This is ugly, and it doesn’t solve all problems. • E. g. DTDs still won’t recognize an equivalence between names in the same namespace, if they have the same local name but different prefixes: so far as DTDs are concerned they are different XML names. 176
Namespaces and XML Schema n n Later lectures discuss XML Schema—one of the alternative validation frameworks to DTDs—in detail. You will probably find the issues associated with namespaces become much more concrete once XML Schema (which are fully namespace-aware) are understood. 177
Elementary XPath 178
XML Path Language n The XML Path Language, or XPath, is a language for addressing nodes of an XML document. • Understand “nodes” as in the DOM (although technically there are minor differences). n n XPath is an important part of XML Schema, XSLT, and XPointer, and is used as a query language in some XML databases. In simple cases an XPath expression looks like a UNIX path name, with nested directory names replaced by nested element names, so for example: • “/” corresponds to the document node. • Expressions may be absolute (relative to /) or relative to some context node. 179
Simple Examples n The XPath: /planets n represents a document element called planets. The XPath: /planets/planet/density n represents the set of all elements named density that are directly nested in any element named planet that is directly nested in a document element planets. The XPath: /planets/planet/* represents all elements directly nested in an element planet directly nested in document element planets. 180
Example Document is a simplified version of an example given the in the book Inside XML, by Holzner. Highlighted node is evaluation of /planets <? xml version="1. 0" encoding="UTF-8"? > <planets> <planet> <name>Mercury</name> <mass>0. 0553</mass> <day units="days">58. 65</day> <radius units="miles">1516</radius> <density>0. 983</density> </planet> <name>Venus</name> <mass>0. 815</mass> <day units="days">116. 75</day> <radius units="miles">3716</radius> <density>0. 943</density> </planet> <name>Earth</name> <mass>1</mass> <day units="days">1</day> <radius units="miles">3960</radius> <density>1</density> </planets> 181
Example Highlighted nodes are evaluation of /planets/planet/density <? xml version="1. 0" encoding="UTF-8"? > <planets> <planet> <name>Mercury</name> <mass>0. 0553</mass> <day units="days">58. 65</day> <radius units="miles">1516</radius> <density>0. 983</density> </planet> <name>Venus</name> <mass>0. 815</mass> <day units="days">116. 75</day> <radius units="miles">3716</radius> <density>0. 943</density> </planet> <name>Earth</name> <mass>1</mass> <day units="days">1</day> <radius units="miles">3960</radius> <density>1</density> </planets> 182
Example Highlighted nodes are evaluation of /planets/planet/* <? xml version="1. 0" encoding="UTF-8"? > <planets> <planet> <name>Mercury</name> <mass>0. 0553</mass> <day units="days">58. 65</day> <radius units="miles">1516</radius> <density>0. 983</density> </planet> <name>Venus</name> <mass>0. 815</mass> <day units="days">116. 75</day> <radius units="miles">3716</radius> <density>0. 943</density> </planet> <name>Earth</name> <mass>1</mass> <day units="days">1</day> <radius units="miles">3960</radius> <density>1</density> </planets> 183
Types of XPath Expressions n n n In full generality XPath expressions can evaluate to various kinds of thing (including numbers, Booleans, and strings) but one is usually interested in expressions that evaluate to node-sets. A node-set is a collection of nodes in an XML document. In general he nodes that can appear in a node-set include: • element nodes, attribute nodes, • text nodes (plain text children of some element), root nodes, namespace nodes, processing instruction nodes and comment nodes. n We will only discuss the first two cases. 184
Location Paths and Location Steps n n The most important kind of XPath expression is the location path. In general a location path consists of a series of location steps separated by the slash “/”. • Compare to a UNIX path, where the individual “step” is the name of an immediately nested subdirectory or file, or a wildcard (*), or a move up into the parent directory (. . ), etc. 185
Steps and Context Nodes n n While a UNIX path may be relative to some current directory, an XPath expression is generally evaluated relative to some context node. Starting from this context node of the path, the location path takes a series of steps in various possible “directions”, e. g. • into the set of child elements of the current element, • into the set of attributes of the current element, • into the set of siblings of the current element, etc. n n This “direction” is called the axis of the step. Each individual step has its own context node, determined by preceding steps in the path. 186
Syntax for Location Steps n The commonest example of a location step—analogous to a UNIX directory name—is an XML element name. • This should be the name of an element that is an immediate child of the context node. n Recalling this example: /planets/planet it has two steps, planets and planet. • Actually this is an example of what is called abbreviated syntax. In the full, unabbreviated, syntax, that location path would be: /child : : planets/child : : planet child being the name of the axis. In this brief exposition we only cover abbreviated syntax. 187
Abbreviated Syntax for Steps n The location step Name selects some element nodes: the children of the context node called Name. • “*” selects all children of the context node. n n n The location step “. ” represents the context node. The location step “. . ” represents the parent node of the context node. The location step @Name represents an attribute node— the attribute of the context node named Name. • @* selects all attributes of the context node. n A blank location step effectively selects the context node or any descendant element, e. g. • //Name selects all Name elements in the document. • . //Name selects all Name descendants of the context node. 188
<? xml version="1. 0" encoding="UTF-8"? > <planets> <planet> <name>Mercury</name> <mass>0. 0553</mass> <day units="days">58. 65</day> <radius units="miles">1516</radius> <density>0. 983</density> </planet> <planet> Highlighted nodes are <name>Venus</name> evaluation of <mass>0. 815</mass> <day units="days">116. 75</day> /planets/planet/radius/@units <radius units="miles">3716</radius> <density>0. 943</density> </planet> <name>Earth</name> <mass>1</mass> <day units="days">1</day> <radius units="miles">3960</radius> <density>1</density> </planets> Example 189
Example Highlighted nodes are evaluation of //mass <? xml version="1. 0" encoding="UTF-8"? > <planets> <planet> <name>Mercury</name> <mass>0. 0553</mass> <day units="days">58. 65</day> <radius units="miles">1516</radius> <density>0. 983</density> </planet> <name>Venus</name> <mass>0. 815</mass> <day units="days">116. 75</day> <radius units="miles">3716</radius> <density>0. 943</density> </planet> <name>Earth</name> <mass>1</mass> <day units="days">1</day> <radius units="miles">3960</radius> <density>1</density> </planets> 190
<? xml version="1. 0" encoding="UTF-8"? > <planets> <planet> <name>Mercury</name> <mass>0. 0553</mass> <day units="days">58. 65</day> <radius units="miles">1516</radius> <density>0. 983</density> </planet> <planet> If the context node is the <name>Venus</name> second <planet/> element, <mass>0. 815</mass> the Highlighted nodes are <day units="days">116. 75</day> <radius units="miles">3716</radius> evaluation of <density>0. 943</density> */@units </planet> <name>Earth</name> <mass>1</mass> <day units="days">1</day> <radius units="miles">3960</radius> <density>1</density> </planets> Example 191
Unions n n The operator “|” forms the union of two node sets. For example: //radius | //density represents the set of all radius elements and all density elements in the document. 192
Example Highlighted nodes are evaluation of //radius | //density <? xml version="1. 0" encoding="UTF-8"? > <planets> <planet> <name>Mercury</name> <mass>0. 0553</mass> <day units="days">58. 65</day> <radius units="miles">1516</radius> <density>0. 983</density> </planet> <name>Venus</name> <mass>0. 815</mass> <day units="days">116. 75</day> <radius units="miles">3716</radius> <density>0. 943</density> </planet> <name>Earth</name> <mass>1</mass> <day units="days">1</day> <radius units="miles">3960</radius> <density>1</density> </planets> 193
XPath and Namespaces n If the elements or attributes named an XPath expression belong to a namespace, then in general those names must have a namespace prefix. • You cannot use a default namespace in an XPath expression. n When XPath expressions are embedded in XML documents (which is normal), the prefixes for the qualified names are scoped in the usual way—by namespace declarations (i. e. xmlns: Prefix attribute specifications) in the XML. 194
Summary n In its full generality, XPath is a fairly complex (but powerful) language for addressing subsets of the nodes of an XML document. • It incorporates extensive computational and filtering functionalities that we have not described here. n The subset we covered in this lecture is nevertheless useful in itself. • And happens to include all the XPath needed to understand XML Schema. 195
- Slides: 195