Metadata Principles Practices Challenges Sandra Payette Digital Library

Metadata: Principles, Practices, Challenges Sandra Payette Digital Library Research Group Cornell University payette@cs. cornell. edu

Metadata CREATOR: Plato TITLE: The Republic Metadata is structured data about data that facilitates discovery, use, and management of the data to which it refers. Image 1 Image 2 Image 3 cdrom 1 cdrom 2 Access Control List

Metadata enables … n n Resource Discovery Resource Presentation and Navigation Rights Management Preservation We must support all these functions, but also recognize that these artificial categories for metadata.

General Principles: Metadata n n Designing one kind (e. g, descriptive) without consideration of others (e. g. , usage, rights, preservation) can compromise utility and interoperability over time The metadata problem should be approached analytically and methodically, not ad hoc Metadata is expensive; must cost-benefit Common metadata sets often represent a flattened or simplified view of reality.

Challenge of Interoperability n Semantic n Structural n Syntactic Media: CD-ROM Media: 35 mm film (refers to physical storage medium for digital image) (refers to original source)

Challenge of Interoperability n n Semantic Structural Date: 10 -6 -99 Date: 6 -10 -99 Type: image/tiff Type: TIFF 4. 0 Author: Sam Brown Author: Brown, S n Syntactic

Challenge of Interoperability n Semantic n Structural <META name="DC. creator" content=”Junger, S"> <META name="DC. title" content=”The Perfect Storm"> n Syntactic Creator: Junger, S Title: The Perfect Storm

Syntaxes for Expressing Metadata n HTML META tags n n n XML and SGML n n Communities can define own vocabularies (DTDs/Schema) Separation of structural description from rendering info Increasing support for XML in browsers and other software Resource Description Framework (RDF) using XML n n Embed metadata in HTML documents Search engines can extract it Express complex relationships between resources Proprietary n n TIFF headers Vendor-specific data structures

Functional Views of Metadata n n Resource Discovery Resource Presentation and Navigation Rights Management Preservation

Resource Discovery on Web Challenges google Collection “A” Web Server lycos excite Text-based search engines n n Image DB n n Scale: much content is not visible or not found, so it’s not indexed Format: much content non -textual (e. g. , images!) Context: lacking! (causing precision error in search) Rights: valuable content hidden behind firewalls

Resource Discovery on Web Collection “A” search Web Server engine Welcome to Collection ‘A” Search: Web Browser google lycos Image DB Metadata Store excite Context established - customized metadata created at source

Resource Discovery: Dublin Core n n 15 descriptive elements Facilitates simple resource discovery on Web Cross-disciplinary, international, genreindependent Very active and accepted “standard” n n 100+ major projects 20+ countries http: //purl. oclc. org/dc/

Resource Discovery: Dublin Core Caveats n n Designed for simple discovery, don’t force it to do more than it can (rights, preservation) Qualification – can compromise meaning and interoperability right w y a l p. r : creato dc Hamlet “Shakespeare” dc: creator. birthplace “Stratford” Roll-up to root element and … “dc: creator = Stratford” ? ?

Open Archives Initiative (OAi) n n Specification of simple metadata harvesting protocol to facilitate interoperability Adoption of unqualified Dublin Core Element Set as required metadata Common XML container format for metadata packaging Institutional backing of n n CNI (Coalition for Networked Information) DLF (Digital Library Federation) http: //www. openarchives. org/

Exposing and Exchanging Metadata using OAi Image Collections OAi harvester Electronic journals metadata OPAC E-texts

OAi Registered Repositories ar. Xiv OCLC Thesis and Dissertations Perseus Digital Library Phys. Net Oxford Text Archive Library of Congress -- American Memory Cog. Prints Humboldt University MIT Thesis Linguistic Data Consortium Resource Discovery Network … and more

Resource Discovery: Principles n n n Decide what you want to be visible to which search services (Site home page? Specific items? ) Adopt standard metadata (e. g. , DC) for crossdomain visibility of resources Develop context-specific metadata to meet collection requirements Design/adopt a metadata model that allows for graceful co-existence of multiple metadata sets Express or expose metadata in syntax that promotes interoperability (e. g. , XML, RDF)

Functional Views of Metadata n n Resource Discovery Resource Presentation and Navigation Rights Management Preservation

Structural Metadata n Facilitates n n n Direct access to key points in objects Browsing objects Navigation (e. g. , turning pages) Identification of relationships (e. g. , parent/child) Access to different formats (e. g. , TIFF, GIF, PDF) Where is it? n n n ASCII text files in directories Relational databases Embedded in documents or surrogates (e. g. XML, SGML)

Structural metadata can be a byproduct of data management Level 1 (journals) harpurs atlantic Relational Database Level 2 (volumes) V 0001 Level 3 i 0001 (issues) Level 4 (articles) v 0002 0001. tif 0002. tif v 0003 i 0002 0003. tif File System atlantic (dir) V 0002 (dir) I 0001 (dir) 0001. tif 0002. tif

Structure via Document Encoding n n Current trend is to use mark-up languages to encode the structure of document objects SGML n n Text Encoding Initiative (TEI/TEI-Lite DTDs) Memory of World DTD for rare library materials Encoded Archival Description (EAD) XML n n n Goettingen Digitization Center (XML and RDF) Making of America II (archival object DTD, plus EAD) METS (under development, support by DLF)

MOA 2 DTD: Structural “Binding” of Images <Struct. Map> <div N='1' TYPE='Book' LABEL='Diary of Patrick Breen one of the Donner Party 1846 -57…> <fptr FILEID='HRJ 1' MIMETYPE='image/jpeg' /> <fptr FILEID='LRG 1' MIMETYPE='image/gif' /> <fptr FILEID='T 1' MIMETYPE='text/sgml' TAGID='titlepage' /> <div N='1' TYPE='Entry' LABEL='Friday Nov. 20 th 1846 [Page 1]'> <fptr FILEID='HRJ 2' MIMETYPE='image/jpeg' /> <fptr FILEID='LRG 2' MIMETYPE='image/gif' /> <fptr FILEID='T 1' MIMETYPE='text/sgml' TAGID='entry 1'/> </div> <div N='2‘ TYPE='Entry‘ LABEL='Entry sat. 21 st [Page 2]'> <fptr FILEID='HRJ 3' MIMETYPE='image/jpeg' /> <fptr FILEID='LRG 3' MIMETYPE='image/gif' /> <fptr FILEID='T 1' MIMETYPE='text/sgml' TAGID='entry 2'/> </div> Source: sunsite. berkeley. edu/moa 2

Enabling Fine-grained Access to Images Step 1 …. Monday 30 th Snowing fast wind W about 4 or 5 feet deep, no drifts looks as likely to continue as when it commenced no liveing thing without wings can get about transcribe December 1 st Tuesday Still snowing wind W snow about 5 1/2 feet or 6 deep difficult to get wood no going from the house completely housed up looks as likely for snow as when it commenced, our cattle all killed but three or four [of] them, the horses & Stantons mules gone & cattle suppose lost in the Snow no hopes of finding them alive wedns. 2 nd. Continues to snow wind W sun shineing hazily thro the clouds dont snow quite as fast as it has done snow must be over six feet deep bad file this morning …. Image file (TIFF) Text file (ASCII)

Enabling Fine-grained Access to Images Step 2 <div id=‘entry 11’> Monday 30 th Snowing fast wind W about 4 or 5 feet deep, no drifts looks as likely to continue as when it commenced no liveing thing without wings can get about </div> …. Monday 30 th Snowing fast wind W about 4 or 5 feet deep, no drifts looks as likely to continue as when it commenced no liveing thing without wings can get about December 1 st Tuesday Still snowing wind W snow about 5 1/2 feet or 6 deep difficult to get wood no going from the house completely housed up looks as likely for snow as when it commenced, our cattle all killed but three or four [of] them, the horses & Stantons mules gone & cattle suppose lost in the Snow no hopes of finding them alive wedns. 2 nd. Continues to snow wind W sun shineing hazily thro the clouds dont snow quite as fast as it has done snow must be over six feet deep bad file this morning <div id=‘entry 12’> December 1 st Tuesday Still snowing wind W snow about 5 1/2 feet or 6 deep difficult to get wood no going from the house completely housed up looks as likely for snow as when it commenced, our cattle all killed but three or four [of] them, the horses & Stantons mules gone & cattle suppose lost in the Snow no hopes of finding them alive</div> mark-up <div id=‘entry 13’> wedns. 2 nd. Continues to snow wind W sun shineing hazily thro the clouds dont snow quite as fast as it has done snow must be over six feet deep bad file this morning</div> …. Text file Encoded text file

Enabling Fine-grained Access to Images Step 3 <div id=‘entry 11’> Monday 30 th Snowing fast wind W about 4 or 5 feet deep, no drifts looks as likely to continue as when it commenced no liveing thing without wings can get about </div> <div id=‘entry 12’> December 1 st Tuesday Still snowing wind W snow about 5 1/2 feet or 6 deep difficult to get wood no going from the house completely housed up looks as likely for snow as when it commenced, our cattle all killed but three or four [of] them, the horses & Stantons mules gone & cattle suppose lost in the Snow no hopes of finding them alive</div> <div id=‘entry 13’> wedns. 2 nd. Continues to snow wind W sun shineing hazily thro the clouds dont snow quite as fast as it has done snow must be over six feet deep bad file this morning</div> Encoded text file parse and render Title Page Entry 1 Entry 2 Entry 3 Entry 4 Entry 5 … … File viewed in browser

EAD: Encoded Archival Description n n DTD for SGML mark-up of descriptive finding aids (e. g. , inventories, registers, indexes, and guides) Provides more detail about a collection than in typical catalog record Facilitates access - “drill down” into collection Potential international standard Maintained jointly by Library of Congress and Society of American Archivists (SAA)

EAD Example http: //sunsite. Berkeley. EDU/Cal. Heritage/

Presentation and Navigation: Principles n n Decide how fine-grained you want the access experience to be Determine the cost-benefit of creating this amount of structural metadata Design/adopt a model (esp. DTD/Schema) that can be shared Be prepared to express in XML, since it is poised to become standard on Web

Functional Views of Metadata n n Resource Discovery Resource Presentation and Navigation Rights Management Preservation

Rights and Security Metadata n Facilitates n n n Access control Protection of intellectual property rights Transactions (e-commerce) Security (protect materials from attack) Monitoring

Digital Library Federation (DLF) Requirements n n Must account for perspectives of publishers, intermediaries, users Must not compromise privacy of users Must accommodate ambiguity, as found in copyright (e. g. , fair use) Metadata relationships n n n Descriptive (about objects) User profiles Rights declarations (Policies)

Expressing Policies for Automated Enforcement n Rights Metadata efforts (XML/RDF oriented) n n n <indecs> Digital Object Identifier (DOI) Policy Language initiatives n Extensible Rights Markup Language (Xr. ML) (www. xrml. org) n n Key. Note (www. crypto. com/trustmgt/kn. html) Cornell’s PSLang (Language-based security)

Modeling the “rights” problem: <indecs> n n Supported by copyright societies, publishers, recording industry Fundamental Entities are modeled n n n Creation Person Agreement Inter-relationship of descriptive and rights metadata Event-oriented (time and transactions) Model will be expressed in RDF Schema

<indecs> Sample Encoding for Rights Metadata [Event. Identifier=License No 12345] [Event. Type=Agreement] [Person=John Smith] [Role=Granter. Of. Right] [Person=Bill Brown] [Role=Grantee] [Event=Event No 11111 [Role=Permitted Act] … [Event. Identifier=11111] [Event. Type = Usage Event] [Person=Bill Brown] [Role=Downloader] [Manifestation=Text. File 1 “Make Money…” … Source: http: //www. indecs. org/pdf/model 3. pdf

Functional Views of Metadata n n Resource Discovery Resource Presentation and Navigation Rights Management Preservation

Preservation Metadata Image File Attributes: • formats • versions • compression Process Data: • creation date/time • equipment used Descriptive Data: • author • title • publish date Structure Data: • pagination • sub-groups Image Attributes: • resolution • bit depth • orientation Rights Data: • Expiration dates • Copyright info • source statements

Electronic Records Community Perspective n Metadata requirements for preserving n Six-layer metadata model evidence n n n Unique identifier Resource discovery metadata Data structure Terms and conditions Provenance information Source: Pittsburgh Project, www. lis. pitt. edu/~nhprc

Unique Identifiers n n n Globally unique names (e. g. , URN specification) Name is permanent, location changes Resolution services to locate the object Implementations: PURL, Handles, DOI Can create your own local resolution system Unique Identifier: cnri. dlib/april 97 -payette Naming Authority URL: Item Name http: //www. somewebserver. org/somedirectory/somefile

Conceptual Model Preservation Metadata View Technical View Structural View Rights View Descriptive View A model for preservation should accommodate different metadata views

Model Projects for Preservation Metadata n Cedars (UK) n n n Developed extensive preservation metadata set Evaluated all major initiatives, and influenced by RLG and Pittsburgh work Using OAIS model for distributed archives http: //www. leeds. ac. uk/cedars/metadata. html National Library Australia n n Metadata to manage collections, objects, files Desired output of a metadata system http: //www. nla. gov. au/preserve/pmeta. html http: //www. nla. gov. au/padi

Wrap Up: Questions for setting metadata requirements n n How will users locate digital image objects? How will users interact with digital image objects or collections? What policies are necessary to protect rights and provide access controls How will the program assure permanence of digital materials?

Wrap Up… Best Practices for Metadata n Well-conceived data models n n n n Understand functional requirements metadata will support Modularity in design (provides flexibility and extensibility) Prevent data anomalies (remember DC example? ) Well-structured metadata (machine-interpretable) Express or expose metadata using standard syntax (e. g. , XML) Define “standard” community semantics and rules (e. g. , XML Schema, DTDs) Anticipate need to interoperate n Exposure of metadata in standard syntax and semantics