Emerging Standards for Complex Works Howard Besser UCLA

Emerging Standards for Complex Works Howard Besser UCLA School of Education & Information http: //www. gseis. ucla. edu/~howard Besser--CNI/JISC 6/16/00 1

Emerging Standards for Complex Works _ _ _ Background & Context for Standards MOA 2: Structural & Administrative Metadata NISO/DLF: Technical Imaging Standards Identification/provenance Rich Media Longevity Besser--CNI/JISC 6/16/00 2

Key problems we’re facing Discovery Longevity Interoperability Besser--CNI/JISC 6/16/00 3

Traditional Digital Library Model DL DL DL search & presentation user Besser--CNI/JISC 6/16/00 DL search & presentation user 4

Ideal Digital Library Model DL DL search & presentation user Besser--CNI/JISC 6/16/00 user 5

For Interoperability Digital Libraries Need Standards Descriptive Metadata for consistent description Discovery Metadata for finding Administrative Metadata for viewing and maintaining Structural Metadata for navigation . . . Terms & Conditions Metadata for controlling access. . . Besser--CNI/JISC 6/16/00 6

Why are Standards and Metadata consensus important? Managing digital files over time Longevity Interoperability Veracity Recording in a consistent manner Will give vendors incentive to create applications that support this Besser--CNI/JISC 6/16/00 7

Collaborative Metadata Projects Dublin Core NSF/ERCIM Digital Collaboratory OCLC CORC Project Visual Resources Association (VRA) Core Encoded Archival Description (EAD) Computerized Interchange of Museum Information (CIMI) Records Export for Art and Cultural Heritage (REACH) Besser--CNI/JISC 6/16/00 8

CORC--Cooperative Online Resource Catalog _ _ both bib records & webliographies (pathfiinders) supports both AACR 2/MARC and DC began 1/99, scheduled availability 7/00 100 -200 participants – – Academic libraries OCLC networks, special libraries, public libraries, state & national libraries, consortia Besser--CNI/JISC 6/16/00 9

Making of America II Background of the DLF Project Administrative Metadata Structural Metadata Besser--CNI/JISC 6/16/00 10

MOA 2 Goal is Interpoerability Book example Besser--CNI/JISC 6/16/00 11

R DLF Metadata for Interoperability Testbed: the MOA II Project &D Distributed Repositories Transportation, 1869 -1900 Testbed Project Best Practices Structural and administrative metadata Besser--CNI/JISC 6/16/00 12

Previous Projects/Background Library Standards Background UC Berkeley Background Finding Aids EAD SGML EAD “Digital Archives” Besser--CNI/JISC 6/16/00 13

MOA II Classes of Objects Continuous Tone Photos Photo Albums Diaries, journals, letterpress books Ledgers Correspondence Besser--CNI/JISC 6/16/00 14

MOA II Metadata _ Administrative Metadata – _ Structural Metadata – _ for enhancing resource management for reflecting internal hierarchies and relationships btwn parts Raw/Seared/Cooked Besser--CNI/JISC 6/16/00 15

MOA II Behaviors Navigation Display/Print Besser--CNI/JISC 6/16/00 16

MOA II Best practices Use/Users/Collection: Benchmarking Masters vs. Derivatives Scanning Administrative Metadata Structural Metadata. Besser--CNI/JISC 6/16/00 17

Scanning Best Practices _ _ Think about users (and potential users), uses, and type of material/collection Scan at the highest quality that does not exceed the likely potential users/uses/material Do not let today’s delivery limitations influence your scanning file sizes; understand the difference between digital masters and derivative files used for delivery Many documents which appear to be bitonal actually are better represented with greyscale scans Besser--CNI/JISC 6/16/00 _ _ _ Include color bar and ruler in the scan Use objective measurements to determine scanner settings (do NOT attempt to make the image good on your particular monitor or use image processing to color correct) Don’t use lossy compression Store in a common (standardized) file format Capture as much metadata as is reasonably possiple (including metadata about the scanning process itself) 18

Why Scale is important Besser--CNI/JISC 6/16/00 19

Administrative Metadata to uniquely identify a digital resource and manage it over time _ _ _ Information about where the various pieces/versions of the object reside Information to view the digital object Information about the scanning process Besser--CNI/JISC 6/16/00 20

Structural Metadata: that which is relevant to presentation of the digital object to the user _ _ metadata defining the "object”: a book, a diary, a photo album metadata defining the “sub-objects”: pages (physical) or chapters and subheads (intellectual) Besser--CNI/JISC 6/16/00 21

SGML, XML, HTML _ _ TEI for structured humanities text EAD for Finding Aids Besser--CNI/JISC 6/16/00 22

NISO/DLF Image Metadata Workshop Possible Goals Metadata fields Rules for Field Contents (authority control) Core set of necessary fields Syntax for expressing fields and contents (headers) Besser--CNI/JISC 6/16/00 23

Image Metadata Focus on Metadata that may prove helpful for management use preservation . . . Besser--CNI/JISC 6/16/00 24

Image Metadata Break-out Groups: Work Done Characteristics and Features of Images Image Production and Reformatting Features Image Identification and Integrity Besser--CNI/JISC 6/16/00 25

NISO/DLF Image Metadata Workshop (4/99) Image Technical Information : Possible Goals Metadata fields Rules for Field Contents (authority control) Core set of necessary fields Syntax for expressing fields and contents (headers) Besser--CNI/JISC 6/16/00 26

Image Metadata Focus on Metadata that may prove helpful for management use preservation . . . Besser--CNI/JISC 6/16/00 27

Image Metadata Break-out Groups: Work Done Characteristics and Features of Images Image Production and Reformatting Features Image Identification and Integrity Besser--CNI/JISC 6/16/00 28

Image Metadata Elements for Data Dictionary Entries _ _ _ _ _ Element Name Definition (short) of the element name Is the element required? (Identified as: Mandatory, Mandatory if Applicable, Recommended, Optional) How is the value of the element represented? Examples When is this data collected? What is the purpose of this data? Who would the identified users be? How is the metadata used? What other metadata standards reference it? Besser--CNI/JISC 6/16/00 29

Image Metadata Elements for Data Dictionary Characteristics and Features Element List _ _ _ Format Issues: Resolution Issues: Encoding: Compression: Others: Besser--CNI/JISC 6/16/00 30

Image Metadata Elements for Data Dictionary Image Production Element List _ _ _ (Pertaining to the Image) In-image target(s): System target(s), associated with the object: Responsible agent Rationale: Hardware: Software: Besser--CNI/JISC 6/16/00 31

Image Metadata Elements for Data Dictionary Image Production Element List (Pertaining to the Process) _ _ _ _ Format of the image Intrinsic characteristics of the image Identification Provides a means for defining methodology including documentation and rationale Who is involved with the file? Who created the image file? Who commissioned the creation of the image file (i. e. , the chartering entity), as opposed to: Who is the responsible agency? Who is the owner? Where What When: necessary dates including: capture date/time, modification Checksum Navigational aid Encoding tools Besser--CNI/JISC 6/16/00 32

Image Metadata NISO/DLF Image Metadata: In Progress _ Data Dictionary for both “Characteristics & Features” and for “Image Production Elements” due end of 6/00 Besser--CNI/JISC 6/16/00 33

Finding Image Origins Besser--CNI/JISC 6/16/00 34

Identification/Provenance (Images) The number of variant forms of a work can be enormous Image Families A digital image frequently has many layers of parentage Information about the parentage that can indicate the quality and veracity of the image (Dublin Core "Source" and "Relation") how to deal with different versions derived from the same scan or different encoding schemes Vocabulary Standards to express this Besser--CNI/JISC 6/16/00 35

The number of variant forms of a work can be enormous different views of the same object different lighting of the same object different scans of the same photo different resolutions different compression schemes different compression ratios different file storage formats different details of the same image . . . Besser--CNI/JISC 6/16/00 36

Image Families

Identification/Provenance how to deal with different versions (browse, hi-res, medium res) derived from the same scan or different encoding schemes (TIFF, PICT, JFIF) Vocabulary Standards to express this – – VRA Surrogate Categories CIMI's "Image Elements” Besser--CNI/JISC 6/16/00 38

Other Metadata _ _ Description of depiction/surrogate (What VRA calls its "Surrogate Categories") Description of original object Rights and Reproduction Information Location Information Besser--CNI/JISC 6/16/00 39

Metadata for Digital Commerce DOI <indecs>- Besser--CNI/JISC 6/16/00 40

<Indecs> formal structure for describing and uniquely identifying intellectual property itself, the people and businesses involved in its trading, and the agreements which they make about it (primarily for publishing, music, and visual arts) will develop high-level specifications for the services that will be required to implement a global IP trading system based on this <indecs> generic data model focus is on encoding rights at a high level, not on resource discovery likely to involve metadata schma registration and directory to allow interoperation of personal identifiers for rightsholders and users supported by EEC DG-13 First meeting July 1999 http: //www. indecs. org/ Besser--CNI/JISC 6/16/00 41

Problems & Potentials of Rich Media_ _ _ Types of Rich Media Technologies and problems Opportunities--a scenario Metadata Indexing Besser--CNI/JISC 6/16/00 42

Some Types of Rich Media _ _ Moving image materials Multimedia Interactive programs Computer art Besser--CNI/JISC 6/16/00 43

After an uphill battle, tech and Tinseltown find common ground (USA Today, 3/3/00) Besser--CNI/JISC 6/16/00 44

Projected Changes: Prospect of digitized movies already has some mourning loss of film (SF Chronicle, 3/5/00) Besser--CNI/JISC 6/16/00 45

Video Technology to Make the Head Spin (NYT 3/2/00) Besser--CNI/JISC 6/16/00 46

ECI - Hole in Space (both) Besser--CNI/JISC 6/16/00 47

ECI - 84 -locations Besser--CNI/JISC 6/16/00 48

ECI - 84 -Community Memory Besser--CNI/JISC 6/16/00 49

ECI - 84 -MOCA Besser--CNI/JISC 6/16/00 50

ECI - Avatars & Humans Besser--CNI/JISC 6/16/00 51

ECI - Avatar Stage Besser--CNI/JISC 6/16/00 52

Complexity of Rich Media _ _ Works often have artistic nature (including video games) Enormous number of elements can, at times, be very important to preserve (pacing, original artifact, elements used to construct the artifact) Too complex to save every one of these aspects for every type of material Importance of saving documentation Besser--CNI/JISC 6/16/00 53

Rich Media Technologies _ _ _ Streaming media vs. Downloaded files Bandwidth and compression Need to offload functions onto clients Besser--CNI/JISC 6/16/00 54

The Inter-relation Problem -Info is increasingly inter-related to other info -How do we make our own Info persist when it points to and integrates with Info owned by others? -What is the boundary of a set of information (or even of a digital object)? Besser--CNI/JISC 6/16/00 55

The Translation Problem Content translated into new delivery devices changes meaning – – – -A photo vs. a painting -If Info is produced originally in digital form in one encoded format, will it be the same when translated into another format? Behaviors Besser--CNI/JISC 6/16/00 56

Problems of Rich Media _ _ _ _ Complexity of formats (storage & compression) Synchronicity between media/streams Pieces and Boundaries Persistent IDs Interactivity Historical context Content Recontextualization (Postmodernism) Besser--CNI/JISC 6/16/00 57

Opportunities--a scenario _ _ _ Huge stable online DB of rich media (Prelinger Archives) Creators create new works that consist mainly of links to and transitions btwn pieces of the rich media DB Works are not really assembled until run-time Securing IP permission may shift from capital-intensive producer to end-user Economics of media production may change drastically Besser--CNI/JISC 6/16/00 58

Structural Metadata for Complex Objects_ _ MPEG 4 SMIL Besser--CNI/JISC 6/16/00 59

Synchronized Multimedia Integration Language (SMIL) _ _ _ For repurposing and reuse in different ways Use XML to reference various pieces in different ways Supported by Realmedia but not Microsoft or Macromedia Besser--CNI/JISC 6/16/00 60

MPEG 4 _ _ Object-oriented Very low level of granularity (even objects vs backgrounds) Scaleable bandwidth use Binary Format for Scenes (BIFS) borrows concepts from VRML Besser--CNI/JISC 6/16/00 61

Indexing of Moving Image Materials _ _ _ Whole works vs. parts of Works MPEG 7 Approaches to segmentation & thumbnail representation Closed caption indexing Audio description indexing Semiotics Besser--CNI/JISC 6/16/00 62

Other Types of Metadata_ _ _ Longevity Identification/Provenance Rights Management Besser--CNI/JISC 6/16/00 63

The Short Life of Digital Info: Digital Longevity Problems Disappearing Information The Viewing Problem The Scrambling Problem The Inter-relation Problem The Custodial Problem The Translation Problem Besser--CNI/JISC 6/16/00 64

The Viewing Problem Digital Info requires a whole infrastructure to view it Each piece of that infrastructure is changing at an incredibly rapid rate How can we ever hope to deal with all the permutations and combinations Besser--CNI/JISC 6/16/00 65

The Scrambling Problem Dangers from: Compression to ease storage & delivery Container Architecture to enhance digital commerce Besser--CNI/JISC 6/16/00 66

The Inter-relation Problem -Info is increasingly inter-related to other info -How do we make our own Info persist when it points to and integrates with Info owned by others? -What is the boundary of a set of information (or even of a digital object)? Besser--CNI/JISC 6/16/00 67

The Custodial Problem How do we decide what to save? Who should save it? How should they save it? – – -methods for later access: emulation, migration, etc. -issues of authenticity and evidence Besser--CNI/JISC 6/16/00 68

The Translation Problem Content translated into new delivery devices changes meaning – – – -A photo vs. a painting -If Info is produced originally in digital form in one encoded format, will it be the same when translated into another format? Behaviors Besser--CNI/JISC 6/16/00 69

Pieces of the Solution (1/2) -We need to insist upon clearly readable standardized ways for digital objects to selfidentify their formats -We should discourage scrambling -We need to better understand information inter-relates to other Info, and what constitutes “boundaries” of Info objects Besser--CNI/JISC 6/16/00 70

Pieces of the Solution (2/2) -People and organizations wishing to make information persist need guidelines of how to go about doing it -We need to better understand how translating from one storage or display format to another affects the meaning of a work -We need to save the “behaviors” of a digital 6/16/00 object, not just it’s “contents” Besser--CNI/JISC 71

Metadata can be the first line of defense Can – – – tell you where the file is (if you can’t find the file) where more info about the file is (if you have the file but most other metadata has become separated) what the file format is what the compression scheme is what application program and version is needed for the file Besser--CNI/JISC 6/16/00 72

Groups Working on the Big Longevity Problem http: //sunsite. Berkeley. EDU/Imaging/Databases/Longevity/ CPA Task Force CPA Study Group Getty “Time & Bits” Conference Internet Archive Long Now Besser--CNI/JISC 6/16/00 73

Migration/Refreshing Impact on evidential value Besser--CNI/JISC 6/16/00 74

Emerging Standards for Complex Works _ _ Howard Besser UCLA School of Education & Information http: //www. gseis. ucla. edu/~howard/image-meta. html _ http: //sunsite. Berkeley. EDU/moa 2/ http: //www. gseis. ucla. edu/~howard/Classes/287 -moving. html http: //www. gseis. ucla. edu/~howard/Classes/287 -mov-index-bib. html _ http: //www. gseis. ucla. edu/~howard/Metadata/UC-May 00/ _ http: //www. getty. edu/gri/standard/intrometadata/ _ http: //sunsite. Berkeley. EDU/Imaging/Databases/#standards _ http: //sunsite. Berkeley. EDU/Longevity/ _ http: //www. ifla. org/II/metadata. htm _ http: //purl. oclc. org/metadata/dublin_core/ _ http: //purl. oclc. org/corc/ _ http: //lcweb. loc. gov/ead/ Besser--CNI/JISC 6/16/00 _ http: //sunsite. berkeley. edu/Metadata/sp 2000. html _ 75

Data Structures: The VRA Core 28 elements specifically for visual resource collections Work Description Categories Visual Document Description Categories http: //www. oberlin. edu/~art/vra/dsc. html Besser--CNI/JISC 6/16/00 76

VRA Core: Work Description Categories Work type Title Measurements Material Technique Creator Role Date Repository name Repository place Besser--CNI/JISC 6/16/00 _ _ _ _ _ Repository number Current site Original site Style/period/group/movem ent Nationality/culture Subject Related work Relationship type Notes 77

VRA Core: Visual Document Description Categories Visual document type Visual document format Visual document measurements Visual document date Visual document owner number Visual document view description Visual document subject Visual document source Besser--CNI/JISC 6/16/00 78

Thesaurus for Graphic Materials designed for subject indexing of pictorial materials, particularly large general collections of historical images for cataloging and retrieval good for general audiences and broad approaches to the material TGM-I: Subject Terms & TGM-II: Genre and Physical Characteristic Terms http: //lcweb. loc. gov/rr/print/tgm/toc. html Besser--CNI/JISC 6/16/00 79

AAT 120, 000 terms for describing objects, textual materials, images, architecture, and material culture from antiquity to present large and complex http: //www. getty. edu/gri/vocabularies/ Besser--CNI/JISC 6/16/00 80

ULAN name authority http: //www. getty. edu/gri/vocabularies/ Besser--CNI/JISC 6/16/00 81

Thesaurus of Geographic Names over 1 million records hierarchical and global throughout history most records include coordinates and descriptive notes Besser--CNI/JISC 6/16/00 82

Semantics/Syntax/Structure _ Semantics – _ Syntax – – _ meaning, as defined by a community to meet their particular needs (DC) a systematic arrangement of data elements for machine processing facilitates the exchange and use of metadata among various applications (HTML, XML, RDF) Structure – a formal arrangement of the syntax with the goal of consistent representation of the semantics (rules defining field contents like 1/11/99) Besser--CNI/JISC 6/16/00 83

Metadata Mapping Crosswalks Resource Description Framework (RDF) Besser--CNI/JISC 6/16/00 84

Crosswalks mapping btwn differing metadata structures eliminate the need for monolithic, universally adopted standards focus on flexibility and interoperatiblity RDF-based metadata registries Besser--CNI/JISC 6/16/00 85

Crosswalk Example Besser--CNI/JISC 6/16/00 86

Resource Description Framework (RDF, spec released 2/99) _ _ _ W 3 C Metadata activity designed to move the Web beyond simple links to semantically-rich relationships btwn resources metadata application using XML as a common syntax for exchange and processing flexible architecture for managing diverse applicationspecific metadata packets that can be processed by machines associates resources, property types, and corresponding values http: //www. w 3. org/RDF/ _ Besser--CNI/JISC 6/16/00 87

RDF _ _ _ Resources (character strings, names, digital objects) Property (“is the author of”) Value resources+properties=relationships many different relationships can be reflected Besser--CNI/JISC 6/16/00 88

XML-encoded RDF _ _ _ <? xml: namespace ns=http: //www. w 3. org/RDF prefix="RDF" ? > <? xml: namespace ns=http: //purl. oclc. org/DC/ prefix="DC" ? > <RDF: RDF> <DC: Creator>Howard Besser</DC: Creator> </RDF: Description> </RDF: RDF> Besser--CNI/JISC 6/16/00 89

Besser--CNI/JISC 6/16/00 90