Reference Model for an Open Archival Information System

Reference Model for an Open Archival Information System (OAIS) ESIP Summer Meeting John Garrett – ADNET Systems at NASA/GSFC 2009 -07 -09

Topics (time permitting) • OAIS Reference Model • Follow-on/Related Standards – Producer-Archive Interface Methodology Abstract Standard (PAIMAS) – Repository Audit and Certification Metrics (draft) • Requirements for Bodies Providing Audits (draft) – Producer-Archive Interface Specification (PAIS) (draft) – XML Formatted Data Unit (XFDU) Contributors: Don Sawyer, Daniele Boucon, Lou Reich, David Giaretta and many others involved in CCSDS Archiving and Packaging standards development

OAIS Reference Model Home • Consultative Committee for Space Data Systems (CCSDS) • International group of space agencies • Develop variety of science discipline-independent standards • Became working body for an ISO TC 20/ SC 13 about 1990 TC 20: Aircraft and Space Vehicles SC 13: Space Data and Information Transfer Systems – http: //www. ccsds. org/ – Ensured broad participation, including traditional archives, libraries, companies (Not restricted to space communities; all participation was welcomed!)

In the Beginning: OAIS Reference Model CCSDS 650. 0 -B-1 Reference Model for an Open Archival Information System (OAIS) (ISO 14721: 2003) http: //public. ccsds. org/publications/archive/650 x 0 b 1. pdf OAIS Mandatory Responsibilities: 1. 2. 3. 4. 5. 6. OAIS Information Model Negotiating and accepting information Obtaining sufficient control of the information to ensure long-term preservation Determining the "designated community" Ensuring that information is independently understandable Following documented policies and procedures Making the preserved information available OAIS Functional Model OAIS RM OAIS Environment and Data Flows

OAIS Responsibilities • Negotiates and accepts information from information producers • Obtains sufficient control to ensure long-term preservation • Determines which communities (designated) need to be able to understand the preserved information • Ensures the information to be preserved is independently understandable to the Designated Communities • Follows documented policies and procedures that ensure the information is preserved against all reasonable contingencies • Makes the preserved information available to the Designated Communities in forms understandable to those communities

OAIS Archival Information Package Description derived from Archival Information Package (AIP) e. g. , Information supporting customer searches for AIP Content Information Packaging Information delimited by e. g. , How to find Content information and PDI on some medium further described by e. g. , • Hardcopy document • Document as an electronic file together with its format description • Scientific data set consisting of image file, text file, and format descriptions file describing the other files Preservation Description Information (PDI) e. g. , • How the Content Information came into being, who has held it, how it relates to other information, and how its integrity is assured

Preservation Description Information (PDI) • Reference Information – Provides one or more identifiers, or systems of identifiers, by which the Content Information may be uniquely identified – Bibliographic Description, Persistent IDs • Provenance Information – Describes the source of Content Information, who has had custody of it, what is its history – Logs of migrations • Context Information – Describes how the Content Information relates to other information outside the Information Package – Pointers to related collections • Fixity Information – Protects the Content Information from undocumented alteration – Digital signatures, Checksums

View of an OAIS Environment • Producer provides the information to be preserved • Management sets overall OAIS policy • Consumer seeks and acquires preserved information of interest Producer OAIS (archive) Management Consumer

OAIS Functional Entities Preservation Planning P R O D U C E R Descriptive Info. Data Management Descriptive Info. Ingest Access queries result sets orders SIP Archival Storage AIP Administration MANAGEMENT SIP = Submission Information Package AIP = Archival Information Package DIP = Dissemination Information Package DIP C O N S U M E R

External Data Flow View Producer Submission Information Packages OAIS Archival Information Packages queries result sets Dissemination Information Packages orders Consumer

Conformance • How does an archive conform? – It discharges the set of minimal responsibilities – It supports the basic information concepts that address a definition of information and types of information packages • How do other documents conform? – By using OAIS terms and concepts • Certification Standard in progress

OAIS Update Many improvements including: • Authenticity • Information properties • Risk management • Emulation • Federation

Producer-Archive Interface Methodology Abstract Standard PAIMAS Focus Preservation Planning P R O D U C E R Data Management Descriptive Info. SIP Ingest Archival Storage AIP Administration SIP = Submission Information Package MANAGEMENT AIP = Archival Information Package DIP = Dissemination Information Package Access queries result sets orders DIP C O N S U M E R

PAIMAS Methodology CCSDS 651. 0 -B-1 Producer-Archive Interface Methodology Abstract Standard. (ISO 20652: 2006) http: //public. ccsds. org/publications/archive/651 x 0 b 1. pdf • The Archive Project is broken into 4 main phases: • • Preliminary Phase, Formal Definition Phase, Transfer Phase, Validation Phase. • PAIMAS identifies: • • the phases in the process of transferring information, the objective of the phases, Extensive action tables of actions that must be carried out, the expected results. • PAIMAS is a basis: • for further specialization by a particular community • for the identification of standards and implementation guides, • for identification and development of a set of software tools.

Phase objective PAIMAS phases & relationships to ry ta nta a (d me n t o i en ple at n io em om le) rm t e o u a f gr d, c ed in ved tim a e i p ere sch th rch es o e a el liv ts, s n v e fi be rc e de n e D to u be lem o s e re • e r fe h ft o he t te red a lid sfer a V an ts tr jec ob s n tra l s ua ect t j Ac ob Transferred object files Preliminary Phase Formal Definition Phase Preliminary Agreement Transfer Phase Submission Agreement including Dictionary and Formal Model Validation Phase Anomalies Validation agreement Data ready to archive

Preliminary phase: sub-phases Action table Description

Repository Audit and Certification - Metrics • http: //wiki. digitalrepositoryauditandcertification. org/bin/view • Closing in on public draft that will be submitted to CCSDS and ISO • Builds on previous audit work by TRAC and many others INCLUDED TOPICS ORGANISATIONAL INFRASTRUCTURE GOVERNANCE & ORGANIZATIONAL VIABILITY ORGANIZATIONAL STRUCTURE & STAFFING PROCEDURAL ACCOUNTABILITY &. PRESERVATION POLICY FRAMEWORK FINANCIAL SUSTAINABILITY CONTRACTS, LICENSES, & LIABILITIES DIGITAL OBJECT MANAGEMENT INGEST: ACQUISITION OF CONTENT INGEST: CREATION OF THE AIP PRESERVATION PLANNING AIP PRESERVATION INFORMATION MANAGEMENT ACCESS MANAGEMENT INFRASTRUCTURE AND SECURITY RISK MANAGEMENT Technical Infrastructure Risk Management Security risk management 4. 2. 8 The repository shall verify each AIP for completeness and correctness at the point it is created. Supporting Text This is necessary in order to ensure that what is maintained over the long term is as it should be and can be traced to the information provided by the Producers. Examples of Ways the Repository can Demonstrate it is Meeting this Requirement Description of the procedure that verifies completeness and correctness of the AIPs; logs of the procedure. Discussion The repository should be sure that the AIPs it creates are as they are expected to be by …

PAIS Objectives • Producer-Archive Interface Specification • Provide formal modelling of data objects that are to be transferred from Producer to Archives – XML-based interchange of the model and SIPs • Implementation standard for Producer – Archive Interface – Conformity with the OAIS Reference Model – Conformity with the PAIMAS – Conformity with the XFDU • Aimed mainly at Formal Definition Phase with applicability to Transfer Phase with Validation • Closing in on public draft that will be submitted to CCSDS and ISO possibly by end of year 18

PAIS Basic Entities 19

XFDU Packaging Standard Rationale Technology and Requirements Evolution • • • Physical media -->Electronic Transfer No standard language for metadata--> XML Homogeneous Remote Procedure Call-->CORBA, SOAP Little understanding of long-term preservation-->OAIS RM Record formats-->Self describing data formats New Requirements • • Describe multiple encodings of a data object Better describe the relationships among a set of data objects. Technical Drivers • Use of XML based technologies • Designed to be extensible to include new XML technologies as they emerge • Linkage of data and software • Direct mapping to OAIS Information Models • Support both media and network exchange • Support for multiple encoding/compression on individual objects or on entire package • Mapping to current SFDU Packaging & Data Description Metadata where possible • Maximal use of existing standards and tools from similar efforts

XFDU Conceptual View CCSDS 661. 0 -B-1 XML Formatted Data Unit (XFDU) Structure and Construction Rules. (ISO 13527: 2009) http: //public. ccsds. org/publications/archive/661 x 0 b 1. pdf Open source XFDU Toolkit Library developed as a reference implementation (available at: http: //sindbad. gsfc. nasa. gov/xfdu • • • Interoperability testing completed with ESA XFDU Implementation Partnered with JPL/PDS to establish a NASA Testbed XFDU Briefing on Scalability at the Collaborative Expedition Workshop / Toward Scalable Data Management (available at: http: //colab. cim 3. net/cgibin/wiki. pl? Expedition. Workshop/Toward. Scalable. Data. Management_2008_06_10)