The e Xtensible Access Method XAM Standard by

The e. Xtensible Access Method (XAM) Standard by Steve Todd (EMC) Presented by Paul Mc. Keown (EMC) at the 4 th Annual Digital Curation Conference December 3, 2008 Edinburgh, Scotland

Purpose This paper was written to introduce XAM technology to the Digital Curation community. Agenda Why XAM? XAM Coding Example Standards-based Digital Archives Vendor plug-fest: Sun, EMC, and HP Opportunities for Digital Curation Research

Why XAM? SNIA (Storage Networking Industry Association) 100 Year Archive Survey Key Problem: Logical and Physical Retention of Fixed Content Logical: Data Formats Physical: Migration to new Digital Archives XAM Solution Strong binding between content and meta-data Vendor-neutral API for storing “fixed content objects” Standards-based export-import between different vendors

The XAM Approach Location-independent Object naming Each ‘Data Object’ (XSet) stored in XAM is assigned a globally unique name (a XUID). These Location Independent Names allow data to migrate within or between XAM storage systems without impacting applications accessing the data. Rich Metadata XAM allows applications to bundle MIME-typed contextual metadata together with application data, facilitating easier data interchange among applications and longer “shelf life” for application data. XAM provides for SQL like Query functionality Pluggable Architecture for Storage System support XAM Storage System vendors can plug their systems into the XAM API by creating a provider for the Vendor Interface Module API. XAM also provides a standardized set of management disciplines and semantics for fixed content, such as retention, query, shredding etc.

XAM Software Architecture Application Program ISV / Custom XAM Toolkit API XAM Toolkit Library XAM API SNIA XAM API Library SNIA VIM API VIM Reference Vim - SNIA VIM API VIM Vendor A VIM Vendor B

XAM Object Model XAM defines 3 Primary Objects XSet Primary storage abstraction in XAM Stores metadata and data in a collection of Fields XSystem Logical Container of XSets Each XSystem Instance represents a connection with a particular XAM Storage System XAM Library Responsible for discovering and managing all VIMs in the XAM application environment Provides Fields to report on and control global XAM attributes (API Revision Level, API Logging Level etc. ) Serves as a factory for XSystem Instances via Connect method

XAM Primary Objects XSystem is an abstraction in the XAM API representing a logical container of XSets. This is distinct from and possibly a subset/superset of the physical XAM Storage System. An XSystem Instance combines an XSystem with an authenticated connection to one or more XAM Storage Systems. An XSystem Instance is equivalent to a XAM Session. In the XAM API, an XSystem Instance is created by calling XAM_Connect with a valid XRI (XSystem Resource Identifier). XSystem instances are used to create, retrieve and delete XSets. XSystem instances also have fields describing the system’s supported capabilities and management policies.

XAM Primary Objects - XSet • XSet – Addressable “Unit of Storage” in XAM Model • To store data in an XSystem an application must: • Create XSet Instance via XSystem • Create/Populate XSet fields with data • Commit the new XSet to persistent storage, saving resultant XUID XSet Fields Properties - “Simple” types (Boolean, Int, Float, String, Date. Time, XUID) - Type checked/enforced by storage system - Manipulated via “Property Get/Set” methods XStreams - Bytestreams, up to 2^64 bytes - Type assumed to be a valid MIME-type, but not checked/enforced by storage system - Manipulated via Posix-style I/O methods (e. g. , open, read, write, close)

XSet Field ‘Binding’ Attribute XSet Fields • When a field is marked ‘Binding’ ( ) it means that this field’s value has a direct correspondence with this XSet’s XUID (the field’s value is relevant for the XSet’s identity) • ‘Non-Binding’ fields may be freely modified within the XSet, just as with traditional read/write storage • However, on an attempt to modify a ‘Binding’ field, the XSystem silently creates a completely new XSet, an identical copy of the original XSet, and assigns it a new XUID; the original XSet must be preserved under the original XUID • Applications are free to decide which XSet fields are ‘Binding’ at the time they are created

XUID – XSet Unique Identifier • XUID is the permanent name for an XSet • • Assigned by XAM Storage System XUIDs are Globally Unique • XUID Native format is binary sequence (10 – 80 bytes) • Base 64 (RFC 2045) recommended for printable interchange • XSet’s XUID has a strict relationship with the XSet’s ‘Binding’ fields • • If a ‘Binding’ field is modified, a new XSet with a new XUID is created upon commit The old XSet is preserved as-is XUID Format 0 1 2 3 4 5 6 7 8 9 10 … 78 79 reserved XUID Vendor OID opaque data (zero) length CRC-16 … data end

XAM Write Example Overview Connect to XSystem Create New XSet Add XSet Metadata Write XSet Data Commit XSet Release Resources XAM 1. 0 Application Program XAM API Library xam. dll VIM API VIM example_vim. dll XSS Protocol XAM Storage System

XAM Write Example – Connect to XSystem xam_string v. XRI = “snia-xam: //example_vim!10. 1. 1. 1”; v. Status = XAMLibrary_Connect(v. XRI, &v. XSystem); XAMLibrary_Connect snia-xam: //example_vim!10. 1. 1. 1 XAM API Library xam. dll VIM example_vim. dll XAM Storage System 10. 1. 1. 1

XAM Connection Authentication Once connected, an XSystem Instance must authenticate XAM uses SASL Authentication Framework Simple Authentication and Security Layer (RFC 4422) PLAIN and ANONYMOUS methods always available More advanced SASL methods (DIGEST-MD 5, SECURID, KERBEROS-V 5) advertised via XSystem property. xsystem. auth. SASLmechanism. list. <mechanism> PLAIN Authentication “PLAIN<name><secret>”

XAM Write Example – Authenticate auth. Data. Length = Build. Auth. Buffer(auth. Buffer, auth. Name, auth. Secret); v. Status = XSystem_Authenticate(v. XSystem, auth. Buffer, auth. Data. Length, &auth. Stream); XStream_Close(auth. Stream); XSystem_Authenticate “PLAIN<name><secret>” XAM API Library xam. dll VIM example_vim. dll XAM Storage System 10. 1. 1. 1

XAM Write Example – Create XSet xset_handle v. XSet; v. Status = XSystem_Create. XSet(v. XSystem, XSET_MODE_MODIFY, &v. XSet); XSystem_Create. XSet XSET_MODE_MODIFY XAM API Library xam. dll VIM example_vim. dll XAM Storage System 10. 1. 1. 1

XAM Write Example – Add XSet Metadata v. Status = XAM_Create. String(v. XSet, “com. example. archive. invoice_id”, TRUE, v. Invoice. ID); XAMLibrary_Connect “com. example. archive. invoice_id” XAM API Library xam. dll VIM example_vim. dll /* field name */ /* binding */ /* value */ XAM Storage System 10. 1. 1. 1

XAM Write Example – Write Binary Data XAM stores binary data in special fields called Streams present POSIX-like write/read interface xstream_handle v. XStream; v. Status = XAM_Create. XStream(v. XSet, “com. example. archive. invoice_image", /* field name */ TRUE, /* binding */ “image/tiff", /* MIME type */ &v. XStream); … Transfer Image Data …. v. Status = XStream_Close(v. XStream); /* completes write*/

XAM Write Example – Writing XStream Data Transferring data from a file to an XStream do { /* Read data from file into buffer. */ v. Read. Length = fread(v. Data. Buffer, sizeof(char), BUFFER_SIZE, v. Input. File); if (v. Read. Length > 0) { v. Position = 0; while (v. Position < v. Read. Length) { /* Write data from buffer to XStream. */ v. Status = XStream_Write(v. XStream, &v. Data. Buffer[v. Position], v. Read. Length - v. Position, &v. Write. Length); v. Position += v. Write. Length; } /* Update accounting info */ v. Total. Write += v. Position; v. Total. Read += v. Read. Length; } } while (v. Read. Length > 0);

XAM Write Content – Commit XSet v. Status = XSet_Commit(v. XSet, &v. XUID); After Commit, application stores XUID return value permenently, for future XSet retrieval. XSet_Commit xset_handle XAM API Library xam. dll VIM example_vim. dll XAM Storage System 10. 1. 1. 1

XAM Write Content – Code Overview // Connect to XSystem v. Status = XAMLibrary_Connect(v. XRI, &v. XSystem); // XSystem Authentication v. Auth. Data. Length = Build. Auth. Buffer(v. Auth. Buffer , s. Auth. Method, s. Auth. Name, s. Auth. Secret); v. Status = XSystem_Authenticate(v. XSystem, v. Auth. Buffer, v. Auth. Data. Length, &v. Auth. Stream); // Close auth stream v. Status = XStream_Close(v. Auth. Stream); // Create XSet v. Status = XSystem_Create. XSet(v. XSystem, XSET_MODE_MODIFY, &v. XSet); // Add some identifying metadata v. Status = XAM_Create. Int(v. XSet, PROPERTY( format_revision), true, 1); v. Status |= XAM_Create. String(v. XSet, PROPERTY( invoice_id), true, v. Invoice. ID); v. Status |= XAM_Create. String(v. XSet, PROPERTY( cust_id), true, v. Customer. Name)); // Write invoice image data using multiple threads and streams … v. Status = write. File. Data(v. XSet, v. Filename, &v. Total. Bytes, &v. Segment. Count); // Additional metadata v. Status = XAM_Create. Int(v. XSet, PROPERTY( image_count), true, v. Segment. Count); // Commit XSet v. Status = XSet_Commit(v. XSet, &v. XUID); // Display resulting XUIDTo. String(v. XUID, &v. XUIDString); printf("Successfully wrote XSet, Base 64 XUID: n %sn", v. XUIDString);

XAM Retention Model Retention is the ability of an XSystem to prevent XSet deletion for an application-specified time period. All XAM Retention policies have 3 primary aspects: Enabled: Has the associated retention been turned on? Duration: How long shall deletion be prevented (in milliseconds)? Start Time: When does the retention clock start counting down? XAM Base Retention Start Time == XSet XUID Time XAM Event Retention Start Time is defined at some future date, triggered by business conditions

Opportunities for Research XAM provides a globally flat namespace. Reduces management burden for configuring large digital archives (potentially no file systems, LUNs, or databases to configure) XAM provides a location independent object model for data. XUIDs are a form of persistent identifier; XAM places no limits on object counts. XAM defines a canonical format for data migration. Standardized data exchange format allows customers to migrate data from one system to another without engaging the applications that reference the data. XAM provides a framework for coupling policy and other metadata together with data. Archival Information Packages can be stored in one XSET and represented with one XUID.

For Additional Information SNIA XAM Initiative XAMI home page - http: //www. snia. org/forums/xam/ XAM API specs -http: //www. snia. org/forums/xam/specs XAM Demo - http: //www. snia. org/forums/xam/flshdemo/1282_SNIA_XAM. htm SNIA Data Management Forum (DMF) DMF home page - http: //www. snia. org/forums/dmf/ DMF Long Term Archive - http: //www. snia. org/forums/dmf/programs/ltacsi/ DMF 100 Year Survey http: //www. snia. org/forums/dmf/knowledge/100 Yr. ATF_Archive-Requirements. Survey_20070619. pdf Steve Todd’s weblog http: //stevetodd. typepad. com/my_weblog/digital_preservation/ http: //stevetodd. typepad. com/my_weblog/centera/ EMC Developer’s Network (Centera) http: //community. emc. com/community/edn/centera