HDF for the Ages Keeping HDF Data Accessible

HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term 1 HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30 -Dec. 2, 2005

Outline • The Long Term Problem • A Few Ideas • Discussion 2 HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30 -Dec. 2, 2005 by R. Duerr Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30 -Dec. 2, 2005

The Digital Preservation Challenge “digital objects require constant and perpetual maintenance, and they depend on elaborate systems of hardware, software, data and information models, and standards that are upgraded or replaced every few years” NSF and Library of Congress, August 2003 3 HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30 -Dec. 2, 2005 by R. Duerr Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30 -Dec. 2, 2005

What is a Good Long-Term Archive Format? • Per a recent paper by Mike Folk and Bruce Barkstrom § § § 4 Ease of archival storage Ease of archival access Usability Data scholarship enablement Support for data integrity Maintainability and durability HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30 -Dec. 2, 2005 by R. Duerr Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30 -Dec. 2, 2005

What Makes a Good File Format? • Per Eric Raymond, in “The Art of Unix Programming”, Addison-Wesley, 2004 § § Transparency Interoperability Extensibility Storage economy • He argues that the best general purpose file format is text • He also argues that the only good justification for binary data is with very large data sets 5 HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30 -Dec. 2, 2005 by R. Duerr Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30 -Dec. 2, 2005

Disaster-Proofing Your Data • If you can’t keep a data set as text, then at least keep the representational information in a human readable format (preferably right with the data itself) 6 HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30 -Dec. 2, 2005 by R. Duerr Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30 -Dec. 2, 2005

HDF and HDF-EOS as an Archive Format • Neither are optimized for archival § Not a text-based format § Does not enforce inclusion of semantic metadata § Meant to be used with the HDF* libraries 7 HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30 -Dec. 2, 2005 by R. Duerr Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30 -Dec. 2, 2005

Towards the Future? • Discussions with Don Sawyer, Lou Reich, Mike Folk and others at NCSA • A Few Ideas § Specifying an HDF-archive format a’la PDF-A § Text wrapper/encoding for HDF § Tools to translate to/from an archive format • Mechanisms? § CODATA § International workshop § Proposals 8 HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30 -Dec. 2, 2005 by R. Duerr Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30 -Dec. 2, 2005

Discussion • Any suggestions for moving forward? • Questions? • Concerns? 9 HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30 -Dec. 2, 2005 by R. Duerr Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30 -Dec. 2, 2005

Contact Info • For questions about data preservation, etc. contact me at rduerr@nsidc. org • For information about NSIDC data products, programs, etc. see http: //nsidc. org or contact our user services at nsidc@nsidc. org 10 HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30 -Dec. 2, 2005 by R. Duerr Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30 -Dec. 2, 2005