The HDF Group HDF Project Update Mike Folk

The HDF Group HDF Project Update Mike Folk, Elena Pourmal And the HDF ESDIS Project Team The HDF Group April 18, 2012 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 1 www. hdfgroup. org

Outline • What’s up with The HDF Group • Review ESDIS activities • Maintenance, QA and support 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 2 www. hdfgroup. org

WHAT’S UP WITH THE HDF GROUP? 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 3

The HDF Group • Dedicated to supporting HDF and its users • Non-profit company since 2006 • At U of Illinois National Center for Supercomputing Applications from 1988 -2006 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 4 www. hdfgroup. org

Data challenges addressed by HDF Need to organize complex collections of data lat | lon | temp ----|----E Se xp| 12 | 923 ria erim 3. 1 93 l 7 N ent N te: 892|um 4. 2 15 S |Co. Dn a 24 0 ber otes 3 : tan fig /13 : ur /0 d a a 9 3. 6 17 | rd 21 ti | 3 on : Long term data preservation Efficient, scalable storage and access 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 5 www. hdfgroup. org

Members of the HDF support community 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 6 www. hdfgroup. org

Revenues by source Other Govt & Academic 25% commercial 32% NASA & NOAA 43% 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 7 www. hdfgroup. org

Revenue changes 2010 -2011 2010 2011 48% 43% 31% 32% 25% 20% commercial 4/17/2012 NASA & NOAA Other Govt & Adademic HDF AND HDF-EOS WORKSHOP XV 8 www. hdfgroup. org

The HDF Group 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 9 www. hdfgroup. org

The HDF Group Services • Helpdesk and Mailing Lists • Standard Support • Consulting • Training • Enterprise Support • Special Projects 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 10 www. hdfgroup. org

Downloads of HDF 4, HDF 5, HDFView 33, 591 29, 701 2010 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 2011 11 www. hdfgroup. org

HDFView Binaries Downloads 2010 2011 16, 120 13, 131 3, 132 3, 679 920 1, 187 Linux 4/17/2012 Windows Mac Intel HDF AND HDF-EOS WORKSHOP XV 65 Solaris 12 www. hdfgroup. org

HDF 5 Binaries Downloads 4, 324 3, 977 2010 2011 5, 436 4, 777 676 707 Linux Windows Mac Intel 117 344 77 Free. BSD AIX • Other platforms are on the order of 1 - 10 downloads 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 13 www. hdfgroup. org

HDF 4 Binaries Downloads 2010 2011 1017 674 534 512 Linux Windows • Other platforms are on the order of 1 - 10 downloads 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 14 www. hdfgroup. org

CMMI APPRAISAL AND PROCESS IMPROVEMENT 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 15 www. hdfgroup. org

CMMI Appraisal and Process Improvement CMMI: Capability Maturity Model Integration 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 16 www. hdfgroup. org

ESDIS ACTIVITIES 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 17

The ESDIS project • HDF development work • Code maintenance • HDF Support • Studies, analyses, etc. 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 18 www. hdfgroup. org

HDF-EOS WEBSITE 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 19 www. hdfgroup. org

HDF-EOS Website • Improved the Quality of Comprehensive Examples. HDF-EOS Website • Added new products in Comprehensive Examples. http: //hdfeos. org • Added forum feed in the main page. 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 20 www. hdfgroup. org

New products covered by examples • GOSAT/ACOS • Aquarius • Cloud. SAT • Ocean Productivity NPP GOSAT/ACOS 4/17/2012 HDF AND HDF-EOS WORKSHOP XV Aquarius 21 www. hdfgroup. org

HDF-EOS Examples web stats 7/22/2010 Examples Announced 4/17/2012 2/1/2012 HDF AND HDF-EOS WORKSHOP XV 22 www. hdfgroup. org

Forum Feed in the Main Page 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 23 www. hdfgroup. org

HDF NETCDF OPENDAP 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 24 www. hdfgroup. org

4/17/2012 HDF AND HDF-EOS WORKSHOP XV 25 www. hdfgroup. org

HDF 5 NASA products and net. CDF-4 Use HDF 5 • Aura • OMI, HIRDLS, MLS, TES • Aquarius • ACOS • MEa. SUREs • SMAP • ICESat-2 4/17/2012 • OMI, HIRDLS, MLS, TES • MEa. SUREs • GSSTF, Sea. Wi. FS, Ozone Zonal Means • Future Want net. CDF-4 accessibility • GSSTF, Sea. Wi. FS • Future • ICESat-2 HDF AND HDF-EOS WORKSHOP XV 26 www. hdfgroup. org

Net. CDF 4 -friendly efforts • Work with net. CDF-4 developers and users • Net. CDF-4 • Augmentation • eos 52 nc 4 • Test net. CDF-4 daily • OPe. NDAP 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 27 www. hdfgroup. org

(See “Mapping project Update”) HDF 4 FILE CONTENT MAPS 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 28 www. hdfgroup. org

MAINTENANCE, QUALITY ASSURANCE, SUPPORT 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 29 www. hdfgroup. org

The HDF Group GMQS 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 30 www. hdfgroup. org

GMQS: General Maintenance, QA, and Support GMQS ACTIVITIES 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 31 www. hdfgroup. org

4/17/2012 HDF AND HDF-EOS WORKSHOP XV 34 www. hdfgroup. org

GMQS ACTIVITIES USER SUPPORT HELPDESK STATISTICS 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 35 www. hdfgroup. org

Number of issues – HDF 4 vs. HDF 5 1000 900 800 700 600 579 500 hdf 5 400 300 hdf 4 200 100 63 0 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 36 www. hdfgroup. org

HDF 4 issues by category 120 100 2005 2006 2007 2008 2009 2010 2011 80 60 40 20 0 build 4/17/2012 library util java and doc HDF AND HDF-EOS WORKSHOP XV apps general 37 www. hdfgroup. org

HDF 5 issues by category 350 300 2005 2006 2007 2008 2009 2010 2011 250 200 150 100 50 0 build 4/17/2012 library util java doc HDF AND HDF-EOS WORKSHOP XV apps general 38 www. hdfgroup. org

GMQS ACTIVITIES USER SUPPORT COMMUNITY ENVOLVEMENT 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 39 www. hdfgroup. org

The HDF Group hdf-forum@hdfgroup. org 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 40 www. hdfgroup. org

hdf-forum members help with • Release testing • Maintaining CMake build systems on platforms beyond Windows • Answering questions • The HDF Group’s Help. Desk focuses on ESDIS and other paying customers while referring users to FORUM for difficult topics that require domain knowledge or very specific HDF 5 usage • Securing funding, especially for parallel HDF 5 www. hdfgroup. org

Most discussed hdf-forum topics • • Parallel questions and performance Windows including. NET Compound datatypes Searching for data in HDF 5 files How to organize data in the HDF 5 files Fortran and C++ interfaces Bug reports www. hdfgroup. org

HDF-FORUM Usage hdf-forum emails 1800 1600 1400 1200 1000 800 600 400 200 0 1717 785 hdf-forum emails 144 2007 4/17/2012 1334 1314 2008 2009 2010 HDF AND HDF-EOS WORKSHOP XV 2011 43 www. hdfgroup. org

GMQS Activities HDF PRODUCT MAINTENANCE 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 44 www. hdfgroup. org

Issues and their Priorities • Must Fix • Fix after “Must Fix” • Data corruption • Portability • Backward and Forward Compatibility • Funded Request • • Power User Request Tools Library issues Build Infrastructure • When resources permit • Wrappers • HL Libraries • Other Need your input on priorities! 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 45 www. hdfgroup. org

Maintenance Releases 2011 – 2012 May 2011 Nov Dec HDF 4 HDF 5 4. 2. 7 1. 8. 8 Mar May 4. 2. 7 patch 1 Aug Nov Dec-Jan 2013 Code freeze for 4. 2. 9 4. 2. 8 1. 8. 9 H 4 to. H 5 Java Products Feb 2012 1. 8. 10 Code freeze for 2. 2. 2. 1 2. 8 2. 9 Future releases 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 46 www. hdfgroup. org

GMQS Activities HDF PRODUCTS RELEASES HIGHLIGHTS 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 47 www. hdfgroup. org

HDF 4. 2. 7 • Released in February 2012 • New features • More functions to support H 4 mapping project • Support for Linux PPC 64 with IBM XL Fortran • Minor bug fixes and docmentation improvements 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 48 www. hdfgroup. org

HDF 4. 2. 7 -patch 1 • Released in March 2012 • Fixes configuration problems for compilers with “-” in the name • HDF 4. 2. 7 source code/binaries NOT AFFECTED 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 49 www. hdfgroup. org

Preview of HDF 4. 2. 8 and 4. 2. 9 • HDF 4. 2. 8 • Improvements to support HDF 4 mapping project • Port to Mac OS 10. 7. * (Lion) • HDF 4. 2. 9 • Improve portability by stressing “self-configuration” • Clean HDF 4 issues database • Finalize transition to CMake on Windows (no MS VS project files in the source code!) 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 50 www. hdfgroup. org

HDF 5 1. 8. 7 • Released in May 2011 • New features • Added “silent make mode” to simplify output during builds • Allow dimension size to be 0 (no data can be written); don’t confuse with H 5 S_NULL (empty) • Improved performance by allowing caching files open through external links • Added several verbose levels to h 5 diff • Added an option to enable error stack in h 5 dump • Improved Fortran H 5 LT functions to handle arrays of 4 to 7 dimensions (before up 3 D arrays only) 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 51 www. hdfgroup. org

HDF 5 1. 8. 8 • Released in November 2011 • Added support for Fortran 2003 • Simplified and enhanced many existing routines • Added support for new routines (e. g. , functions with callbacks) • Enabled support for all kinds of INTEGER and REAL • Efficient reading/writing of HDF 5 compound datatypes http: //www. hdfgroup. org/HDF 5/doc/fortran/New. Features_F 2003. pdf • Added Fortran wrappers for Dimension Scale APIs 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 52 www. hdfgroup. org

HDF 5 1. 8. 8 • Released in November 2011 • Improved VFD layer interoperability between Windows and Linux • Improved parallel library by taking advantage of special collective I/O and complex derived datatype MPI functionality • Improved h 5 diff functionality • Improved h 5 repack to handle object references stored in the HDF 5 attributes • It is safe to use h 5 repack on net. CDF-4 files now 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 53 www. hdfgroup. org

Preview of HDF 5 1. 8. 9 • Coming in May 2012 • New function • H 5 LTpath_valid to check if path exists in an HDF 5 file • Tools improvements • H 5 dump allow * in filenames • H 5 dump can display attributes with “/” and datasets with “[“ in their names • H 5 repack considers chunking layout when writing datatsets by hyperslabs • Removed defects from several “corner cases” that cause file corruption or seg faults 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 54 www. hdfgroup. org

Major Improvements • h 5 dump • Show attributes containing "/" for "-a" option • Support wildcard in the filename • h 5 repack • 100 x speedup for some cases involving chunking • h 5 diff • Add options to show different levels of information • Add flag to exclude objects from comparison • Major bug fixes for many tools 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 55 www. hdfgroup. org

GMAS Activities PLATFORM SUPPORT ISSUES 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 56 www. hdfgroup. org

HDF 4 Platforms Supported OS Compilers Linux 2. 6 PPC 64 GNU C and Fortran 4. 4. 6 and IBM XL Fortran V 13 Linux 2. 6 Cent. OS-5 GNU C and Fortran 4. 1. 2 Intel C and Fortran v. 12 PGI C and Fortran v. 11 Linux 2. 6 x 86_64 GNU C and Fortran 4. 1. 2 Intel C and Fortran v. 12 PGI C and Fortran v. 11 Linux Debian, Fedora, SUSE, Ubuntu GNU C and Fortran (default) Sun. OS 5. 10 Sun C 5. 9 and Fortran 8. 3 SGI Altix Intel C and Fortran v. 11 Windows XP, 7 32/64, Cygwin VS 2008, 2010, Intel 10 -11, GNU C and Fortran Mac OS X Intel 10. 6. 8 32/64 GNU C 4. 2. 1 and gfortran -bit 4. 6. 1; Intel C and Fortran 12 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 57 www. hdfgroup. org

HDF 5 Platforms Supported OS Compilers Same as for HDF 4 AIX 5. 3 IBM XL C 10. 1 and Fortran 12. 1 IBM Blue Gene/P IBM compilers Cary Linux PGI C, C++ and Fortran v. 11. 7 Linux Red Hat Enterprise Intel C and Fortran 12. 0 Windows Vista 32/64 VS 2008, 2010, Intel 10 -11 Mac OS X Intel 10. 7. 0 32/64 GNU C 4. 2. 1 and gfortran -bit 4. 6. 1 Open. VMS 8. 3 4/17/2012 HP C, C++ and Fortran HDF AND HDF-EOS WORKSHOP XV 58 www. hdfgroup. org

HDF 4 and 5 Platforms to drop OS 4/17/2012 Compilers Windows Vista, XP(? ) VS 2008, Intel 10, 11 We will use Cmake for building HDF software on Windows Open. VMS HP C, C++ and Fortran HDF AND HDF-EOS WORKSHOP XV 59 www. hdfgroup. org

HDF 4 and 5 Platforms to add OS Compilers Mac OS X 10. 7. * GNU and Intel Compilers Windows 8 VS 2011 Cygwin (? ), Min. GW (? ) Default compilers ? ? We are using virtualization very successfully. Can add any Linux or Windows flavors. Just let us know! 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 60 www. hdfgroup. org

GMQS Activities HDF SOFTWARE EVOLUTION 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 61 www. hdfgroup. org

HDF 4 Software Evolution Themes • Add support for H 4 Mapping project • Make HDF 4 library “self-configurable” • Improves portability • Reduces maintenance cost • Clean-up the code 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 62 www. hdfgroup. org

HDF 4 Quotes • How we documented the code in the last century: • Store calibration information. What is the formula? Good question –GV • Perhaps someone with more time can look into this later. -QAK • Hmm, not working yet? . . . -QAK • This is horribly inefficient, but the separation-of -powers gets really mucked up if we wait till later. . . –Anonymous • Ifdef NOT_YET, NOT_NOW, NOT_USED 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 63 www. hdfgroup. org

HDF 5 Software Evolution Themes • • • Concurrent access Remote Access Parallel I/O performance Real-time write performance Support for high level libraries 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 64 www. hdfgroup. org

New features in the works • Saving space (development completed) • Persistent File Free Space tracking/recovery (1. 10. 0) • Saving time (taking more time) • Asynchronous I/O • Allow an application to proceed while the HDF 5 library performs I/O (1. 10. 0) • File image • Create and read in-memory HDF 5 files without requiring I/O operations (1. 8. 9) 4/17/2012 HDF AND HDF-EOS WORKSHOP XV www. hdfgroup. org 65

New features in the works • Saving time (taking even more time) • Metadata aggregation (1. 10. 0) • Improves I/O by aggregating small pieces of HDF 5 metadata • Allocation MD in page size blocks in a file, perform I/O in pages 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 66 www. hdfgroup. org

New features in the works • Saving files when disaster strikes (1. 10. 0) • Journaling • Journal metadata changes saved in a file • H 5 recover tool to restore metadata in a file • Single Writer/Multiple Readers (SWMR) • Allows simultaneous reading of HDF 5 file while the file is being modified by another process • H 5 watch tool completed • Provides fault tolerance aspects for a file; if writer crashes the file is in the consistent state. 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 67 www. hdfgroup. org

New features in the works • By popular demand: • Object compare API and tool • Based on a formal definition of the HDF 5 objects comparison • Avoids ambiguity and features creep (as with h 5 diff) • Emphasis on flexibility and efficiency • Control over reporting “differences” • Compare compressed data without uncompressing it 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 68 www. hdfgroup. org

Research/Prototyping • Virtual Object Layer • Leveraging HDF 5 Data Model without enforcing HDF 5 file format • Abstraction layer that allows different plugins for accessing data • Examples • Different file formats (net. CDF, HDF 4, GRIB, FITS) • Directories and files on a file system • Memory objects • Remote objects 4/17/2012 HDF AND HDF-EOS WORKSHOP XV www. hdfgroup. org 69

Virtual Object Layer (VOL) HDF 5 Application HDF 5 API VOL Plugin Layer Net. CDF HDF 5 Library FS Cloud dimensions: lon = 2 ; lat = 2 ; ref_time = UNLIMITED ; // (48 currently) variables: float lon(lon) ; lon: long_name = "longitude" ; lon: FORTRAN_format = "f 6. 1" ; lon: units = "degrees_east" ; float lat(lat) ; lat: long_name = "latitude" ; lat: FORTRAN_format = "f 6. 1" ; net. CDF file 4/17/2012 lat: units = "degrees_north" ; HDF 5 file Directories and files on FS HDF AND HDF-EOS WORKSHOP XV Objects in a cloud www. hdfgroup. org 70

HPC Improvement - Partnerships Improve performance of parallel apps including net. CDF-4 Improve performance of parallel apps Add features anticipating exascale systems 4/17/2012 HDF AND HDF-EOS WORKSHOP XV www. hdfgroup. org 71

GMQS Activities JAVA PRODUCTS MAINTENANCE AND EVOLUTION 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 72 www. hdfgroup. org

HDF Java Products Highlights • All major HDF 5 1. 8 API functions were added to HDF 5 JNI • Unit tests were added to all major HDF 5 JNI functions 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 73 www. hdfgroup. org

Things in the pipeline for HDF-Java • Add CMake to compile and install hdf-java products • Continue bug fixes and enhancements • HDF-Java 2. 9 release with HDF 4. 2. 8 and HDF 5 1. 8. 10 (December 2012) 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 74 www. hdfgroup. org

The HDF Group Thank You! 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 75 www. hdfgroup. org

Acknowledgements • This work was supported by cooperative agreement number NNX 08 AO 77 A from the National Aeronautics and Space Administration (NASA). • Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author[s] and do not necessarily reflect the views of the National Aeronautics and Space Administration. 4/17/2012 HDF AND HDF-EOS WORKSHOP XV 76 www. hdfgroup. org