The HDF Group Introduction to HDF 5 Quincey

The HDF Group Introduction to HDF 5 Quincey Koziol The HDF Group Unidata net. CDF Workshop October 28 -29, 2010 October 28, 2010 Netc. DF Workshop 1 www. hdfgroup. org

What is HDF 5? • Open file format • Designed for high volume or complex data • Open source software • Works with data in the format • A data model • Structures for data organization and specification October 28, 2010 Netc. DF Workshop 2 www. hdfgroup. org

HDF = Hierarchical Data Format • HDF 4 is the first HDF • Originally called HDF; last major release was version 4 • HDF 5 benefits from lessons learned with HDF 4 • Changes to file format, software, and data model • HDF 5 and HDF 4 are different • No plans for an HDF 6! October 28, 2010 Netc. DF Workshop 3 www. hdfgroup. org

HDF 5 is like … October 28, 2010 Netc. DF Workshop 4 www. hdfgroup. org

HDF 5 is designed … • for high volume and/or complex data • for every size and type of system (portable) • for flexible, efficient storage and I/O • to enable applications to evolve in their use of HDF 5 and to accommodate new models • to support long-term data preservation October 28, 2010 Netc. DF Workshop 5 www. hdfgroup. org

HDF 5 Technology Platform • HDF 5 data model • The “building blocks” for data organization and specification Let’s looakll… Rec at …. • HDF 5 software • Library, language interfaces, tools • HDF 5 file format • Bit-level organization of HDF 5 file October 28, 2010 Netc. DF Workshop 6 www. hdfgroup. org

HDF 5 Data Model Dataset Group Attribute Link HDF 5 Objects Datatype Dataspace File a. k. a. HDF 5 Abstract Data Model a. k. a. HDF 5 Logical Data Model October 28, 2010 Netc. DF Workshop 7 www. hdfgroup. org

HDF 5 File lat | lon | temp ----|----12 | 23 Se|r. Exper 3. 1 D ial im Nu ent Co ate| 15 | 24 m N nf : 3/1 4. 2 ig 3 be ote ur /0 r: s: a 9 99 17 | 21 |tion 3. 6 37 An HDF 5 file is a container that holds data objects. October 28, 2010 : S 89 tan da Netc. DF Workshop 8 rd 20 3 www. hdfgroup. org

HDF 5 Dataset HDF 5 Datatype Integer 32 bit LE HDF 5 Dataspace Rank 3 Dimensions Dim_0 = 4 Dim_1 = 5 Dim_2 = 7 Specifications for single data element and array dimensions Multi-dimensional array of identically typed data elements • HDF 5 datasets organize and contain “raw data values”. • HDF 5 datatypes describe individual data elements. • HDF 5 dataspaces describe the logical layout of the data elements. October 28, 2010 Netc. DF Workshop 9 www. hdfgroup. org

HDF 5 Dataspaces • Describe the logical layout of the elements in an HDF 5 dataset • NULL • no elements • Scalar • single element • Simple array (most common) • multiple elements organized in a rectangular array • rank = number of dimensions • dimension sizes = number of elements in each dimension • maximum number of elements in each dimension • may be fixed or unlimited October 28, 2010 HDF/HDF-EOS Workshop XIV 10 www. hdfgroup. org

HDF 5 Dataspaces Two roles: Dataspace contains spatial information (logical layout) about a dataset stored in a file • Rank and dimensions • Permanent part of dataset definition Rank = 2 Dimensions = 4 x 6 Partial I/0: Dataspace describes application’s data buffer and data elements participating in I/O Rank = 1 Dimension = 10 October 28, 2010 Netc. DF Workshop 11 www. hdfgroup. org

HDF 5 Dataset & Dataspace Dim_1 = 5 Dim_2 = 7 HDF 5 Dataspace Rank 3 Dim_0 = 4 Dimensions Dim_0 = 4 Dim_1 = 5 Dim_2 = 7 Specifications for array dimensions Multi-dimensional array of identically typed data elements • HDF 5 datasets organize and contain “raw data values”. • HDF 5 dataspaces describe the logical layout of the data elements. October 28, 2010 Netc. DF Workshop 12 www. hdfgroup. org

HDF 5 Datatypes • Describe individual data elements in an HDF 5 dataset • Wide range of datatypes supported • Signed/unsigned Integer • • Float User-defined (e. g. , 13 -bit integer) Fixed and variable-length strings Variable length sequences Arrays Compound (similar to C structs) Enumerated Many more … October 28, 2010 HDF/HDF-EOS Workshop XIV 13 www. hdfgroup. org

HDF 5 Dataset 3 5 12 Datatype: 32 -bit Integer Dataspace: Rank = 2 Dimensions = 5 x 3 October 28, 2010 Netc. DF Workshop 14 www. hdfgroup. org

HDF 5 Dataset with Compound Datatype 3 5 V Compound Datatype: Dataspace: October 28, 2010 int 16 char int 32 V V 2 x 3 x 2 array of float 32 Rank = 2 Dimensions = 5 x 3 Netc. DF Workshop 15 www. hdfgroup. org

HDF 5 Dataset & Datatype HDF 5 Datatype Integer 32 bit LE Specifications for single data element Multi-dimensional array of identically typed data elements • HDF 5 datasets organize and contain “raw data values”. • HDF 5 datatypes describe individual data elements. October 28, 2010 Netc. DF Workshop 16 www. hdfgroup. org

HDF 5 Dataset HDF 5 Datatype Integer 32 bit LE HDF 5 Dataspace Rank 3 Dimensions Dim_0 = 4 Dim_1 = 5 Dim_2 = 7 Specifications for single data element and array dimensions Multi-dimensional array of identically typed data elements • HDF 5 datasets organize and contain “raw data values”. • HDF 5 datatypes describe individual data elements. • HDF 5 dataspaces describe the logical layout of the data elements. October 28, 2010 Netc. DF Workshop 17 www. hdfgroup. org

HDF 5 Data Model: Are we there yet? HDF 5 Objects Group and Link Attribute Dataspace Datatype Dataset File October 28, 2010 HDF/HDF-EOS Workshop XIV 18 www. hdfgroup. org

HDF 5 Attributes • Typically contain user metadata • Have a name and a value • Are associated with HDF 5 objects. • Value is described by a datatype and a dataspace • analogous to a dataset October 28, 2010 HDF/HDF-EOS Workshop XIV 19 www. hdfgroup. org

HDF 5 Groups and Links HDF 5 groups and links organize data objects. / Experiment Notes: Serial Number: 99378920 Date: 3/13/09 Configuration: Standard 3 Every HDF 5 file has a root group Sim. Out Viz Parameters 10; 1000 lat | lon | temp ----|----12 | 23 | 3. 1 15 | 24 | 4. 2 17 | 21 | 3. 6 Timestep 36, 000 October 28, 2010 Netc. DF Workshop 20 www. hdfgroup. org

HDF 5 Technology Platform • HDF 5 data model • The “building blocks” for data organization and specification Let’s look at …. • HDF 5 software • Library, language interfaces, tools October 28, 2010 Netc. DF Workshop 21 www. hdfgroup. org

HDF 5 Home Page HDF 5 home page: http: //hdfgroup. org/HDF 5/ • Latest release: HDF 5 1. 8. 5 (1. 8. 6 coming in November!) HDF 5 source code: • • Written in C, and includes optional C++, Fortran 90 APIs, and High Level APIs Contains command-line utilities (h 5 dump, h 5 repack, h 5 diff, . . ) and compile scripts HDF 5 pre-built binaries: • When possible, include C, C++, F 90, and High Level libraries. Check. /libhdf 5. settings file. • Built with and require the SZIP and ZLIB external libraries October 28, 2010 Netc. DF Workshop 22 www. hdfgroup. org

HDF 5 API and Applications EOS Application Domain Data Objects EOS library MATLAB … HDF 5 Library Storage October 28, 2010 Netc. DF Workshop 23 www. hdfgroup. org

HDF 5 Software Layers & Storage HDF 5 Library Tools API … Language Interfaces C, Fortran, C++ Internals Virtual File Layer Storage I/O Drivers HDF 5 File Format October 28, 2010 High Level APIs h 5 dump tool h 5 repack tool HDF 5 Data Model Objects Groups, Datasets, Attributes, … Memory Mgmt Datatype Conversion Filters Split Files Posix I/O File HDFview tool Java Interface Tunable Properties Chunk Size, I/O Driver, … Chunked Storage Version and so on… Compatibility Custom MPI I/O Split Files Netc. DF Workshop File on Parallel Filesystem 24 ? Other www. hdfgroup. org

Useful Tools For New Users h 5 dump: Tool to “dump” or display contents of HDF 5 files h 5 cc, h 5 c++, h 5 fc: Scripts to compile applications HDFView: Java browser to view HDF 4 and HDF 5 files http: //www. hdfgroup. org/hdf-java-html/hdfview/ October 28, 2010 Netc. DF Workshop 25 www. hdfgroup. org

Introduction to HDF 5 Programming Model and APIs October 28, 2010 Netc. DF Workshop 26 www. hdfgroup. org

General Programming Paradigm • Object is opened or created • Object is accessed, possibly many times • Object is closed • Properties of object are optionally defined üCreation properties üAccess properties October 28, 2010 Netc. DF Workshop 27 www. hdfgroup. org

Order of Operations • An order is imposed on operations by argument dependencies For Example: A file must be opened before a dataset -becausethe dataset open call requires a file handle as an argument. • Objects can be closed in any order. October 28, 2010 Netc. DF Workshop 28 www. hdfgroup. org

The General HDF 5 API • Currently C, Fortran 90, Java, and C++ bindings. • C routines begin with prefix H 5? ? is a character corresponding to the type of object the function acts on Example Functions: H 5 D : Dataset interface e. g. , H 5 Dread H 5 F : File interface e. g. , H 5 Fopen H 5 S : data. Space interface e. g. , H 5 Sclose October 28, 2010 Netc. DF Workshop 29 www. hdfgroup. org

HDF 5 Defined Types For portability, the HDF 5 library has its own defined types: hid_t: object identifiers (native integer) hsize_t: size used for dimensions (unsigned long or unsigned long) herr_t: function return value hvl_t: variable length datatype Note: This is not an exhaustive list! For C, include hdf 5. h in your HDF 5 application. October 28, 2010 Netc. DF Workshop 30 www. hdfgroup. org

The HDF 5 API • For flexibility, the API is extensive Victronix Swiss Army Cybertool 34 ü 300+ functions • This can be daunting… but there is hope üA few functions can do a lot üStart simple üBuild up knowledge as more features are needed October 28, 2010 Netc. DF Workshop 31 www. hdfgroup. org

Basic Functions H 5 Fcreate (H 5 Fopen) create (open) File H 5 Screate_simple/H 5 Screate H 5 Dcreate (H 5 Dopen) H 5 Dread, H 5 Dwrite H 5 Dclose create (open) Dataset access Dataset close Dataset H 5 Sclose data. Space H 5 Fclose October 28, 2010 create data. Space close File Netc. DF Workshop 32 www. hdfgroup. org

Other Common Functions Data. Spaces: H 5 Sselect_hyperslab (Partial I/O) H 5 Sselect_elements (Partial I/O) H 5 Dget_space Groups: H 5 Gcreate, H 5 Gopen, H 5 Gclose Attributes: H 5 Acreate, H 5 Aopen_name, H 5 Aclose, H 5 Aread, H 5 Awrite Property lists: H 5 Pcreate, H 5 Pclose H 5 Pset_chunk, H 5 Pset_deflate October 28, 2010 Netc. DF Workshop 33 www. hdfgroup. org

The HDF Group Example HDF 5 Code October 28, 2010 Netc. DF Workshop 35 www. hdfgroup. org

Steps to Create a File 1. Decide on properties the file should have and create them if necessary: • Creation properties • Access properties • We will use Default properties. 2. Create the file 3. Close the file and the property lists, as needed October 28, 2010 Netc. DF Workshop 36 www. hdfgroup. org

Code: Create a File hid_t herr_t file_id; status; file_id = H 5 Fcreate("file. h 5", H 5 F_ACC_TRUNC, H 5 P_DEFAULT); status = H 5 Fclose (file_id); “/” (root) Note: Return codes not checked for errors in code samples. October 28, 2010 Netc. DF Workshop 37 www. hdfgroup. org

Steps to Create a Dataset 1. Define dataset characteristics a) Datatype – integer b) Dataspace - 4 x 6 c) Properties if needed, or use H 5 P_DEFAULT 2. Decide where to put it 2. Group or root group 3. Create dataset in file 4. Close everything October 28, 2010 Netc. DF Workshop “/” (root) A 38 www. hdfgroup. org

HDF 5 Pre-defined Datatype Identifiers HDF 5 defines* set of Datatype Identifiers per HDF 5 session. For example: C Type HDF 5 File Type HDF 5 Memory Type int H 5 T_STD_I 32 BE H 5 T_STD_I 32 LE H 5 T_NATIVE_INT float H 5 T_IEEE_F 32 BE H 5 T_IEEE_F 32 LE H 5 T_NATIVE_FLOAT double H 5 T_IEEE_F 64 BE H 5 T_IEEE_F 64 LE H 5 T_NATIVE_DOUBLE * Value of datatype is NOT fixed October 28, 2010 Netc. DF Workshop 39 www. hdfgroup. org

Pre-defined File Datatype Identifiers Examples: H 5 T_IEEE_F 64 LE H 5 T_STD_I 32 LE Architecture* Eight-byte, little-endian, IEEE floating-point Four-byte, little-endian, signed two's complement integer Programming Type NOTE: What you see in the file. Name is the same everywhere and explicitly defines a datatype. *STD= “An architecture with a semi-standard type like 2’s complement integer, unsigned integer…” October 28, 2010 Netc. DF Workshop 40 www. hdfgroup. org

Pre-defined Native Datatypes Examples of predefined native types in C: H 5 T_NATIVE_INT H 5 T_NATIVE_FLOAT H 5 T_NATIVE_UINT H 5 T_NATIVE_LONG H 5 T_NATIVE_CHAR NOTE: October 28, 2010 (int) (float ) (unsigned int) (long ) (char ) Memory types. Different for each machine. Used for reading/writing. Netc. DF Workshop 41 www. hdfgroup. org

Code: Create a Dataset 1 2. . 5 6 7 hid_t hsize_t herr_t dataspace_id; dims[2]; status; file_id = H 5 Fcreate (”file. h 5", H 5 F_ACC_TRUNC, Define a dataspace H 5 P_DEFAULT, H 5 P_DEFAULT); dims[0] = 4; current dims rank dims[1] = 6; dataspace_id = H 5 Screate_simple (2, dims, NULL); 8 dataset_id = H 5 Dcreate (file_id, ”A", H 5 T_STD_I 32 BE, dataspace_id, H 5 P_DEFAULT, H 5 P_DEFAULT); 9 status = H 5 Dclose (dataset_id); 10 status = H 5 Sclose (dataspace_id); 11 status = H 5 Fclose (file_id); October 28, 2010 Netc. DF Workshop 42 www. hdfgroup. org

Code: Create a Dataset 1. . . . 8 hid_t hsize_t herr_t file_id, dataset_id, dataspace_id; dims[2]; status; file_id = H 5 Fcreate (”file. h 5", H 5 F_ACC_TRUNC, H 5 P_DEFAULT); dims[0] = 4; dims[1] = 6; dataspace_id = H 5 Screate_simple (2, dims, NULL); Where to put it Datatype dataset_id = H 5 Dcreate (file_id, ”A", H 5 T_STD_I 32 BE, dataspace_id, H 5 P_DEFAULT, H 5 P_DEFAULT); Size & shape October 28, 2010 Properties (Link Creation, Dataset Creation and Access) Netc. DF Workshop 43 www. hdfgroup. org

Code: Create a Dataset 1 2 3 hid_t hsize_t herr_t file_id, dataset_id, dataspace_id; dims[2]; status; 4 file_id = H 5 Fcreate (”file. h 5", H 5 F_ACC_TRUNC, H 5 P_DEFAULT); 5 6 7 dims[0] = 4; dims[1] = 6; dataspace_id = H 5 Screate_simple (2, dims, NULL); 8 dataset_id = H 5 Dcreate (file_id, ”A", H 5 T_STD_I 32 BE, dataspace_id, H 5 P_DEFAULT, H 5 P_DEFAULT); Terminate access to 9 status = H 5 Dclose (dataset_id); dataspace, dataset, file 10 status = H 5 Sclose (dataspace_id); 11 status = H 5 Fclose (file_id); October 28, 2010 Netc. DF Workshop 44 www. hdfgroup. org

Example Code - H 5 Dwrite Dataset ID from H 5 Dcreate/H 5 Dopen Memory Datatype status = H 5 Dwrite (dataset_id, H 5 T_NATIVE_INT, H 5 S_ALL, H 5 P_DEFAULT, wdata); October 28, 2010 Netc. DF Workshop 45 www. hdfgroup. org

Partial I/O status = H 5 Dwrite (dataset_id, H 5 T_NATIVE_INT, H 5 S_ALL, H 5 P_DEFAULT, wdata); Memory Dataspace H 5 S_ALL File Dataspace (disk) To Modify Dataspace: H 5 Sselect_hyperslab H 5 Sselect_elements October 28, 2010 Netc. DF Workshop 46 www. hdfgroup. org

Example Code – H 5 Dwrite status = H 5 Dwrite (dataset_id, H 5 T_NATIVE_INT, H 5 S_ALL, H 5 P_DEFAULT, wdata); Data Transfer Property List (MPI I/O, Transformations, …) October 28, 2010 Netc. DF Workshop 47 www. hdfgroup. org

Example Code – H 5 Dread status = H 5 Dread (dataset_id, H 5 T_NATIVE_INT, H 5 S_ALL, H 5 P_DEFAULT, rdata); October 28, 2010 Netc. DF Workshop 48 www. hdfgroup. org

Steps to Create a Group 1. Decide where to put it – “root group” 2. Define properties or use H 5 P_DEFAULT 3. Create group in file. 4. Close the group. October 28, 2010 Netc. DF Workshop 51 www. hdfgroup. org

Example: Create a Group A “/” (root) B 4 x 6 array of integers file. h 5 October 28, 2010 Netc. DF Workshop 52 www. hdfgroup. org

Code: Create a Group hid_t file_id, group_id; . . . /* Open “file. h 5” */ file_id = H 5 Fopen (“file. h 5”, H 5 F_ACC_RDWR, H 5 P_DEFAULT); /* Create group "/B" in file. */ group_id = H 5 Gcreate (file_id, "B", H 5 P_DEFAULT, H 5 P_DEFAULT); /* Close group and file. */ status = H 5 Gclose (group_id); status = H 5 Fclose (file_id); October 28, 2010 Netc. DF Workshop 53 www. hdfgroup. org

HDF 5 Tutorial and Examples HDF 5 Tutorial: http: //www. hdfgroup. org/HDF 5/Tutor/ HDF 5 Example Code: http: //www. hdfgroup. org/ftp/HDF 5/examples-by-api/ October 28, 2010 Netc. DF Workshop 54 www. hdfgroup. org

HDF 5 Technology Platform • HDF 5 data model • The “building blocks” for data organization and specification Let’s look at …. • HDF 5 software • Library, language interfaces, tools • HDF 5 file format • Bit-level organization of HDF 5 file October 28, 2010 Netc. DF Workshop 55 www. hdfgroup. org

HDF 5 File Format • Defined by the HDF 5 File Format Specification. http: //www. hdfgroup. org/HDF 5/doc/H 5. format. html • Specifies the bit-level organization of an HDF 5 file on storage media. • HDF 5 library adheres to the File Format, so for the most part basic users do not need to know the guts of this information. October 28, 2010 HDF/HDF-EOS Workshop XIV 56 www. hdfgroup. org

HDF 5 Technology Platform • HDF 5 data model • The “building blocks” for data organization and specification Recall … • HDF 5 software • Library, language interfaces, tools • HDF 5 file format • Bit-level organization of HDF 5 file October 28, 2010 Netc. DF Workshop 57 www. hdfgroup. org

The HDF Group Thank You! October 28, 2010 Netc. DF Workshop 58 www. hdfgroup. org

The HDF Group Questions/comments? October 28, 2010 Netc. DF Workshop 59 www. hdfgroup. org