The HDF Group HDF 5 Tools Peter Cao

The HDF Group HDF 5 Tools Peter Cao The HDF Group The 13 th HDF and HDF-EOS Workshop November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 1 www. hdfgroup. org

Outline November 3 -5, 2009 • An introduction to HDFView • HDF 5 command line tools HDF/HDF-EOS Workshop XIII 2 www. hdfgroup. org

What is HDFView? HDFView HDF Application Browser/Editor Written in Java For more information and download, visit www. hdfgroup. org/hdf-java-html November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 3 www. hdfgroup. org

HDFView Demo • Browse objects in an HDF 5 file • Show file content • Change file content Advanced features and new development such as plugin, region references, full support of hdf 5 1. 8 will be in a separate talk. HDFView is for general use. For specific purpose or more advanced features, you use other tools like MATLAB or IDL, or other earth science related tools such as HEG, NCL, or Gr. ADS. At the same time, welcome ideas for features. November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 4 www. hdfgroup. org

HDFView 2. 6 • • • Release planned in December, 2009 Build with HDF 5 1. 8. 4 Fix bugs Add new features Add support for 64 -bit mac. Drop mac Power. PC November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 5 www. hdfgroup. org

HDF 5 Command Line Tools • Look or check file content h 5 dump, h 5 diff, h 5 ls, h 5 check, etc. • Change file content or create new file h 5 repack, h 5 import, h 5 jam, h 5 copy, etc. • Convert file content h 4 toh 5, h 5 toh 4, gif 2 h 5, h 52 gif, etc. Only show you the basic features of some tools November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 6 www. hdfgroup. org

Where to Start? Tools are distributed with the library release and located under bin/ 1. Use “-h” option for help 2. Consult online help http: //www. hdfgroup. org/hdf 5 tools. html 3. Email help desk help@hdfgroup. org November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 7 www. hdfgroup. org

h 5 dump Dumps contents of HDF 5 file to stdout ASCII text • XML • Binary • November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 8 www. hdfgroup. org

h 5 dump: Object Headers What’s in the file? h 5 dump -H SDS. h 5 HDF 5 "SDS. h 5" { GROUP "/" { GROUP "Floats" { DATASET "Float. Array" { DATATYPE H 5 T_IEEE_F 32 LE DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) } } } DATASET "Int. Array" { DATATYPE H 5 T_STD_I 32 LE DATASPACE SIMPLE { ( 5, 6 ) / ( 5, 6 ) } } November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 9 www. hdfgroup. org

h 5 dump: Dataset What’s the content? h 5 dump -d /Floats/Float. Array SDS. h 5 HDF 5 "SDS. h 5" { DATASET "/Floats/Float. Array" { DATATYPE H 5 T_IEEE_F 32 LE DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) } DATA { (0, 0): 0. 01, 0. 02, 0. 03, (1, 0): 0. 1, 0. 2, 0. 3, (2, 0): 1, 2, 3, (3, 0): 10, 20, 30 } } } November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 10 www. hdfgroup. org

h 5 diff Show differences between… two objects or two files November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 11 www. hdfgroup. org

h 5 diff Any change in the data process? h 5 diff SDS. h 5 SDS 2. h 5 Dataset: </Int. Array> 5 differences found November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 12 www. hdfgroup. org

h 5 diff: Report Differences What are the changes? h 5 diff -r SDS. h 5 SDS 2. h 5 /Int. Array Dataset: </Int. Array> and </Int. Array> position Int. Array difference ---------------------[ 0 0 ] 0 10 10 [ 1 0 ] 10 100 90 [ 2 0 ] 20 200 180 [ 3 0 ] 30 300 270 [ 4 0 ] 40 400 360 5 differences found November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 13 www. hdfgroup. org

h 5 diff: Unsolved Issues How to compare two files with different • Structures • Storage layouts • Other properties November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 14 www. hdfgroup. org

h 5 repack Copies an entire file to a new file… November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 15 www. hdfgroup. org

h 5 repack: Change Data Layout Size of a 2 D byte array of 1024 x 1024? Why is my file so big? November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 3 MB 16 www. hdfgroup. org

h 5 repack: Change Data Layout h 5 repack -i huge. h 5 -o good. h 5 -l CHUNK=64 x 1024 Size of good. h 5? 1 MB November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 17 www. hdfgroup. org

h 5 repack: Compression Filter Most of my data values are zeros and ones Can I make it better? www. hdfgroup. org

h 5 repack: Compression Filter h 5 repack -i good. h 5 -o better. h 5 -f GZIP=6 Size of better. h 5? 6 KB Down from 3 MB to 6 KB, 500 times smaller!!!! November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 19 www. hdfgroup. org

h 5 import 1 GB 1 MB What should I do? November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 20 www. hdfgroup. org

h 5 import float 5 x 4 x 2. txt -c First_set. conf -o First_set. h 5 1. 01, 1. 02, 1. 03, 1. 04 1. 11, 1. 12, 1. 13, 1. 14 1. 21, 1. 22, 1. 23, 1. 24 1. 31, 1. 32, 1. 33, 1. 34 1. 41, 1. 42, 1. 43, 1. 44 GROUP "/" { GROUP "work" { DATASET "First-set" { DATATYPE H 5 T_IEEE_F 64 LE DATASPACE SIMPLE { ( 5, 2, 4 ) / ( 8, 8, H 5 S_UNLIMITED ) } DATA { (0, 0, 0): 1. 01, 1. 02, 1. 03, 1. 04, (0, 1, 0): 1. 11, 1. 12, 1. 13, 1. 14, (1, 0, 0): 1. 21, 1. 22, 1. 23, 1. 24, (1, 1, 0): 1. 31, 1. 32, 1. 33, 1. 34, (2, 0, 0): 1. 41, 1. 42, 1. 43, 1. 44, … } }} Use external datasets PATH work/First-set INPUT-CLASS TEXTFP RANK 3 DIMENSION-SIZES 5 2 4 OUTPUT-CLASS FP OUTPUT-SIZE 64 OUTPUT-ARCHITECTURE IEEE OUTPUT-BYTE-ORDER LE CHUNKED-DIMENSION-SIZES 2 2 2 MAXIMUM-DIMENSIONS 8 8 -1 November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 21 www. hdfgroup. org

Only available in 1. 8, not in 1. 6 h 5 copy h 5 mkgrp h 5 stat h 5 check November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 22 www. hdfgroup. org

h 5 copy Copies objects to files… November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 23 www. hdfgroup. org

h 5 copy: dataset dset. h 5 dset 2. h 5 / integers i 1 i 2 / floats 64 -bit f 1 f 32 f 2 h 5 copy –i dset. h 5 –s /floats/f 32 –o dset 2. h 5 –d /f 32 November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 24 www. hdfgroup. org

H 5 copy: group dset. h 5 dset 2. h 5 / floats integers i 1 i 2 / f 32 floats f 32 64 -bit f 1 f 2 h 5 copy –i dset. h 5 –s /floats –o dset 2. h 5 –d /floats November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 25 www. hdfgroup. org

h 5 mkgrp dset. h 5 / info data test 1 test 2 h 5 mkgrp test. h 5 /info /data/test 1 /data/test 2 November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 26 www. hdfgroup. org

h 5 stat Prints statistics about HDF 5 file Header size • Raw data size • Attributes, datatypes, etc. • Helps Troubleshoot overhead in HDF 5 files • Choose specific object’s properties and storage strategies • November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 27 www. hdfgroup. org

h 5 stat: Example >h 5 stat hdf 5_test. h 5 Filename: hdf 5_test. h 5 File information # of unique groups: 4 # of unique datasets: 13 … Object header size: (total/unused) Groups: 288/32 Datasets: 6824/864 … Dataset storage information: Total raw data size: 1769009 … November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 28 www. hdfgroup. org

Acknowledgements This work was supported by cooperative agreement number NNX 08 AO 77 A from the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author[s] and do not necessarily reflect the views of the National Aeronautics and Space Administration. November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 29 www. hdfgroup. org

The HDF Group Thank You! Questions/comments? November 3 -5, 2009 HDF/HDF-EOS Workshop XIII 30 www. hdfgroup. org