The HDF Group HDF 5 Update HighData Rate

Outline • HDF 5 features requested and sponsored by Synchrotron Community (2012 -2016) •

Direct chunk write: H 5 DOwrite_chunk 9/17/2021 HDRMX NSLS-II at BNL 3 www. hdfgroup.

Performance results for H 5 DOwrite_chunk Test result on Lunux 2. 6, x 86_64

Dynamically loaded filters • Problems with using custom filters • “Off the shelf” tools

Approach • There are predefined default locations where the HDF 5 library searches the

User’s headache and possible solutions • Where to find HDF 5 plugins? • No

SWMR: Data access to file being written New data elements… Writer … are added

SWMR • • Released in HDF 5 1. 10. 0 Restricted to append-data only

VDS • Data stored in multiple files and datasets can be accessed via one

VDS Example Series of images tim e t 3 t 1 k n k

VDS • VDS works with SWMR • File with VDS cannot be accessed by

New requirements and features? • Tell us your needs: • Multi-treaded compression filters •

HDF 5 Roadmap for 2016 • May 31 -HDF 5 1. 10. 0 -patch

Questions? Thank you! HDRMX NSLS-II at BNL 15 www. hdfgroup. org

Slides: 15

Download presentation

The HDF Group HDF 5 Update High-Data Rate MX Meeting NSLS-II at BNL May 26, 2016 epourmal@hdfgroup. org 1 www. hdfgroup. org

Outline • HDF 5 features requested and sponsored by Synchrotron Community (2012 -2016) • Direct chunk I/O (PSI, Dectris) • Dynamically loaded filters (DESY) • Single Write/ Multiple Reader or SWMR (DLS, ESRF, DESY) • Virtual Dataset (VDS) (DLS, DESY, XFEL) • New requirements and features • HDF 5 roadmap for 2016 HDRMX NSLS-II at BNL 2 www. hdfgroup. org

Direct chunk write: H 5 DOwrite_chunk 9/17/2021 HDRMX NSLS-II at BNL 3 www. hdfgroup. org

Performance results for H 5 DOwrite_chunk Test result on Lunux 2. 6, x 86_64 Each dataset contained 100 chunks, written by chunks 1 Speed in MB/s 2 Time in seconds 9/17/2021 HDRMX NSLS-II at BNL 4 www. hdfgroup. org

Dynamically loaded filters • Problems with using custom filters • “Off the shelf” tools do not work with the thirdparty filters • h 5 dump, MATLAB and IDL, etc. • HDF 5 tools cannot read file created by • h 5 py, Py. Tables , etc. • Solution • Modify HDF 5 source with your code and distribute it • And what will happen if a user wants filters from the different distributions? ? Oh…. No…. • Use a 1. 8. 11 and later • Provide maintained library of HDF 5 compression filters 9/17/2021 HDRMX NSLS-II at BNL 5 www. hdfgroup. org

Approach • There are predefined default locations where the HDF 5 library searches the shared libraries or DLLs with the HDF 5 filter functions. • The default location may be overwritten by an environment variable. • Encoding: Standard programming model to register compression filter on write operation • Decoding: Transparent on read; no user action required. • Works great, but…. 9/17/2021 HDRMX NSLS-II at BNL 6 www. hdfgroup. org

User’s headache and possible solutions • Where to find HDF 5 plugins? • No official distribution site • Example of HDF 5 filters distributions on Github Øhttps: //github. com/dectris/HDF 5 Plugin Øhttps: //github. com/Blosc/hdf 5 -blosc Øhttps: //github. com/nexusformat/HDF 5 -External-Filter-Plugins Øhttps: //svn. hdfgroup. org/hdf 5_plugins/ • The HDF Group can help with • Maintaining common repository • Testing plugin with the releases • Distributing source and binaries for Linux, Mac and Windows • Free and licensed (for fee) binary distributions 9/17/2021 HDRMX NSLS-II at BNL 7 www. hdfgroup. org

SWMR: Data access to file being written New data elements… Writer … are added to a dataset in the file… 04/01/16 Reader …which can be HDF 5 File read by a reader… with no IPC necessary. HDRMX NSLS-II at BNL www. hdfgroup. org

SWMR • • Released in HDF 5 1. 10. 0 Restricted to append-data only scenario SWMR doesn’t work on NFS Files are not compatible with HDF 5 1. 8. * libraries • Use h 5 format_convert tool • Converts HDF 5 metadata in place • No raw data is rewritten HDRMX NSLS-II at BNL 9 www. hdfgroup. org

VDS • Data stored in multiple files and datasets can be accessed via one dataset (VDS) using standard HDF 5 read/write HDRMX NSLS-II at BNL 10 www. hdfgroup. org

VDS Example Series of images tim e t 3 t 1 k n k VDS. h 5 A t 2 Image at time t 2 A B C k B C D E F n k D E E F n F 111 n k n M M Dataset B a. h 5 c. h 5 e. h 5 t 2 t 2 Dataset A A Dataset C C Dataset E E E C A A B b. h 5 B B Dataset D t 2 C D d. h 5 E t 2 f. h 5 HDRMX NSLS-II at BNL Dataset F F D D F F 11 www. hdfgroup. org

VDS • VDS works with SWMR • File with VDS cannot be accessed by HDF 5 1. 8. * libraries • Use h 5 repack tool to rewrite data (1. 10. 0 -patch 1) HDRMX NSLS-II at BNL 12 www. hdfgroup. org

New requirements and features? • Tell us your needs: • Multi-treaded compression filters • H 5 DOread_chunk function • Full SWMR implementation • Performance • Backward/forward compatibility • Other requests? HDRMX NSLS-II at BNL 13 www. hdfgroup. org

HDF 5 Roadmap for 2016 • May 31 -HDF 5 1. 10. 0 -patch 1 • H 5 repack, Windows builds, Fortran issues on HPC systems • Mid-summer HDF 5 1. 10. 1 • Some internal issues found for 1. 10. 0 • December • HPC features that didn’t make it into 1. 10. 0 release HDRMX NSLS-II at BNL 14 www. hdfgroup. org

Questions? Thank you! HDRMX NSLS-II at BNL 15 www. hdfgroup. org