HDF 5 i RODS Peter Cao The HDF

HDF 5 -i. RODS Peter Cao The HDF Group Mike Wan San Diego Supercomputer Center HDF and HDF-EOS Workshop XII October 16, 2008 October 15 -17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 1

Imagine 100 Frames x 1 GB = 100 GB 1 GB HPSS DB October 15 -17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO HPC 2

Outline • HDF 5 -i. RODS module • Applications • Demo (if time permits) October 15 -17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 3

What is i. RODS? • Stands for i Rule Oriented Data Systems. • Developed by the Storage Resource Broker (SRB) team at the San Diego Supercomputer Center (SDSC). • A data grid software system that enables a customizable architecture for sharing data distributed across heterogeneous resources. October 15 -17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 4

What is i. RODS? Distributed Storage Database System Rule System For more information and download, visit www. irods. org October 15 -17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 5

Motivation i. RODS Distributed data system Indexing and searching Access control, etc. HDF 5 Large and diverse data High-performance I/O Subsetting, etc. High-performance distributed data system October 15 -17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 6

Whole File Access client I need to see the eye of Hurricane Bob! server Get the file HDF 5 Transfer large file – slow! October 15 -17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 7

HDF 5 Object or Subset Level Access client I need to see the eye of Hurricane Bob! Get me the eye of hurric server ane Bob HDF 5 Small transfer – fast! October 15 -17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 8

HDF 5 -i. RODS Module Distributed Storage HDF 5 i. RODS Module Database System Rule System October 15 -17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO Micro-services 9

HDF 5 -i. RODS Data Flow client server HDF 5 Library HDF 5 Object or Subset (File, Group, Dataset, Subset of Dataset, Attribute) i. RODS message (pack/unpack) October 15 -17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 10

New i. RODS Micro-services • Five i. RODS micro-services - msi. H 5 File_open - msi. H 5 File_close - msi. H 5 Dataset_read • reads entire dataset or subset of dataset - msi. H 5 Dataset_read_attribute - msi. H 5 Group_read_attribute Rule Engine msi. H 5 Dataset_read H 5 Dataset. read() File October 15 -17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 11

HDF 5 -Enabled i. RODS Server • HDF 5 library • Other external libraries (SZIP, ZLIB) • i. RODS version 1. 1 or later from https: //www. irods. org/index. php/Downloads/ Follow the README instruction at module/hdf 5 October 15 -17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 12

Client Application Requirements • • HDF 5 object header files and client handlers i. RODS client library and header files HDF 5 -i. RODS JNI for Java applications only $HOME/. irods. Env irods. Host 'kagiso. hdfgroup. uiuc. edu' irods. Port 1247 irods. User. Name 'rods‘ … For more information and download, visit http: //www. hdfgroup. org/projects/irods October 15 -17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 13

Example: HDFView Client Application October 15 -17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO HDF 5 -Enabled i. RODS Server 14

Example: HDFView October 15 -17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 15

Example: islice FLASH is an adaptive-mesh simulation code for astrophysical hydrodynamics problems • Command-line tool to visualize data produced by FLASH simulation runs • Data is huge (~ 100 GB) • Interesting part is small adaptive mesh 16*16*16*47531 For more information, visit flash. uchicago. edu October 15 -17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 16

Example: islice “. /islice -t flash. pal -m rpv 1 -p 2 rundir_055_8 km_hdf 5_plt_cnt_0424” Star Ash Flow 2048*8 (32 MB) Breakout point A slice from a 3 D simulation of The Detonation of a White Dwarf Star October 15 -17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 17

Thank You! This project is sponsored by CIP/NLADR, NSF PACI Project in Support of the Collaboration between the National Center for Supercomputing Applications (NCSA) and the San Diego Supercomputer Center (SDSC). The project is managed under the Cyber. Infrastructure Partnership (CIP), a joint effort led by NCSA and SDSC to help scientists and engineers take full advantage of the high-end Cyber. Infrastructure resources funded by the National Science Foundation (NSF). October 15 -17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 18

Questions/comments? October 15 -17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 19