The HDF Group Data Interoperability The HDF Group
The HDF Group Data Interoperability The HDF Group Staff Sep. 28 -30, 2010 HDF/HDF-EOS Workshop XIV 1 www. hdfgroup. org
Interoperability • Interoperability is a property of a product or system, whose interfaces are completely understood, to work with other products or systems, present or future, without any restricted access or implementation. (From Wikipedia) Sep. 28 -30, 2010 HDF/HDF-EOS Workshop XIV 2 www. hdfgroup. org
Background for data interoperability • Focus on Earth Sciences • Data needs to be shared across communities • Self-described data format • Many different types of data • Grids, moving-sensor multidimensional fields, time series, profiles, trajectories, geospatial framework data, points Sep. 28 -30, 2010 HDF/HDF-EOS Workshop XIV 3 www. hdfgroup. org
Why data interoperability? • Data format • Several data formats available HDF 4/5, net. CDF 3/4, GRIB, BUFR, Binary • Data are created with some data formats by following different physical models • End users would like to use familiar tools to access data via different formats Sep. 28 -30, 2010 HDF/HDF-EOS Workshop XIV 4 www. hdfgroup. org
Why data interoperability? • Semantic/content interoperability • An example: the missing value of a physical variable • Tools or applications need to know the fill value of a physical variable • How can they identify? Normally via an attribute that stores the fill value • The name can be “Fill_value, _Fill. Value, bad. Value, ……” Sep. 28 -30, 2010 HDF/HDF-EOS Workshop XIV 5 www. hdfgroup. org
Goals • Discuss a few cases about data interoperability • Hear opinions or issues about data interoperability from audience. Sep. 28 -30, 2010 HDF/HDF-EOS Workshop XIV 6 www. hdfgroup. org
The HDF Group Data Format Interoperability Sep. 28 -30, 2010 HDF/HDF-EOS Workshop XIV 7 www. hdfgroup. org
Case 1 • One set of APIs to handle multiple data formats Sep. 28 -30, 2010 HDF/HDF-EOS Workshop XIV 8 www. hdfgroup. org
Case 1 • net. CDF 4 • Combine powerful model and simplicity of net. CDF with features of HDF 5 • net. CDF interface on top of HDF 5 • CDM(Common Data Model) • Harmonize net. CDF, HDF 4, HDF 5, OPe. NDAP, GRIB and others Sep. 28 -30, 2010 HDF/HDF-EOS Workshop XIV 9 www. hdfgroup. org
Discussions • Any comments/sharings about the experiences with net. CDF 4 and CDM? Sep. 28 -30, 2010 HDF/HDF-EOS Workshop XIV 10 www. hdfgroup. org
Case 2 • Format conversions and translations Sep. 28 -30, 2010 HDF/HDF-EOS Workshop XIV 11 www. hdfgroup. org
Data format Conversions/Translations • Format conversions • • The HDF 4 to HDF 5 conversion tool The HDF-EOS 2 to net. CDF 3/4 conversion tool HDF-EOS to Geo. TIFF conversion tool …… • Data format Translations • net. CDF tools to access HDF 4/5 files via OPe. NDAP • …… Sep. 28 -30, 2010 HDF/HDF-EOS Workshop XIV 12 www. hdfgroup. org
Use net. CDF tools to access HDF via OPe. NDAP Aqua/Aura Translation Layers Users IDV HDF 4/5 Handlers libnc-dap AIRS/OMI DAP Visualization Tools Sep. 28 -30, 2010 OPe. NDAP Clients HDF 4/5 Files OPe. NDAP Servers HDF/HDF-EOS Workshop XIV 13 www. hdfgroup. org
net. CDF 4 to access HDF-EOS 5 files • Augmentation • One file can be used for both EOS 5 and Net. CDF-4. • Note that EOS 5 users are not affected at all. Augmentation HDF-EOS 5 HDF 5 Sep. 28 -30, 2010 HDF-EOS 5 file Augmented HDF-EOS 5 file HDF/HDF-EOS Workshop XIV Net. CDF 4 Net. CDF-4 file HDF 5 14 www. hdfgroup. org
Discussions • Any comments/sharings about the experiences with data format conversions and translations Sep. 28 -30, 2010 HDF/HDF-EOS Workshop XIV 15 www. hdfgroup. org
The HDF Group Semantic/Content Interoperability Sep. 28 -30, 2010 HDF/HDF-EOS Workshop XIV 16 www. hdfgroup. org
An example • Degree of latitude and longitude • Champaign’s latitude is 40° 6´ 38" N • It can be represented in two formats: • 400638 in DDDMMSS format • 40. 1105556 in decimal format • How can the applications know which format the file is used? • Better to have a common standard to facilitate the exchange this kind of information Sep. 28 -30, 2010 HDF/HDF-EOS Workshop XIV 17 www. hdfgroup. org
Semantic/Content Interoperability • CF conventions become such a standard • Many applications/tools follow CF conventions to interpret the metadata Sep. 28 -30, 2010 HDF/HDF-EOS Workshop XIV 18 www. hdfgroup. org
Use net. CDF tools to access HDF via OPe. NDAP Aqua/Aura Not follow CF conventions Users IDV HDF 4/5 Handlers libnc-dap AIRS/OMI DAP Visualization Tools Sep. 28 -30, 2010 OPe. NDAP Clients HDF 4/5 Files OPe. NDAP Servers HDF/HDF-EOS Workshop XIV 19 www. hdfgroup. org
Discussions • Any comments about semantic/content interoperability • Any comments about the usage of CF conventions Sep. 28 -30, 2010 HDF/HDF-EOS Workshop XIV 20 www. hdfgroup. org
The HDF Group HDF 5 and net. CDF-4 Libraries Interoperability Two sources of interoperability problems and how to deal with them Sep. 28 -30, 2010 HDF/HDF-EOS Workshop XIV 21 www. hdfgroup. org
net. CDF-4 library and arbitrary HDF 5 files • net. CDF-4 is built on top of HDF 5 • net. CDF-4 files are HDF 5 files with specific characteristics 1. Tracked creation order of objects and their attributes 2. Absence of datasets with complex compound, and region and object references 3. Presence of dimension scales • net. CDF-4 expects to find properties 1 and 2 in an HDF 5 file and fails to open or read the file when those are not present (e. g. , the NPOESS files) • Solution: TB discussed with the net. CDF-4 folks 12/14/2021 Sep. 2830, 2010 HDF/HDF-EOS Workshop XIV 22 www. hdfgroup. org
Programming models and libraries settings • Use case 1 (libraries settings): Simultaneous access to an HDF 5 file via HDF 5 and net. CDF-4 • One of the libraries will not be able to open the file due to the different access properties used by each library (H 5 F_CLOSE_WEAK in HDF 5 vs. H 5 F_CLOSE_ STRONG in net. CDF-4) • Solution: • Design improvements in HDF 5 • New APIs to detect situation • Automatic detection and correction of the access properties • Document the best practices for applications 12/14/2021 Sep. 2830, 2010 HDF/HDF-EOS Workshop XIV 23 www. hdfgroup. org
Programming models and libraries settings • Use case 2 (programming models): Usage of HDF 5 wrapper libraries and net. CDF-4 library in the same application • C++ H 5 Topen call fails due to the net. CDF-4 library shutting down the HDF 5 library underneath by calling H 5 close • Solution: • Resolved in the newest net. CDF-4 • Document the best practices for applications • For more detailed discussion see “Net. CDF-4/HDF 5 Libraries Interoperability Issues” at http: //www. hdfgroup. uiuc. edu/RFC/HDF 5/net. CDF 4 -HDF 5/ • Let us know your use case to improve testing! 12/14/2021 Sep. 2830, 2010 HDF/HDF-EOS Workshop XIV 24 www. hdfgroup. org
Thank you ! Sep. 28 -30, 2010 HDF/HDF-EOS Workshop XIV 25 www. hdfgroup. org
- Slides: 25