Status of HDFEOS Related Software and Tools Abe
Status of HDF-EOS, Related Software and Tools Abe Taaheri, Raytheon IIS HDF & HDF-EOS Workshop XV Riverdale, MD April 18, 2012 1
Overview • Support Overview • Downloadable Software • What is: TOOLKIT, HDF-EOS • Software Releases • Major Development & Bug Fixes in the last releases • Future Plans • User Support 2
TOOLKIT / HDF-EOS Support • Maintenance support for TOOLKIT/HDF-EOS is funded on EOSDIS Evolution & Development (EED) contract - May 2010 through March 2015 • Current support § § § HDF-EOS 2 & 5 SDP & MTD Toolkits HDF-EOS 2 to Geo. TIFF Converter (HEG) HDF-EOS plug-in for HDFView Other Toolkit/HDF-EOS related software Toolkit/HDF-EOS/HEG helpdesk 3
ECS Support of HDF-EOS • Both HDF 4 and HDF 5 based flavors of HDF-EOS are part of ECS baseline. • OS’s: Solaris (10) Linux (32 & 64 -bit) Mac OS X (Power PC, Mac Intel 32 & 64 -bit) Cygwin Windows XP, VISTA, 7 (MTD Toolkit, HDF-EOS 2 & 5, HEG, HDFView) • Compilers: Fortran 77/90 , g 77/pgf 90 C, C++, gcc, g++, gfortran (Mac Intel) Intel Fortran (SGI Altix) 4
Downloadable Software & Applications 5
Downloadable Software • DAACS are supported with – TOOLKIT/HDF-EOS & HEG deliveries during yearly DAAC releases or patch deliveries – On demand • Other users can download our software from – our tools’ web sites – directly from our ftp site: ftp: //edhs 1. gsfc. nasa. gov/edhs 6
Downloadable Software • SDP Toolkit (Science Data Processing Tools) • MTD Toolkit (Metadata, Time/ Date Tools) • HDF-EOS 2 (HDF 4 based) • HDF-EOS 5 (HDF 5 based) • heconvert (converts HDF 4 - based Grid/Point/Swath structures to HDF-EOS 5 equivalents) • Dump. Metadata (Dumps HDF-EOS metadata) 7
Downloadable Applications • HEG (for HDF-EOS 2 products only) - Geo. TIFF conversion - Spatial subsetting, Field Subsetting - reprojection - stitching, sub-sampling and more …. - Support for Over 210 HEGable products from AIRS, AMSR-E, ASTER, MISR, and MODIS • HDFView Plug-in, Java-based browser plug-in (HDF 4, HDF 5, HDF-EOS 2 and 5 access) • HE 5 View* (HDF 5 - based browser) • EOSView* (HDF 4 - based browser) * Dropped support for OS updates on July 2006 8
Software/Tools What is: TOOLKIT & HDF-EOS 9
Toolkit • Science Data Production (SDP) Toolkit : collection of tools used by - Science software developers who produce code to process instrument data. - HDF-EOS users who need to • • Access metadata in HDF-EOS files Perform Time/Date conversion Access Digital Elevation Model Data Access Ancillary Data, such as Digital Chart of the World database (DCW), Olson World Ecosystem files, DEM data files, etc. 10
Toolkit (cont. ) • Tools provided in SDP Toolkit: - AA - CBP CSC CUC DEM EPH GCT IO MEM MET PC SMF TD (Ancillary Data Access) (CELESTIAL Body Position) (Coordinate System Conversion) (Constant and Unit Conversions) (Digital Elevation Model Access) (Ephemeris Data Access) (Geo Coordinate Transformation) (Input Output (File I/O)) (Memory Management) (Metadata Access) (Process Control) (Status Message File (Error/Status)) (Time and Date Conversion) 11
Toolkit (cont. ) • MTD Toolkit: – A short version of SDP Toolkit – Tools provided: • MET (Metadata Access) • TD (Time and Date Conversion) – Also supported in windows 12
HDF-EOS • HDF-EOS: – an extension to HDF which standardizes storage of and access to common Earth Sciences data structures • By providing a single interface to data structures common to earth science • By providing a container for EOS inventory, archive and product specific metadata • By establishing a consistent relationship between geolocation and science data. 13
HDF-EOS comes in 2 flavors 14
HDF-EOS 2 • HDF 4 - based, storage format for EOS standard products. • Used operationally by: – MODIS, MISR, ASTER, Landsat, AIRS – And some other EOS instrument teams • Support for Grid/Point/Swath structures 15
HDF-EOS 5 • HDF 5 - based, a complete rewrite of HDF-EOS 2 with a different user interface. • Used operationally by EOS Aura instrument teams • Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. • Supports same data structures (and more: Zonal average) • Added prefix ‘HE 5_’ to HDF-EOS 2 functions to distinguish two libraries 16
Software Releases 17
Software Releases SDP Toolkit/HDF-EOS: • Last release – January 2012 – TOOLKIT 5. 2. 18 – HDF-EOS 2. 18, HDF- EOS 5. 1. 14 • with HDF 4. 2. 6, HDF 5 -1. 8. 8, Szip 2. 1, zlib 1. 2. 5 • Next release – (January 2013) – TOOLKIT 5. 2. 19 – HDF-EOS 2. 19, HDF-EOS 5. 1. 15 • with new releases of HDF 4 (2. 8), 5 (1. 8. 9) 18
Software Releases MTD Toolkit: • Last release – Dec 2006 – Version 5. 2. 13 • in-sync with SDP TOOLKIT 5. 2. 13 • with HDF 4. 2 r 0, HDF 5 -1. 6. 3, Szip 1. 2 • Next release – June 2012 – Version 5. 2. 18 • in-sync with SDP TOOLKIT 5. 2. 18 • with HDF releases of HDF 4 (2. 6), 5 (1. 8. 8) 19
Software Releases HDF-EOS Plug-in for HDFView: • Last release – August 2011 – For HDFView Version 2. 7 • Next release – Augst 2012 – For HDFView Version 2. 8 20
Software Releases HECONVERT (HDF-EOS 2 to HDF-EOS 5 • Converter): Last releases – 2001, 2004 No Versioning – Oct. 2008 Version 2. 0 – Aug. 2011 Version 3. 0 • Next release – Aug. 2012 Version 4. 0 Ø Ø Fix for some missing global attributes in the converted file Fix for missing some swath attributes in hybrid files Dump. Metadata (ASCII Dump of HDF-EOS metadata) • Last release – Dec. 2008 Version 1. 0 21
Recent Major Developments and Bug Fixes 22
Major Development, Bug Fixes TOOLKIT (5. 2. 18): – 15 -arc second (500 m) DEM elevation and Standard deviation of Elevation data support in Toolkit – 15 -arc second (500 m) land/water mask data support in Toolkit – 15 -arc second DEM elevation and Land/water data (from USGS/UMD) conversion from Geo. TIFF to hdfeos format for use with DEM tools in TOOLKIT 23
Major Developments, Bug Fixes – Support for 32 & 64 -bit Mac OSX Intel with X 86_64 and i 386 structures – Added capability to install Toolkit/HDF-EOS 5 in threadsafe mode for Linux and Mac. Intel (32 and 64 -bit) – Capability for adding non-standard C and FORTRAN flags to TOOLKIT on installation command line – Use of gfortran (instead of PGF 90) for f 90 DAAC TOOLKIT compilation 24
Major Developments, Bug Fixes HDF-EOS 2 (2. 18): HDF-EOS 5 (1. 14): – Support for HDF 4 version 4. 2. 6 and HDF 5 version 1. 8. 8 – A few fixes for correct handling of unlimited dimensions in hdfeos 5 – New Toolkit style INSTALL scripts for HDF-EOS 2 and HDF-EOS 5 – support of GDij 2 ll for State Plane Projection 25
Major Developments, Bug Fixes HDF-EOS Plug-in for HDFView 2. 7: • Support of Linux and MAC OSX Intel 64 -bit Platforms. • Support for Zonal Average (ZA) objects of HDF-EOS 5. • Draw Map of Coastlines, international borders, and US state lines onto GRID images for all HDF-EOS supported projections. • heconvert 3. 0 usage. 26
Major Developments, Bug Fixes heconvert (3. 0): • Handle hybrid hdf files. Write HDF objects (created by HDF 4 calls) in addition to the HDF-EOS objects (created by HDF-EOS 2 calls) to the output. • Convert Dimension Scales when present. • Copy ECS's Inventory and Archive metadata into the output • Write HDF written field/group attributes to the output fields/groups. 27
Future Plans 28
Future Plans TOOLKIT/HDF-EOS: • Support for new versions of HDF 4 - HDF 5 • XML Metadata reading in TOOLKIT • Add APIs for local and group attributes in HDF-EOS 2 • Fix for getting correct compression info for a field in HDF-EOS 5, when GDcompinfo/SWcompinfo fails to get the info from Structure metadata 29
Future Plans • Store info in structural metadata somewhere else • Auto Configuration for TOOLKIT • RPMs for HDF-EOS and Toolkit • Validation tool for HDF-EOS 5 files • Fix for user/DAAC reported bugs • Break SDP TOOLKIT into small pieces separating general tools from PGE used tools 30
Future Plans HDF-EOS Plug-in for HDFView: • More functionalities : – Cut/paste objects, modify/delete attributes. – Create new grid/swath – Adding some HEG capabilities (such as Geotif output, swath to grid conversion, projection conversion, subsetting, resampling, etc) – Default data directory selection 31
Future Plans heconvert : Ø Writing all global attributes to the output Ø Fix for missing some swath attributes in hybrid files 32
Future Plans HEG: – Performance improvement for conversion of products with Azimuthal and Conformal Conic Projections – Performance improvement by parallelizing and better chunked/compressed SDS data reading – Chunking/Compressing data fields for HDF -EOS outputs – Exact UTM output for requested UTM zone 33
Future Plans – Reading Parameter file from stand-alone GUI – Support of HEG in MAC Intel – Setting spatial subseting corner lat/lon using field data image – Support of STP projection for swath conversion 34
Future Plans – Better handling of MISR grids for online orders and with stand-alone HEG • Capability to stitch all 180 blocks if memory is available • Handling of blocks near dateline for GEO grid output • Handling of 5 or more dimensional fields in MISR products • Support of STP projection conversion • Resolve problems associated with too many fields in some MISR products (such as AEROSOL products) 35
User Support 36
Availability • Access to libraries, applications, Documentation: TOOLKIT: http: //newsroom. gsfc. nasa. gov/sdptoolkit/toolkit. html HDFView: http: //newsroom. gsfc. nasa. gov/sdptoolkit/HDFView/ HDFView_hdfeos_plugin. html HEG: http: //newsroom. gsfc. nasa. gov/sdptoolkit/HEGHome. html HDF-EOS: http: //hdfeos. net/ • Email ( sample codes + HDF-EOS Forum ) Abe_Taaheri@raytheon. com Landover_PGSTLKIT@raytheon. com 37
Thank You! 38
- Slides: 38