HDFEOS Tools Abe Taaheri Raytheon IIS ESIP Meeting
HDF-EOS Tools Abe Taaheri, Raytheon IIS ESIP Meeting Chapel Hill, NC July 9, 2013 Page 1
Outline • Overview of current support under EED project • Need for HDF-EOS tools • HEG & hdf-eos Plug-in for HDFView • Demo ? Page 2
Overview • The HDF has been selected by the EOSDIS Project as the format of choice for standard product distribution of EOS data • Under NASA contracts (ECS, EMD, EEB, EED) we developed HDF-EOS, and performed other development and maintenance activities Page 3
Overview – Libraries § HDF-EOS 2 & 5 § SDP & MTD Toolkits – Tools § HDF-EOS 2 to Geo. TIFF Converter (HEG) § HDF-EOS plug-in for THG HDFView § HDF-EOS 2 to HDF-EOS 5 conversion – User support § Helpdesk for Toolkit/HDF-EOS/HEG Page 4
Tools Why we need some HDF-EOS tools ? Page 5
Why HDF? • HDF was selected for EOS data since – Many of the HDF defined datatypes map well to EOS datatypes Ø Raster images, multi-dimensional arrays, and text blocks – HDF provides efficient data management Ø Storage & performance Page 6
HDF-EOS Datatypes • Some other EOS datatypes do not map directly to HDF datatypes, particularly geolocated datatypes Therefore, some additions to traditional HDF are required to fully support these datatypes. Page 7
HDF-EOS Datatypes To bridge the gap between the needs of EOS data products and the capabilities of HDF, 4 new EOS specific datatypes were introduced: Ø Ø Point, Swath, and Grid Zonal Average (hdf-eos 5) Page 8
HDF-EOS Datatypes • Point : Unorginzed data (spatial or temporal) that has associated geolocation information • Swath: Time-ordered data such as satellite swaths (time-ordered series of scanlines), or profilers (time-ordered series of profiles). • Grid: Data stored in a rectilinear array based on a well defined and explicitly supported projection. • ZA: Swath like datatype without geolocation mapping Page 9
HDF 4 File Contents – User View Objects & Relationships Object Data User Metadata Page 10
HDF 4 File Contents – Format View Complicated! variable ? name = variable_name rank type storagetype 1 Vgroup name = variable_name class = Var 0. 0 1 1 Object Data 1 1 1 NT 1 SD SDD 1 0. . . 1 data 0…* byte order, chunked storage, compression, … attribute name = attribute_name HDF 4 Mapping Project Update - Ruth Aydt 1 1 0. . . 1 1 1 1 NDG 0…* Vdata name = attribute_name class = Attr 0. 0 Page 11
HDF/HDF-EOS are Complicated! • HDF is not efficient for Users since they need to learn - how to use HDF & HDF-EOS libraries - how to process the data correctly • Licenses for commercial tools are expensive The HDF Group & Raytheon have developed some tools to help users Page 12
HDF & HDF-EOS Tools For most users HDF readers or Net. CDF/CF conversion applications are enough - hdp, h 5 dump, eos 2 dump(for lat/lon), HDFView - h 4 tonccf, h 4 cf, Ope. NDAP Handlers But some need more processing and tools appropriate for EOS data • HDF-EOS to Geo. TIFF Conversion Tool (HEG) • HDF-EOS plug-in for HDFView Page 13
HEG Page 14
HEG Ø Name says HEG is: A utility to convert EOSDIS data from HDF-EOS 2 to Geo. TIFF format. Ø But in practice: It can do more than just conversion to Geo. TIFF Page 15
HEG Functionality • Conversion – HDF-EOS (Swath or Grid ) to a single-band or multi-band Geo. TIFF’s – HDF-EOS Swath to HDF-EOS Grid --> Geo. TIFF OR – ASTER Swath is written directly into Geo. TIFF – HDF-EOS Swath or Grid to generic Binary (with metadata file) Page 16
HEG Functionality • Subsetting – spatial: using Swath/Grid corner lat/lon or pixel numbers – field: Grid, Swath, and Field selection – band: Selection of data layers in 3 -D and 4 -D datasets ( 3 -D to 6 -D MISR SOM datasets ) Page 17
HEG Functionality • Stitching – (mosaicing) + subsetting + Reprojecting • Reprojection To: UTM, PS, TM, STP, LCC, LAMAZ, GEO, SIN, Albers, CEA • Subsampling Subsample stacks Page 18
HEG Functionality • Running from Command Line (batch jobs) or • with Java-based GUI • Metadata preservation/creation. Page 19
HEG Conversion GUI Page 20
HEG Stitch GUI Page 21
HEG Geo. TIFF Hdf-Eos Binary Multi-Band Geo. TIFF Multi-Band Hdf-Eos Conversion GUI Bilinear Nearest Neighbor Cubic Convolution Geographic Polar Stereographic UTM Mercator Transverse Mercator Lambert Azimuthal Lambert Conformal Conic Sinusoidal Albers State Plane Cylindrical Eqyual-Area No-Reprojection Lat-Lon Rows-Columns Page 22
HEG Parameter File Used for running HEG on command line. Page 23
Status log for HEG run Page 24
HEG Data/Platform Support • Currently supports MODIS, MISR, ASTER, AIRS, AMSR-E products on TERRA and AQUA – more than 210 products • Operable on Sun, Windows, Linux, MAC Page 25
HEG Versions Ø Versions Available: § Stand-alone: A downloadable desktop version with a User Interface Portable, written in Java. Not dependent on COTS (eg. IDL). Page 26
HEG Versions § Integrated into ECS Data Pool: Access through NASA archive online storage Reduces the transfer time of HDF-EOS data sets (if subset is requested) Provides the end-user with the exact file required by their application Page 27
Examples (MISR) Mt. Etna Eruption: MISR L 1 B Stitched Blocks Page 28
Examples (MODIS) 2004 Hurricane Francis, MODIS MOD 021 KM, Multi-Band Geo. TIFF, 3 bands: EV_1 KM_Ref. SB, Bands 1, 2, 5 Page 29
Examples (ASTER) AST_L 1 B, 2003 California Fires Geo. projection, Multi-Band Geo. TIFF (3 bands): Visible Near Infrared (VNIR) …. . and Subsetted Two Swaths Stitched …. . Page 30
Examples ( AMSR_E_L 2_Ocean) swath Swath Conversion to Geographic Grid/HDF-EOS output Page 31
Future Work • 64 -bit versions - Linux (already available for Data Access ) - Windows - Mac • Support for HDF-EOS 5 Products, such as - OMI, MEa. SUREs * • Capability for Processing hundreds of granules for the same product by opening only one file in the HEG GUI * Ozone Monitoring Instrument Making Earth System Data Records for Use in Research Environments Page 32
HDF(EOS)View Page 33
HDFView A Java-based browser from The HDF Group for HDF 4 and HDF 5 based data. Page 34
HDFView with HDF-EOS Plug-in Page 35
HDFView with HDF-EOS Plug-in –Extends HDFView to browse HDF-EOS data in both versions (2 & 5). –Is a plug-in module does not change HDFView functionality –Identical look and feel when displaying HDF and HDF-EOS objects –Access to complete breakdown of HDF-EOS objects. e. g. grid info, projection info, dims, datafields, and attributes Page 36
HDFView Plug-in What else HDF-EOS plug-in adds to HDFView? Page 37
Swath, Grid, and Point Objects Plug-in file treeview Swath/Grid Lat/Lon display 38 Page 38
HE 2 to HE 5 Conversion One click to convert HE 2 to HE 5 file format 39 Page 39
AMSR-E Swath Getting Image of selected pixels in a table Getting lat/lon for a selected pixel in a table 40 Page 40
MODIS (HE 2) and OMI (HE 5) Grids Drawing Coastlines & International Borders on HE 2 & HE 5 Grids 41 Page 41
Future work • Coastlines & International borders for HE 2 & HE 5 swath • Adding single-band/multi-band Geo. TIFF writer • Adding Geo. TIFF reader with geolocating capability Page 42
Availability • Download, Documentation, Help: HDFView: http: //newsroom. gsfc. nasa. gov/sdptoolkit/ HDFView/HDFView_hdfeos_plugin. html HEG: http: //newsroom. gsfc. nasa. gov/sdptoolkit/ HEG/HEGHome. html Page 43
Help, Comments • Questions, Comments, Requests: Ø Abe_Taaheri@raytheon. com Ø PGSTLKIT@raytheon. com Ø http: //hdfeos. org/forums/index. php Page 44
Thank You! Questions/comments? Page 45
- Slides: 45