The HDF Group Easy Access of HDF data

The HDF Group Easy Access of HDF data via NCL/IDL/MATLAB Kent Yang, Tong Qi, Ziying Li, Yi Wang, Shu Zhang, Joe Lee The HDF Group September 28, 2010 HDF/HDF-EOS Workshop XIV 1 www. hdfgroup. org

Motivation • Many Heterogeneous NASA HDF data products • To visualize the data, different products need to be handled differently • Users need to spend extra time figuring out the solutions • Individual data centers have already provided data services for the data they distributed • Some end-users prefer to use their favorite tools to access HDF data September 28, 2010 HDF/HDF-EOS Workshop XIV 2 www. hdfgroup. org

Learning Curve of accessing HDF data NCL From the ESIP wiki page: http: //wiki. esipfed. org/index. php/Making_Science_Data_Easier_to_Use_with_OPe. NDAP Making Science Data Easier to Use with OPe. NDAP September 28, 2010 HDF/HDF-EOS Workshop XIV 3 www. hdfgroup. org

Motivation September 28, 2010 HDF/HDF-EOS Workshop XIV 4 www. hdfgroup. org

How? September 28, 2010 HDF/HDF-EOS Workshop XIV 5 www. hdfgroup. org

The HDF Group Basic Examples September 28, 2010 HDF/HDF-EOS Workshop XIV 6 www. hdfgroup. org

Introduction to NCL/IDL/MATLAB • Interpreted languages • Visualization, analysis and computation • NCL - Free package, developed by NCAR - Support HDF-EOS 2, HDF-EOS 5 and HDF 4 • IDL - Widely used by Earth Science Community - Support HDF-EOS 2, HDF 4 and HDF 5 • MATLAB - Widely used by Computation and Engineering Communities - Support HDF-EOS 2, HDF 4 and HDF 5 September 28, 2010 HDF/HDF-EOS Workshop XIV 7 www. hdfgroup. org

A simple NCL example load "$NCARG_ROOT/lib/ncarg/nclex/gsun/gsn_code. ncl" load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm. ncl" begin cdf_file = addfile("AMSR_E_L 3_Rain. Grid_B 05_200707. he 2", "r") rrland = cdf_file->Rr. Land. Rain_Monthly. Rain. Total_Geo. Grid(: , : ) rrland@_Fill. Value = -1 resources = True xwks = gsn_open_wks("pdf", "AE_Rn. Gd. hdfeos 2") plot = gsn_csm_contour_map_ce(xwks, rrland, resources) end September 28, 2010 HDF/HDF-EOS Workshop XIV 8 www. hdfgroup. org

• Complete Code can be found under http: //hdfeos. org/software/ncl. php#ref_sec: ncl-hdfeos 2 -grid-1 d-unabridged September 28, 2010 HDF/HDF-EOS Workshop XIV 9 www. hdfgroup. org

A simple IDL example PRO AMSR_E_L 2 A_Brightness. Temperatures FILE_NAME= "AMSR_E_L 2 A_Brightness. Temperatures_V 09_200206190029_D. hdf“ SWATH_NAME='Low_Res_Swath' DATAFIELD_NAME='23. 8 H_Approx. _Res. 3_TB_(not-resampled)' file_id = EOS_SW_OPEN(FILE_NAME) swath_id = EOS_SW_ATTACH(file_id, SWATH_NAME) status = EOS_SW_READFIELD(swath_id, DATAFIELD_NAME, data) status = EOS_SW_READFIELD(swath_id, 'Longitude', lon) status = EOS_SW_READFIELD(swath_id, 'Latitude', lat) status = EOS_SW_DETACH(swath_id) status = EOS_SW_CLOSE(file_id) MAP_SET, /GRID, /CONTINENTS CONTOUR, data, lon, lat, /OVERPLOT, NLEVELS=20, /CELL_FILL END September 28, 2010 HDF/HDF-EOS Workshop XIV 10 www. hdfgroup. org

A simple MATLAB example • FILE_NAME='AMSR_E_L 2 A. hdf' SWATH_NAME='Low_Res_Swath‘ file_id = hdfsw('open', FILE_NAME, 'rdonly') swath_id = hdfsw('attach', file_id, SWATH_NAME) DATAFIELD_NAME='23. 8 H_Approx. _Res. 3_TB_(not-resampled)' [data, fail] = hdfsw('readfield', swath_id, DATAFIELD_NAME, [], []) [lon, status] = hdfsw('readfield', swath_id, 'Longitude', [], []) [lat, status] = hdfsw('readfield', swath_id, 'Latitude', [], []) hdfsw('detach', swath_id); hdfsw('close', file_id); contourf(lon, lat, data); September 28, 2010 HDF/HDF-EOS Workshop XIV 11 www. hdfgroup. org

• More information on the descriptions of these examples • Check hdfeos. org • NCL: http: //hdfeos. org/software/ncl. php • IDL: http: //hdfeos. org/examples/idl. php • MATLAB: http: //hdfeos. org/examples/matlab. php September 28, 2010 HDF/HDF-EOS Workshop XIV 12 www. hdfgroup. org

Motivation September 28, 2010 HDF/HDF-EOS Workshop XIV 13 www. hdfgroup. org

More helpful • A comprehensive NCL/IDL/MATLAB example codes and plots for sample data from most NASA Data centers: GES DISC MODAPS(LAADS) NSIDC LP-DAAC P. O DAAC GHRC OBPG(Ocean Color) La. RC September 28, 2010 HDF/HDF-EOS Workshop XIV 14 www. hdfgroup. org

Where are these examples located? • http: //hdfeos. org/zoo/ • We welcome you to send us feedback on these examples. You can use the HDF-EOS forum(http: //hdfeos. org/forums) to share your comments or contact us at eoshelp@hdfgroup. org. September 28, 2010 HDF/HDF-EOS Workshop XIV 15 www. hdfgroup. org

Common Issues for Tools • MATLAB and IDL - For IDL 7. x and before, cannot add color bar by using scripts - MATLAB - For data array > 1 MB, one needs to use 64 -bit MATLAB to generate plots; Takes very long time. September 28, 2010 HDF/HDF-EOS Workshop XIV 16 www. hdfgroup. org

Common Issues for Tools • MATLAB and IDL - HDF-EOS 2 non-geographic projection Grids - Need to provide additional latitude and longitude files - HDF-EOS 2 geographic projection Grids - Need to obtain parameters to calculate latitude and longitude • All Tools - HDF-EOS 2 Swaths with dimension maps - Need to provide additional latitude and longitude files September 28, 2010 HDF/HDF-EOS Workshop XIV 17 www. hdfgroup. org

Other Issues • Different ways to store metadata in an HDF file - Some HDF 4 products don’t provide lat/lon - Some HDF 4 products provide attributes to calculate lat/lon • Users not familiar with HDF 4 and HDF-EOS 2 file structures September 28, 2010 HDF/HDF-EOS Workshop XIV 18 www. hdfgroup. org

A Tip that you need to remember • You can always use HDFView to quickly examine any HDF files. September 28, 2010 HDF/HDF-EOS Workshop XIV 19 www. hdfgroup. org

September 28, 2010 HDF/HDF-EOS Workshop XIV 20 www. hdfgroup. org

The HDF Group Now we will walk through examples for each NASA data center September 28, 2010 HDF/HDF-EOS Workshop XIV 21 www. hdfgroup. org

The HDF Group GES DISC September 28, 2010 HDF/HDF-EOS Workshop XIV 22 www. hdfgroup. org

GES DISC AIRS Swath • Directly read the lat/lon and use the polar view … data=eos_file>radiances_L 2_Standard_cloud_cleared_radiance_product(: , 0) ; read specific subset of data field ; In order to read the radiances data field from the HDF-EOS 2 file, the group ; under which the data field is placed must be appended to the data field in NCL. For more information, ; visit section 4. 3. 2 of http: //hdfeos. org/software/ncl. php. data@lat 2 d=eos_file>Latitude_L 2_Standard_cloud_cleared_radiance_product ; associate longitude and latitude data@lon 2 d=eos_file>Longitude_L 2_Standard_cloud_cleared_radiance_product data@_Fill. Value=-9999 ; … res@gsn. Center. String="radiances at Channel=567" plot(2)=gsn_csm_contour_map_polar(xwks, data_2, res) res@gsn. Center. String="radiances at Channel=1339" plot(3)=gsn_csm_contour_map_polar(xwks, data_3, res) delete(plot) ; cleaning up resources used delete(data) NCL September 28, 2010 HDF/HDF-EOS Workshop XIV 23 www. hdfgroup. org

GES DISC AIRS Swath IDL September 28, 2010 Matlab HDF/HDF-EOS Workshop XIV 24 www. hdfgroup. org

GES DISC AIRS Grid • A typical global grid. Lat. and Lon. are provided. … %Reading Data from a Data Field GRID_NAME='ascending'; grid_id = hdfgd('attach', file_id, GRID_NAME); DATAFIELD_NAME='Rel. Humid_A'; [data 1, fail] = hdfgd('readfield', grid_id, DATAFIELD_NAME, [], []); … %Reading Lat and Lon Data GRID_NAME='location'; grid_id = hdfgd('attach', file_id, GRID_NAME); %Reading Lat Data DATAFIELD_NAME='Latitude'; [lat, status] = hdfgd('readfield', grid_id, DATAFIELD_NAME, [], []); lat=double(lat); [fillvalue, status] = hdfgd('getfillvalue', grid_id, DATAFIELD_NAME); lat(lat==fillvalue) = Na. N; %Reading Lon Data DATAFIELD_NAME='Longitude'; [lon, status] = hdfgd('readfield', grid_id, DATAFIELD_NAME, [], []); lon=double(lon); [fillvalue, status] = hdfgd('getfillvalue', grid_id, DATAFIELD_NAME); lon(lon==fillvalue) = Na. N; … September 28, 2010 HDF/HDF-EOS Workshop XIV Matlab 25 www. hdfgroup. org

GES DISC AIRS Grid NCL September 28, 2010 IDL HDF/HDF-EOS Workshop XIV 26 www. hdfgroup. org

GES DISC TRMM Swath • The global view doesn’t show much information; need a zoom view. 1 B 21_CSI. 990906. 10217. KORA. 6_bin. DIDHmean. idl September 28, 2010 HDF/HDF-EOS Workshop XIV 27 www. hdfgroup. org

GES DISC TRMM Swath Matlab September 28, 2010 HDF/HDF-EOS Workshop XIV 28 www. hdfgroup. org

GES DISC TRMM Grid • Grid, 3 B 43: • Calculate lat/lon based on the formula • Add “units” http: //disc. sci. gsfc. nasa. gov/additional/faq/precipitation_faq. shtml#lat_lon NCL September 28, 2010 HDF/HDF-EOS Workshop XIV 29 www. hdfgroup. org

GES DISC MERRA Grid • A typical global grid … data=eos_file->PLE_EOSGRID(1, 72, : ) ; read specific subset of data field ; ; In order to read the PLE data field from the HDF-EOS 2 file, the group ; under which the data field is placed must be appended to the data field in NCL. For more information, ; visit section 4. 3. 2 of http: //hdfeos. org/software/ncl. php. data@lon 1 d=eos_file->XDim_EOSGRID ; associate longitude and latitude data@lat 1 d=eos_file->YDim_EOSGRID ; here, since the XDim/YDim arrays are 1 -D, we use lon 1 d instead of lon 2 d data@units="Pa" data@long_name="Edge pressures" … data_4=eos_file->PLE_EOSGRID(7, 70, : ) ; read specific subset of data field data_4@lon 1 d=eos_file->XDim_EOSGRID ; associate longitude and latitude data_4@lat 1 d=eos_file->YDim_EOSGRID data_4@units="Pa" data_4@long_name="Edge pressures“ … res@gsn. Center. String="PLE at TIME=1, Height=72" plot(0)=gsn_csm_contour_map_ce(xwks, data, res) … September 28, 2010 HDF/HDF-EOS Workshop XIV NCL 30 www. hdfgroup. org

GES DISC TOMS Grid … ; retrieve data grid_id = EOS_GD_ATTACH(file_id, GRID_NAME) status = EOS_GD_READFIELD(grid_id, DATAFIELD_NAME, data) ; close file status = EOS_GD_DETACH(grid_id) status = EOS_GD_CLOSE(file_id) … ; retrieve lat ; field name should be defined as "YDim: TOMS Level 3" instead of "YDim: TOMS Level 3 (dimension)" DATAFIELD_NAME="YDim: TOMS Level 3" index=HDF_SD_NAMETOINDEX(new. File. ID, DATAFIELD_NAME) this. Sds. ID=HDF_SD_SELECT(new. File. ID, index) HDF_SD_GETDATA, this. Sds. ID, lat ; retrieve lon ; field name should be defined as "XDim: TOMS Level 3" instead of "XDim: TOMS Level 3 (dimension)" DATAFIELD_NAME="XDim: TOMS Level 3" index=HDF_SD_NAMETOINDEX(new. File. ID, DATAFIELD_NAME) … CONTOUR, BYTSCL(data, /NAN), lon, lat, /OVERPLOT, /FILL, C_Colors=Indgen(levels)+3, Background=1, NLEVELS=levels, Color=Black MAP_GRID, /BOX_AXES, COLOR=255 MAP_CONTINENTS, COLOR=255 … TOMS-EP_L 3 -TOMSEPL 3_2000 m 0101_v 8_Ozone. idl September 28, 2010 HDF/HDF-EOS Workshop XIV 31 www. hdfgroup. org

The HDF Group MODAPS MODIS Level 1 Swath September 28, 2010 HDF/HDF-EOS Workshop XIV 32 www. hdfgroup. org

MODAPS(LAADS) Swath MYD 021 KM. A 2002226. 0000. 005. 200919322273 September 28, 2010 HDF/HDF-EOS Workshop XIV 33 www. hdfgroup. org

MODAPS(LAADS) • MODIS Swath using dimension map • It needs additional latitude and longitude files provided by NASA • Where to obtain the corresponding latitude and longitude files? -ftp: //ladsweb. nascom. nasa. gov/all. Data/5/MYD 03 Or ftp: //ladsweb. nascom. nasa. gov/all. Data/5/MOD 03 MODIS Swath with dimension map September 28, 2010 HDF/HDF-EOS Workshop XIV 34 www. hdfgroup. org

MODAPS(LAADS) • Need to apply the scale and offset factor, need special formula • Normal: Data = scale* data + offset Matlab September 28, 2010 HDF/HDF-EOS Workshop XIV 35 www. hdfgroup. org

The HDF Group NSIDC September 28, 2010 HDF/HDF-EOS Workshop XIV 36 www. hdfgroup. org

NSIDC Swath Matlab September 28, 2010 HDF/HDF-EOS Workshop XIV 37 www. hdfgroup. org

NSIDC Polar Sterographic Grid NCL September 28, 2010 HDF/HDF-EOS Workshop XIV 38 www. hdfgroup. org

The HDF Group LP- DAAC September 28, 2010 HDF/HDF-EOS Workshop XIV 39 www. hdfgroup. org

LP-DAAC Sinusoidal Grid • The lat/lon are calculated by the hdfeos 2 dumper tool http: //hdfeos. org/software/eosdump. php September 28, 2010 MOD 09 GA. A 2007268. h 10 v 08. 005. 2007272184810_sur_refl_b 01_1. idl HDF/HDF-EOS Workshop XIV 40 www. hdfgroup. org

The HDF Group PO. DAAC September 28, 2010 HDF/HDF-EOS Workshop XIV 41 www. hdfgroup. org

PO. DAAC Geographic Grid • Need to calculate the lat/lon based on the information provided by the product document. NCL Matlab September 28, 2010 HDF/HDF-EOS Workshop XIV 42 www. hdfgroup. org

The HDF Group OBPG(Ocean Color) September 28, 2010 HDF/HDF-EOS Workshop XIV 43 www. hdfgroup. org

OBPG Sea. Wi. FS Matlab September 28, 2010 HDF/HDF-EOS Workshop XIV 44 www. hdfgroup. org

MODISA Swath Matlab September 28, 2010 HDF/HDF-EOS Workshop XIV 45 www. hdfgroup. org

MODIST Grid Matlab September 28, 2010 HDF/HDF-EOS Workshop XIV 46 www. hdfgroup. org

The HDF Group GHRC September 28, 2010 HDF/HDF-EOS Workshop XIV 47 www. hdfgroup. org

GHRC Level 3 Grid NCL September 28, 2010 HDF/HDF-EOS Workshop XIV 48 www. hdfgroup. org

The HDF Group La. RC September 28, 2010 HDF/HDF-EOS Workshop XIV 49 www. hdfgroup. org

La. RC • Special projections: Space Oblique Mercator (SOM) • Nested CERES Grid September 28, 2010 HDF/HDF-EOS Workshop XIV 50 www. hdfgroup. org

La. RC GOMACCS Space Oblique Mercator(SOM) Grid MISR_ELLIPSOID_GM_P 017_O 036105_DF_F 03_0024_GOM_b 64 -72_Blue_Radiance. RDQI. idl September 28, 2010 HDF/HDF-EOS Workshop XIV 51 www. hdfgroup. org

La. RC MISR SOM MISR_AM 1_AGP_P 001_F 01_24_Pt. Elev. idl September 28, 2010 HDF/HDF-EOS Workshop XIV 52 www. hdfgroup. org

La. RC MOPITT NCL September 28, 2010 HDF/HDF-EOS Workshop XIV 53 www. hdfgroup. org

La. RC_CERES cross-section ; Open file FILE_NAME='CER_ZAVG_Aqua-FM 4 MODIS_Edition 2 B_007005. 200503. hdf' new. File. ID=HDF_SD_START(FILE_NAME, /READ) ; Define datafield DATAFIELD_NAME='Ice Particle Diameter' index=HDF_SD_NAMETOINDEX(new. File. ID, DATAFIELD_NAME) ; Retrieve data this. Sds. ID=HDF_SD_SELECT(new. File. ID, index) HDF_SD_GETDATA, this. Sds. ID, data … ; generate lat=FINDGEN(180)*(-1)+89. 5 ; generate ngmt=FINDGEN(8)*1+1 … ; Start off generating the plot levels = 250 device, decomposed=0 Load. CT, 33, Ncolors=levels, Bottom=3 WINDOW, title='Ice Particle Diameter at Stats=0'+' '+'units: '+units, XSIZE=800 CONTOUR, data 2 D, ngmt, lat, /Fill, C_Colors=Indgen(levels)+3, Background=1, NLEVELS=levels, Color=Black, XTITLE='Monthly 3 hourly GMT time increments', YTITLE='latitude', POSITION=[0. 1, 0. 82, 0. 95] … CER_ZAVG_Aqua-FM 4 -MODIS_Edition 2 B_007005. 200503. idl September 28, 2010 HDF/HDF-EOS Workshop XIV 54 www. hdfgroup. org

CERES Nested Grid • http: //eosweb. larc. nasa. gov/PRODOCS/ceres/ SRBAVG/Quality_Summaries/srbavg_ed 2 d/ne stedgrid. html • Such projection is not supported. We have to emulate the projection. September 28, 2010 HDF/HDF-EOS Workshop XIV 55 www. hdfgroup. org

La. RC_CERES Hammer Sinusoidal September 28, 2010 HDF/HDF-EOS Workshop XIV 56 www. hdfgroup. org

Limitations • No tools can generate LAMAZ(Lambert Azimuthal projection ) grid properly. • No latitude and longitude files can be found for 250 meter and 500 meter MOD and MYD swaths using dimension maps distributed by MODAPS. • Some tools don’t support all projections. September 28, 2010 HDF/HDF-EOS Workshop XIV 57 www. hdfgroup. org

Again • Example codes and plots are under http: //hdfeos. org/zoo • We welcome you to send us feedback on these examples. You can use the HDF-EOS forum(http: //hdfeos. org/forums) to share your comments or contact us at eoshelp@hdfgroup. org. September 28, 2010 HDF/HDF-EOS Workshop XIV 58 www. hdfgroup. org

The HDF Group Thank you ! September 28, 2010 HDF/HDF-EOS Workshop XIV 59 www. hdfgroup. org

Acknowledgements This work was supported by cooperative agreement number NNX 08 AO 77 A from the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author[s] and do not necessarily reflect the views of the National Aeronautics and Space Administration. September 28, 2010 HDF/HDF-EOS Workshop XIV 60 www. hdfgroup. org

The HDF Group Questions/comments? September 28, 2010 HDF/HDF-EOS Workshop XIV 61 www. hdfgroup. org