GRIB 1 GRIB 2 and Some other Formats
GRIB 1, GRIB 2, and Some other Formats used in Meteorology and Oceanography Wesley Ebisuzaki Climate Prediction Center National Centers for Environmental Prediction
Types of data Point observations • • Station data Rawinsonde, a line of observations Aircraft data, a line of observations “Raw” satellite data, a swath of observations
Types of data Point observations Ø Ø Station data Rawinsonde, a line of observations Aircraft data, a line of observations “Raw” satellite data, a swath of observations Gridded fields, 2 D or 3 D fields Ø Ø Ø Numerical model Satellite product Radar product
WMO Formats WMO = World Meteorological Organization Purpose Ø Ø Transfer of information (observations, forecasts) between meteorologial centers NCEP and other NMCs (National Meteorological Centers) share observations and forecasts by GTS (Global Telecommunication System) or by the internet
WMO Formats WMO = World Meteorological Organization 1. BUFR, for points 2. GRIB, for grids § § § WMO specifies the format Many implementations Table based Tables request much metadata Want more metadata? Request new table! Bureaucratic: UN (WMO) + governmental agencies
WMO Formats WMO = World Meteorological Organization Pros Ø Ø Ø Transmission format Good packing/compression Many messages, not one big chuck of data
GRIB Stream of data GRIB is based on messages, and each message is complete. … (1000 h. Pa TMP)(900 h. Pa TMP)(800 h. Pa TMP)(700 h. PA TMP) … (problem) … (1000 h. Pa TMP)? ? ? (800 h. Pa TMP)(700 h. PA TMP) … Suppose there was a problem while the 900 h. Pa TMP was being sent. Since each grib message is complete, you have only lost the 900 h. Pa TMP. Note: grib 2 allows messages to be combined to make larger messages. This allows some space saving because some metadata like the grid description are duplicated.
GRIB: Inventorying the File Versions of grib v 0 (grib 0) very rare, quickly replaced by grib v 1 (grib 1) common, being replaced by grib v 2 (decade long process) Can use wgrib to inventory a grib 1 file grib v 2 (grib 2) common, fixes limitations of grib 1, compression grib 2 is a superset of grib 1 (minor exceptions) Ok: grib 1 → grib 2, sometimes fail: grib 2 → grib 1 Can use wgrib 2 to inventory grib 2 Many programs to invenory a grib file. Using wgrib 2 because of familiarity.
GRIB: a sample file GRIB is based on messages, and each message is complete. ebis@linux-landing: ~$ wgrib 2 narr_ncdc. 2013061700. grb 2 1: 0: d=2013061700: MSLET: mean sea level: anl: : 2: 63615: d=2013061700: PRMSL: mean sea level: anl: : 3: 131558: d=2013061700: PRES: 1 hybrid level: anl: : 4: 170070: d=2013061700: HGT: 1 hybrid level: anl: : 5: 213839: d=2013061700: TMP: 1 hybrid level: anl: : Red message number Green byte location starting at zero Note: only showing the first 5 messages.
GRIB: a sample file GRIB is based on messages, and each message is complete. ebis@linux-landing: ~$ wgrib 2 narr_ncdc. 2013061700. grb 2 1: 0: d=2013061700: MSLET: mean sea level: anl: : 2: 63615: d=2013061700: PRMSL: mean sea level: anl: : 3: 131558: d=2013061700: PRES: 1 hybrid level: anl: : 4: 170070: d=2013061700: HGT: 1 hybrid level: anl: : 5: 213839: d=2013061700: TMP: 1 hybrid level: anl: : Blue Reference Time (year, month, . . , seconds), minutes and seconds are not shown. The reference time is the time of the analysis or start of the forecast. If a flag is set, the reference time could be the observation time or the forecast verification time. However, this is extremely rare.
GRIB: a sample file GRIB is based on messages, and each message is complete. ebis@linux-landing: ~$ wgrib 2 narr_ncdc. 2013061700. grb 2 1: 0: d=2013061700: MSLET: mean sea level: anl: : 2: 63615: d=2013061700: PRMSL: mean sea level: anl: : 3: 131558: d=2013061700: PRES: 1 hybrid level: anl: : 4: 170070: d=2013061700: HGT: 1 hybrid level: anl: : 5: 213839: d=2013061700: TMP: 1 hybrid level: anl: : Green Variable such as temperature, zonal wind, RH Names shown are the NCEP standard http: //www. nco. ncep. noaa. gov/pmb/docs/grib 2_table 4 -1. shtml or wgrib 2 -v Red Vertical level or layer
GRIB: a sample file GRIB is based on messages, and each message is complete. ebis@linux-landing: ~$ wgrib 2 narr_ncdc. 2013061700. grb 2 1: 0: d=2013061700: MSLET: mean sea level: anl: : 2: 63615: d=2013061700: PRMSL: mean sea level: anl: : 3: 131558: d=2013061700: PRES: 1 hybrid level: anl: : 4: 170070: d=2013061700: HGT: 1 hybrid level: anl: : 5: 213839: d=2013061700: TMP: 1 hybrid level: anl: : Blue timing information anl = analysis 6 hour fcst = 6 hour forecast 0 -6 hour ave fcst = average of 0 -6 hour fcst 0 -6 hour acc fcst = accumulation of 0 -6 hour fcst 4@6 hour ave (anl) = ave of 4 analysis, daily ave Red other identifiers such as ensemble, aerosol type, etc
GRIB: a sample file Examination of one grib message in more detail. ebis@linux-landing: ~$ wgrib 2 -V -d 1 narr_ncdc. 2013061700. grb 2 1: 0: vt=2013061700: mean sea level: anl: MSLET MSLP (Eta model reduction) [Pa]: ndata=96673: undef=1648: mean=101491: min=98257. 6: max=103123 grid_template=30: winds(grid): Lambert Conformal: (349 x 277) input WE: SN output WE: SN res 56 Lat 1 1. 000000 Lon 1 214. 500000 Lo. V 253. 000000 Lat. D 50. 000000 Latin 1 50. 000000 Latin 2 50. 000000 Lat. SP 0. 000000 Lon. SP 0. 000000 North Pole (349 x 277) Dx 32463. 0000 m Dy 32463. 0000 m mode 56 wgrib 2 a utility for grib 2 files -V Verbose display -d 1 dump grib message 1
GRIB: a sample file Examination of one grib message in more detail. ebis@linux-landing: ~$ wgrib 2 -V -d 1 narr_ncdc. 2013061700. grb 2 1: 0: vt=2013061700: mean sea level: anl: MSLET MSLP (Eta model reduction) [Pa]: ndata=96673: undef=1648: mean=101491: min=98257. 6: max=103123 grid_template=30: winds(grid): Lambert Conformal: (349 x 277) input WE: SN output WE: SN res 56 Lat 1 1. 000000 Lon 1 214. 500000 Lo. V 253. 000000 Lat. D 50. 000000 Latin 1 50. 000000 Latin 2 50. 000000 Lat. SP 0. 000000 Lon. SP 0. 000000 North Pole (349 x 277) Dx 32463. 0000 m Dy 32463. 0000 m mode 56 1 grib message 1 0 byte location vt=2013061700 verification time, minutes and seconds not shown [Pa] units
GRIB: a sample file Examination of one grib message in more detail. ebis@linux-landing: ~$ wgrib 2 -V -d 1 narr_ncdc. 2013061700. grb 2 1: 0: vt=2013061700: mean sea level: anl: MSLET MSLP (Eta model reduction) [Pa]: ndata=96673: undef=1648: mean=101491: min=98257. 6: max=103123 grid_template=30: winds(grid): Lambert Conformal: (349 x 277) input WE: SN output WE: SN res 56 Lat 1 1. 000000 Lon 1 214. 500000 Lo. V 253. 000000 Lat. D 50. 000000 Latin 1 50. 000000 Latin 2 50. 000000 Lat. SP 0. 000000 Lon. SP 0. 000000 North Pole (349 x 277) Dx 32463. 0000 m Dy 32463. 0000 m mode 56 ndata undef mean number of grid points that have undefined values mean value of defined grid points
GRIB: a sample file Examination of one grib message in more detail. ebis@linux-landing: ~$ wgrib 2 -V -d 1 narr_ncdc. 2013061700. grb 2 1: 0: vt=2013061700: mean sea level: anl: MSLET MSLP (Eta model reduction) [Pa]: ndata=96673: undef=1648: mean=101491: min=98257. 6: max=103123 grid_template=30: winds(grid): Lambert Conformal: (349 x 277) input WE: SN output WE: SN res 56 Lat 1 1. 000000 Lon 1 214. 500000 Lo. V 253. 000000 Lat. D 50. 000000 Latin 1 50. 000000 Latin 2 50. 000000 Lat. SP 0. 000000 Lon. SP 0. 000000 North Pole (349 x 277) Dx 32463. 0000 m Dy 32463. 0000 m mode 56 grid template: 0=lat-lon 10=mercator 20=polar 30=lambert conformal winds: earth vs grid, grid north is from grid point (i, j) to (i, j+1)
GRIB: a sample file Examination of one grib message in more detail. ebis@linux-landing: ~$ wgrib 2 -V -d 1 narr_ncdc. 2013061700. grb 2 1: 0: vt=2013061700: mean sea level: anl: MSLET MSLP (Eta model reduction) [Pa]: ndata=96673: undef=1648: mean=101491: min=98257. 6: max=103123 grid_template=30: winds(grid): Lambert Conformal: (349 x 277) input WE: SN output WE: SN res 56 Lat 1 1. 000000 Lon 1 214. 500000 Lo. V 253. 000000 Lat. D 50. 000000 Latin 1 50. 000000 Latin 2 50. 000000 Lat. SP 0. 000000 Lon. SP 0. 000000 North Pole (349 x 277) Dx 32463. 0000 m Dy 32463. 0000 m mode 56 (349 x 277) grid dimensions input WE: SN how grid points are ordered in the grib file 16 types of ordering, WE: SN and WE: NS are common NDFD (National Digital Forecast Database) uses “plowing” order output WE: SN how grid points are ordered within wgrib 2
WE: SN order Third row Second row First row NCEP regional models use WE: SN, NCEP global models Use WE: NS Plowing order Fourth row Third row Second row First row Common at NDFD and seen at ECMWF To save space, store the delta between adjacent grid points. The deltas are smaller in plowing order.
GRIB: grid description Examination of one grib message in more detail. ebis@linux-landing: ~$ wgrib 2 -V -d 1 narr_ncdc. 2013061700. grb 2 1: 0: vt=2013061700: mean sea level: anl: MSLET MSLP (Eta model reduction) [Pa]: ndata=96673: undef=1648: mean=101491: min=98257. 6: max=103123 grid_template=30: winds(grid): Lambert Conformal: (349 x 277) input WE: SN output WE: SN res 56 Lat 1 1. 000000 Lon 1 214. 500000 Lo. V 253. 000000 Lat. D 50. 000000 Latin 1 50. 000000 Latin 2 50. 000000 Lat. SP 0. 000000 Lon. SP 0. 000000 North Pole (349 x 277) Dx 32463. 0000 m Dy 32463. 0000 m mode 56 Various parameters that define the Lambert Conformal Projection Lat 1, Lon 1 location of the first grid point (grib file ordering) Dx, Dy grid spacing at Latin 1
GRIB: monthly mean Examination of time ranges. bash-4. 1$ wgrib 2 flxf 06. gdas. 201604. grb 2 -d 1 1: 0: d=2016040100: UFLX: surface: 120@6 hour ave(0 -6 hour ave fcst), missing=0: Monthly mean zonal wind stress at the surface 1: 0: d=2016040100: UFLX: surface: 30@24 hour ave(0 -6 hour ave fcst), missing=0: Monthly mean of the 00 Z UFLX. In the above examples “ave” which is short for averaged is used. Ave is only one of the possibilities Ave Acc Max Min average accumulation maximum minimum Uncommmon: RMS, Std Dev, Standardized Anomaly, etc
GRIB 2: packing/compression Models are getting bigger faster than the bandwidth increases. Compression helps but still cannot send the full resolution forecasts to the users. GFS 0. 25 degree single forecast hour, all fields in MB. Simple = scaled integers (also used in grb 1)
GRIB 2: packing/compression Jpeg 2000: very good compression but very slow Poor when data has undefined values Widely used at NCEP Complex: encode data, deltas or deltas of deltas, 6 flavors of complex Good compression but not as good jpeg 2000 Best = +9% (size), c 3 = +12% c 3. bitmap = +16% Better than jpeg 2000 when undefined values Very fast decoding, 20 x jpeg 2000 Increasing use at NCEP (speed vs size) Png: compression not as good as jpeg 2000 RLE: run length encoding, Japanese radar products Aec: open source szip, new, fast Simple: big files, very fast to pack/unpack Packing should be transparent to the user.
GRIB 2: packing/compression Examination of one grib message in more detail. bash-4. 1$ wgrib 2 flxf 06. gdas. 201604. grb 2 -d 1 -v 2 -packing 1: 0: packing=grid point data - complex packing and spatial differencing, c 3 val=(-10916+i*2^0)*10^-4, ref=0. . 32767 (#bits=15) group width bits=4 #groups=45351 Packing= many different types of packing/compression Common: jpeg 2000, simple, complex Less common: ieee, RLE (Japan), aec (DWD, new) val=(-10916+i*2^0)*10^-4, ref=0. . 32767 Usually grib stores grid values as scaled integers ECMWF and wgrib 2 usually scale by 10^0 and keep the number of binary bits fixed. NCEP usually scale by specified 2^N and 10^M Scaling depends on variable (ex. RH vs TMP).
GRIB 1: software Grib 1 has been around for 20+ years, it's a relatively simple standard. Many libraries Much software grib software for my cell phone. I don't do much in grib 1 but my tool kit includes Gr. ADS Grib 2 ctl Wgrib Copygb Grb 1 to 2: visualization control file make for Gr. ADS inventory, decode and basic data base functions interpolation grib 1 → grib 2 converter Some notable open source codes CDO GDAL NCL
GRIB 2: software Grib 2 is a newer, more complicated standard Fewer libraries My tool kit includes Gr. ADS G 2 ctl Wgrib 2 visualization control file make for Gr. ADS inventory, decode and more Some notable open source codes CDO degrib GDAL NCL RNomads
Reading a grib 1/2 file 1. Reading the WMO grib format documentation and writing code based on the documentation. STOP! HALT! 2. Use a high-level program to read the grib file. ex. Gr. ADS, NCL, RNomads 3. Use a utility to convert to a familiar format. IEEE: my favorite method, many programs Net. CDF: many programs GIS: GDAL, degrib and others CSV: wgrib 2 My. SQL: wgrib 2 4. Use a library ECMWF ECCodes, formerly ECMWF Grib API Pygrib UCAR's library
Writing Grib 1 1. Use a library ECMWF NCEP Pygrib? UCAR 2. Use a high-level program to write the grib file. ex. NCL, Gr. ADS (lats 4 d, does Gr. ADS use std ids? )
Writing Grib 2 1. Use a high-level library ECMWF Pygrib NCAR 2. Use a high-level program to write the grib file. ex. NCL, Gr. ADS (g 2 grb), wgrib 2
Other Formats Net. CDF common in universities, 1 source of software (Unidata) Grids and observations API is defined Easy to add more metadata Easy to neglect metadata COARDS convention → CF conventions Netcdf 4 files are bigger than grib 2 HDF used at NASA, used in netcdf 4, 1 source of software (The HDF group) GIS various formats
Questions?
- Slides: 30