Grib 2 Wesley Ebisuzaki wesley ebisuzakinoaa gov wgrib

Grib 2 Wesley Ebisuzaki (wesley. ebisuzaki@noaa. gov) wgrib 2: Wesley Ebisuzaki, Reinoud Bokhorst, Jaakko Hyvätti, Kristian Nilssen, Karl Pfeiffer, Pablo Romero, Manfred Schwarb, Arlindo da Silva, Niklas Sondell, Sergey Varlamov

Outline what is grib 2, new features Gr. ADS & grib 2 converting from wgrib to wgrib 2 inventory grib 2 files geolocation read grib 2 with wgrib 2 write grib 2 with wgrib 2

What is grib 2 WMO replacement for grib version 1 Transfer of gridded data between met centers Software to read/write grib 2 NCO fortran/C libraries, used by many codes NCAR library, NCL (NCAR command language) ECMWF grib api, used by many codes wgrib 2 others: UK? Japan?

Why grib 1 -> grib 2 Address the limitations of grib 1 Time stamp Only two bytes to store numbers (255 is ok) can combine 2 bytes for 16 bit integer (64 k is ok) Files size limitation Only 128 standard variables, all other are locally defined Grids only defined to 1/1000 of a degree

New features in grib 2 Submessages Improved metadata: time stamps, level information, variable information Default grid resolution is 1 e-6 degrees but can be specified to be higher *** Selectable compression schemes ***

Gone: kpds(5). . kpds(7) Grib 1: variable, type of level, level # Grib 2: variable: table/version, discipline, parameter category, parameter number New tables to modify variables, ensemble, bias, error, chemical type, statistical processing, etc level: most used templates require several #s. wgrib 2, Gr. ADS: hide complexity, similar to grib 1

Forecast timing information Grib 1: 4 bytes: type, time units, 2 bytes Grib 2: size of time stamp is not fixed No overflows, precise descriptions Means are handled is systematic manner in wgrib 2 notation, monthly means look like “ 124@6 hour ave(6 hour fcst)” “ 124@6 hour ave(0 -6 hour ave fcst)” Statistical operations: ave, acc, min, max, rms, etc

Submessages bash-2. 05 a$ wgrib 2 wafsgfs 44. t 18 z. gribf 84. grib 2 1: 0: d=2011011218: HGT: 70 mb: 84 hour fcst: 2: 5409: d=2011011218: TMP: 70 mb: 84 hour fcst: 3. 1: 8037: d=2011011218: UGRD: 70 mb: 84 hour fcst: 3. 2: 8037: d=2011011218: VGRD: 70 mb: 84 hour fcst: 4: 16727: d=2011011218: HGT: 100 mb: 84 hour fcst: 5: 22349: d=2011011218: TMP: 100 mb: 84 hour fcst: Note: 3. 1 and 3. 2, UGRD and VGRD are in same grib message, share grid description and metadata. Gr. ADS, wgrib 2: process by “fields” not by messages. Fortran: extra complexity Submessages are good for irregular grids. Only need to store one copy of the latitude/longitude values.

Bitmaps & Undefined Values Grib 1: uses bitmap for undefined grid value overhead = 1 bit per grid point Grib 2: use bitmap or special value(s) All packing methods can use bitmap Complex packing can use special values better to use special values if available Gr. ADS, wgrib 2 hides this complexity fortran code: extra complexity

Scan order Decode a grib field – get a list of grid pt values “scan order” specifies how you fill the grid Grib 1: we: sn and we: ns are common Grib 2: 16 different orders but 3 are common, we: sn, we: ns and “Glahn” (NDFD) wgrib 2: default is to convert data to we: sn but other options include we: ns and raw but best to keep in we: sn order

Grib 1, Grib 2 – storing grid data Store scaled integers Value = (i * 2^E + ref) * 10^D ref = min value of the field so i >= 0 E, D are scaling factors for the field Global model: E = 0, D is specified Mesoscale: E and D are specified ECMWF/wgrib 2(default): D = 0, E determined by # bits wanted

Packing schemes: IEEE store as IEEE real numbers Allows full precision files (ex. restart files) No compression Easy to write (no grib library needed) Not supported by NCO software

Simple packing Similar to grib 1 packing Store scaled (positive) integers If biggest scaled integer < 2**N Store integer as N bits No compression relative to grib 1 Fast to write and read common

PNG Compress file by “PNG” compression same png compression as. png files Better packings are available Not recommended

JPEG Uses JPEG 2000 to compress the grid data Best compression for data without undefineds Slow to write and read Used by NCO

Complex Packing Start with scaled integer, 3 variants C 1: store values (Vi) C 2: store increments (Vi - Vi-1) C 3: store increments of increments Split into multiple regions each region stores packed integer with varying minimum value and size

Complex Packing Heuristic methods to determine regions Some programs pack more efficiently Best to use special values rather than bitmap Undefined encoded as a run of N “special values” Reading complex packed files is fast. Similar to grib 1/simple packing.

Complex Packing: file size C 1/C 2/C 3 is best for noisy/somewhat noisy/smooth fields Can produce better complex packing by making C 1, C 2, C 3 and choosing the best (C 0) C 0 produces smaller files than JPEG for files with undefined values C 0 typically 10%-20% bigger than jpeg

Examples: file size 1/14/2011 gfs. t 00 z. sfluxgrbf 06. grib 2 1/14/2001 gfs. t 00 z. prgb 2 f 06 jpeg 55. 1 MB C 0: 56. 5 MB * 36. 0 MB 1/14/2011 nam. t 00 z. edasf 06. tm grib 1: 213 MB jpeg: 65. 1 MB C 0: 65. 9 MB * 50. 7 MB grib 1: 114 MB jpeg: 38. 1 MB C 0: 34. 1 * 24. 8 MB * use best of jpeg, C 1, C 2, C 3

Compression: summary JPEG slow to write and read Complex packing is fast to read Complex packing is smaller for files with undefineds otherwise usually 10 -20% bigger Tapes: best compression (jpeg+C 0) Speed: C 0 Multitasking routines: jpeg <-> C 0. . C 3

Grib 2: common tasks Display with Gr. ADS Inventory Slicing and dicing files Exporting data Averaging Making regional subsets Making grib 2 content: conversion, new data

Grib 2 & Gr. ADS Use Gr. ADS v 2, g 2 ctl Supports ensemble Supports minutes Supports 3 time mapping schemes -0, -b, -verf (use “”)

Gr. ADS: time dimension Mapping grib time to Gr. ADS time dimension. : d=2009060500: APCP: surface: 324 -330 hour acc fcst: d=2009060500 analysis time (-0) d=2009060500+324 hours start of verf period (-b) d=2009060500+330 hours end of verf period (“”) d=2009060500: UGRD: 1000 mb: 330 hour fcst d=2009060500 analysis time (-0) d=2009060500+330 hours verf time (-b or “”)

Gr. ADS examples Reference, analysis time: Use -0 g 2 ctl -0 ipvhnl. gdas. %y 4%m 2%d 2%h 2. grb 2 >anl. ctl gribmap -0 -i anl. ctl

Gr. ADS examples start of verification time period: Use -b option g 2 ctl -b gfs. t 12 z. pgrb 2. f%f 3 >fcst. ctl gribmap -b -i fcst. ctl must be gribmap v 2 end of verification time period: No extra options g 2 ctl gfs. t 12 z. pgrb 2. f%f 3 >fcst. ctl gribmap -i fcst. ctl must be gribmap v 2

Gr. ADS examples (forecast) ensemble: files: gep 01. t 00 z. pgrb 2 cf* gep 02. t 00 z. pgrb 2 cf* gep 03. t 00 z. pgrb 2 cf* g 2 ctl -ens “ 01, 02, 03” gep%e. t 00 z. pgrb 2 cf%f 3 >ens. ctl gribmap -i ens. ctl Can use -b in the above example.

Introduction to wgrib 2 was planned Easy to add modules People donated modules Geolocation Name <-> parameter routines Update of tables from NCO web pages mysql export netcdf export

wgrib 2 changes Each option corresponds to a subroutine call Order is important command line can look like a small program Many options, to view wgrib 2 will list main options wgrib 2 -help all will list all options Functionality Geolocation, if, for, conversion to other formats, writing grib 2, interpolation, multitasking

Inventory Simple inventory: wgrib 2 -s GRB_FILE wgrib 2 GRB_FILE Verbose inventory -s is assumed, no need -4 yr wgrib 2 -V GRB_FILE -s and -V are inventory options

Inventory has changed old names are mostly the same new levels, old levels are expanded years are 4 digits long time stamps can be more complicated NO KPDS 5, KPDS 6, KPDS 7 or equiv g 2 lib info

Slicing and Dicing Grib 1: wgrib IN -s -4 yr | egrep (. . . ) | wgrib IN -i -grib -o OUT Grib 2: wgrib 2 IN | egrep (. . . ) | wgrib 2 IN -i -grib OUT -4 yr, -s are gone, now 4 digit year codes Notice: -grib -o OUT becomes -grib OUT Minor changes to scripts but will need to debug Watch out for scripts that write binary/ieee/text files default is we: sn rather than raw. (we: ns in global)

Efficiency: -match, -not wgrib 2 IN | egrep A | egrep -v B | wgrib 2 -i IN. . . A, B are regular expressions 4 processes, read file up to 2 x wgrib 2 IN -match A -not B. . . multiple -match and -not options More efficient: 1 read and 1 process

Efficiency: multiple write wgrib 2 IN -match ': TMP: ' -grib ht. grb wgrib 2 IN -match ': HGT: ' -append -grib ht. grb wgrib 2 IN -match ': RH: ' -grib rh. grb OR with one command line wgrib 2 IN -if ': TMP: ' -grib ht. grb -if ': HGT: ' -grib ht. grb -if ': RH: ' -grib rh. grb -if is ended by i/o operation or -fi

Geolocation A need to know the values at specified lat-lon. A need to know the lat-lon of the grid points. -lon LON LAT -ijlat I J -lola ARGS interpolation -csv FILE -spread FILE grid value nearest to lat-lon values and lat-lon of grid point (i, j) makes lola file with NN CSV with lat-lon values

wgrib 2: exporting data Many output formats ieee -ieee OUTPUT. bin big endian IEEE bin -bin OUTPUT. bin machine (IEEE) text -text OUTPUT. txt csv -spread OUTPUT. txt comma sep. values csv -csv OUTPUT. txt another CSV mysql -mysql. . . to mysql netcdf -netcdf. . . write netcdf file

Exporting to fortran Use -bin OUT. bin (binary) Default is -header open(. . . , form='unformatted', access='sequential') If you use -no_header option, get direct access file. open(. . , recl=nx*ny*4, form='unformatted', access='direct') Scan order is we: sn by default, use -order raw for original order

Exporting to fortran Order and number of the records can change! Be prepared for potential changes Specify the records that you want (and count) Sort the inventory before decoding to order wgrib 2 IN | egrep “: (UGRD|VGRD): (2|3)00 mb” | sort -t: -k 3, 3 -k 4, 4 -k 5, 5 -k 6, 6 | wgrib 2 IN -i -ieee sorted. bin

Only 1 record to read? call system('grep ": DPT: surface: 60 hour fcst: " ds. td. bin. inv | wgrib 2 -i ds. td. bin -bin fort. 20', status) if (status /= 0) stop 8 open(unit=20, file='fort. 20', form='unformatted', access='sequenti al') read(20) rdata close(20)

Averaging Use wgrib 2: -ave dt GRIB. OUT dt = N(time unit), ex 6 hr, 1 dy The records have to be in sequence d=2011010100: UGRD: 1000 mb: anl: d=2011010106: UGRD: 1000 mb: anl: d=2011010112: UGRD: 1000 mb: anl: d=2011010118: UGRD: 1000 mb: anl: wgrib 2 IN | sort -t: -k 4, 4 -k 5, 5 -k 6, 6 -k 3, 3 | wgrib 2 -i IN -set_grib_type c 3 -ave 6 hr AVE_OUT

Regional Subsets -small_grib lon. E: lon. W lat. S: lat. N OUT -ijsmall_grib i 1: i 2 j 1: j 2 OUT Cookie cutter of existing grid (i, j) coordinates -new_grid A B C OUT Uses IPOLATES like copygb A, B, C are new grid definition

-new_grid subset of copygb calls NCEP iplib library not all functionality of iplib is supported -new_grid is optional, not in opn version

using -new_grid Type of interpolation -new_grid_interpolation TYPE bilinear, neighbor, budget etc Interpolation parameters -new_grid_iopts I: J: K iopts(i) array Type of winds (a must) -new_grid_winds TYPE earth, grid Notes: bilinear interpolation is default Must set the type of winds (earth/grid)

-new_grid: example -new_grid latlon 0: nlon: dlon lat 0: nlat: dlat OUT. grb wgrib 2 IN -new_grid_winds earth -new_grid latlon 0: 360: 1 90: 181: -1 OUT Some NCEP grids are built-in: wgrib 2 IN -new_grid_winds -new_grid ncep grid 3 OUT input projections: latlon, mercator, polar, lambert, gaussian output projections: latlon, gaussian, lambert, nps, sps

RPN Calculator

Why a calculator? CSV: change units from Kelvin to Celcius Net flux = downward flux – upward flux wind speed = sqrt(u*u + v*v) (3 -6 hour prate) = 2*(0 -6 hour prate) – (0 -3 hour prate) growing degree days 500 -1000 mb thickness = hgt(500 mb) – hgt(1000 mb)

RPN calculator old fashioned, stack based calculator 10 registers 0. . 9, stack is 10 deep works on grids operators +, -, *, /, pop, exc, trig, sqrt, log, exp, etc ex. convert from K to C: -rpn “ 273. 15: +”

Saving calculator results The calculation replaces the grid data read from the grib file. The new data can be written out with the standard wgrib 2 output operators except for grib data, you have to use -grib_out. Trivial example: changing sign of LHTFL wgrib 2 -match “: LHTFL: ” -rpn “-” -grib_out neg_lhtfl. grb

calculating wind speed wgrib 2 IN. grb -match GRD -if “: UGRD: 500 mb: ” -rpn sto_1 -fi -if “: VGRD: 500 mb: ” -rpn sto_2 -fi -if_reg 1: 2 -rpn “rcl_1: sq: rcl_2: sq: +: sqrt: clr_1: ” -set_var WIND -grib_type jpeg -grib_out out. grb

Changing metadata previous slide: -set_var WIND many options to alter the metadata -set_var HGT -set_date 2011010212 -set_ftime “ 12 hour fcst” -set_lev “ 200 mb” -set_metadata FILE

Writing out new grib 2 file Select compression -set_grib_type (jpeg|s|c 1|c 2|c 3) Compress data and write new grib message -grib_out NEW. grb Examples of combining -rpn, -set_* and grib_out to write a new grib message are on web site

Writing grib: -import_bin allows over write the grid data wgrib 2 T. GRB -import_bin NEW. bin -grib_out NEW. grb -import_bin, reads a binary file and overwrite data -grib_out writes out a new grib 2 file useless example – same variable/time/level change metadata by: -set_date, -set_var, -set_lev, set_ftime, etc

Conversion Scripts Gr. ADS can read many file formats such as binary, grib, netcdf, hdf If Gr. ADS can read the file, can convert the data into grib 2 by the g 2 grb. gs script Used in converting MERRA to grib 2 Limitations: only lat-lon grids supported, Lambert/polar converted to lat-lon grid

Summary Grib 2 is a more complicated format than grib 1 Compression Fixes limitations of grib 1 Some grib 2 fields have no grib 1 versions Conversion from wgrib to wgrib 2 is straight forward Take advantage of the new features in wgrib 2 export data to csv, netcdf, mysql

Summary: can use wgrib 2 to write grib 2 files -rpn calculator -import_bin import binary data use -set options to modify metadata -grib_out to write grib files multi-processing can be done (see web site)

RLE: run length encoding store as N < 255 discrete IEEE values radar data (Japan) Works well in this application Limited use

Grib 1 Conversion Scripts Everybody has a grib 1 ->grib 2 program cnvgrib, ECMWF api, grib 1 to 2. pl Grib 1 -> grib 2 (used for CFSR conversion) wgrib -> ieee file wgrib + perl -> metadata file cnvgrib -> grib 2 template wgrib 2 + ieee + metadata + template = grib 2 file 228 lines (1 cpu version), 278 lines (multi-cpu)

Multiprocessing: simple example

Parallel operations wgrib 2 can be used for cpu intensive operations such as compression and regional subsetting. On the CCS, a 30 sec job only took 8 seconds when run on 4 copies of wgrib 2 -for_n I: J: K option uses do-loop syntax to select the fields to process. -for_n/-for can also be used to select blocks to process -for 10: 20 will process msgs 10 -20

Example: multicpu -> jpeg # 1 -cpu: wgrib 2 IN -set_grib_type jpeg -grib_out OUT mkfifo pipe 1 pipe 2 pipe 3 wgrib 2 IN -for_n 1: : 3 -set_grib_type jpeg -grib_out pipe 1 & wgrib 2 IN -for_n 2: : 3 -set_grib_type jpeg -grib_out pipe 2 & wgrib 2 IN -for_n 3: : 3 -set_grib_type jpeg -grib_out pipe 3 & gmerge OUT pipe 1 pipe 2 pipe 3 IN = input grib file, OUT = output grib file