An Introduction to GRIB 2 Simon Elliott EUMETSAT

An Introduction to GRIB 2 Simon Elliott EUMETSAT simon. elliott@eumetsat. int WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Saturday Afternoon Agenda • Background (i. e. my background!) • Overview (codes, GRID, GRIB 2) • Code Structure (Sections like others, iterations of sections) • GRIB 2 Code Templates (what kinds are there … examples) • Tables (Template tables, code and flag tables, examples) • Identification of parameters (? ) • GRIB 2 compression (methods, explanations of some) WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Sunday Morning Agenda • Procedure for code form update (addns to tables, addition of tables/templates, new editions) • Changes not requiring code change (adding table/template entries for decoders … down stream code change) • Changes requiring code change (always for data provider, plus new editions) • Case studies of Cloud Mask and Precipitation Data (bespoke Encoder and Decoder, example product) • GRIB 2 Interface for NMC (example of NCEP code) • Review/Discussion WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Overview • Intention is to share information efficiently and unambiguously • Data should be accessible by anyone, not encrypted • Format should allow all information to be included (not rounded or skipped) • Format should allow addition of new data types • WMO “codes” developed per data type, TEMP, SYNOP, SATOB, et c. FM 35 TEMP FM 42 AMDAR FM 41 CODAR WORLD METEOROLOGICAL ORGANIZATION FM 13 SHIP RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 FM 88 SATOB FM 87 SARAD FM 86 SATEM FM 85 SAREP

GRID, GRIB 2 • GRID was developed for gridded data (forecast, analysis) – ASCII -> human readable? – ASCII -> big files, slow transfer – Symbolic letters and code tables: F 1 F 2 NNN in Section 0 • F 1 F 2 is originating centre as per C-1, • NNN is catalogue number of grid used by centre • GRIB introduced for binary exchange (storage and transmission efficiency). – Edition 0, 1985 – Edition 1, 1990 • GRIB Edition 2 (i. e. GRIB 2) because Code Table 2 was full, EPS data, . . . WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

GRIB 1 structure • Section 0 Indicator section: “GRIB” • Section 1 Product definition section: unique standard template • Section 2 Grid description section: one of the standard templates describing a type of grid • Section 3 Bit map section • Section 4 Binary data section • Section 5 End section: “ 7777” GRIB uses the concept of Template: « Description of the standardized layout of a set of data items » WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

GRIB usage • • On the GTS By the WAFS Centres ( « ICAO » products) For archiving of fields on MDD Main limitation is: • One parameter on one level for one grid per GRIB field WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

GRIB 1 weaknesses Transmission and archiving of: • Spectral data • Multi-dimension data • Long-range and climate products • Ensemble products (EPS) Also: • No convention for missing data • IEEE not used for floating point data • No support for small time steps • No cross-sections, no time-sections • No Hovmöller Diagrams (ex. Time-longitude) • Limited support for images WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Need for “object oriented” GRIB 2 (more modules = modularity) • Modularity: code and parameter tables referred to through templates • Flexibility: new tables and templates can be added • ALSO: – All GRIB 1 fields can be described in GRIB 2 – More compression schemes (e. g. introduction of JPEG 2000 and PNG) WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

GRIB 2 calendar • • • Experimental GRIB 2 presented at CBS 98 Finalized in Spring 2000 Approved by CBS in Autumn 2000 Operational in November 2001 First products in 2003 WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Generic structure • Identification: GRIB/BUFR/CREX • Header: Date, time, originator, Table table versions. . . driven • Optional section: Metadata codes (potentially XML), private data … generally have this • Data description: What sort of data follows structure • Actual data: here • Closure: “ 7777” WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

GRIB 2 structure • • • Section 0 Indicator section: “GRIB” Section 1 Identification section: data characteristics Section 2 Optional (local) section: anything or nothing Section 3 Grid definition section: geometry of grid used, potential reference to pre-defined grid Section 4 Product definition section: description of data type, e. g. satellite data from spectral bands xx, yy, zz Section 5 Data representation section: packing method used for data, reference value, scale factor et c. Section 6 Bit map section: data present indicators (if used) Section 7 Data section: data values themselves Note: Sections 1 to 7 start with Section 8 End section: “ 7777” section length and number WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Section 0: Indicator section – – – Octets 1 - 4: Octets 5 - 6: Octet 7: Octet 8: Octets 9 - 16: “GRIB” in ASCII (i. e. 71 82 73 66) Reserved (normally set to 0) Discipline (CT 0. 0, e. g. 3 for space) GRIB edition number (2) Total length in octets Messages of length up to ~18 x 1018 (or 264 - 1) bytes, i. e. ~18, 000 Tb WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Section 1 Identification section – – – Octets 1 - 4: Section length Octet 5: Section number (1) Octets 6 - 9: Originating centre and sub-centre Octets 10 - 11: Master and local table version numbers Octet 12: Time significance (CT 1. 2, e. g. 3 for observation time) – Octets 13 - 19: Date / time – Octet 20: Production status of data (CT 1. 3, e. g. 0 for operational data) – Octet 21: Type of data (CT 1. 4, e. g. 7 for processed radar observations) WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Section repetition WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Repeatable Section 0 (Indicator) Section 1 (Identifier) Section 2 (Local use) Section 3 (Grid) Section 4 (Product) Section 5 (Data Repn) Section 6 (Bit map) Section 7 (Data) Section 8 (Ending) Repeatable • • • Repeatable GRIB 2 allows some groups of sections to be repeated for efficiency (not in GRIB 1, one field per message). Non-repeated sections stay in effect

Section repetition (example 1) • • • • Section 0 (Indicator) Section 1 (Identifier) Section 3 (Grid) Section 4 (Product) Section 5 (Data Repn) Section 6 (Bit map) Section 7 (Data) Section 8 (Ending) WORLD METEOROLOGICAL ORGANIZATION 0600 Z 1200 Z 1800 Z Forecast from same centre on same grid for same parameter at same level but from but for different validity times Validity time is in Section 4 (Product Definition) Section 4 to 7 repeated in message RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Section repetition (example 2) • • • • • Section 0 (Indicator) Section 1 (Identifier) Section 3 (Grid) Section 4 (Product) Section 5 (Data Repn) Section 6 (Bit map) Section 7 (Data) Section 8 (Ending) WORLD METEOROLOGICAL ORGANIZATION GRID A GRID B GRID C Forecast from same centre for same validity time for same parameter at same level but from but on different grids Grid is defined in Section 3 (Grid Definition Section) Section 3 to 7 repeated in message RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Grid definition templates • Grid definition is in Section 3 (GRIB 1, Section 2) • Can refer to centre’s own grid or use Grid Definition Template (GDT) to specify details • Many GDTs have been developed, e. g. – – 3. 0: Latitude/Longitude (or equidistant cylindrical, or Plate Carrée) 3. 1: Rotated Latitude/Longitude (or equidistant cylindrical, or Plate Carrée) 3. 2: Stretched Latitude/Longitude (or equidistant cylindrical, or Plate Carrée) 3. 3: Stretched and Rotated Latitude/Longitude (or equidistant cylindrical, or Plate Carrée) – 3. 20: Polar Stereographic – 3. 90: Space view perspective, or orthographic WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

A grid definition template example GDT 3. 90 - Space view perspective or orthographic Floating point numbers stored in 5 bytes as scale factor (one byte) and scaled value (4 byte IEEE floating point) WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

GDT example continued Repeating sections 4 to 7 means this information need not be repeated for multiple fields in one GRIB 2 message WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Product definition templates • Product definition is in Section 4 (GRIB 1, Section 1) • Many PDTs have been developed, e. g. – 4. 0: Analysis or forecast on horizontal level or layer at a point in time – 4. 1 to 4. 4: Various information pertaining to ensemble forecast systems – 4. 5: Probability forecast on horizontal level or layer at a point in time – 4. 7: Analysis or forecast error on horizontal level or layer at a point in time – 4. 20: Radar products – 4. 30: Satellite products • Hybrid sigma levels can be specified in Section 4 WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

A product definition template example PDT 4. 30 - Satellite products WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

PDT example continued Note: Octets 25 to 24 are given for band 1, but are repeated for each contributing band up to NB, as specified in octet 14 WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Code table 4. 3: Type of generating process Code figure 0 1 2 3 4 5 6 7 8 192 - 254 255 WORLD METEOROLOGICAL ORGANIZATION Meaning Analysis Initialisation Forecast Bias corrected forecast Ensemble forecast Probability forecast Forecast error Analysis error Observation Reserved for local use Missing RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Data representation templates • Data representation is in Section 5 (GRIB 1, within Section 4) • Some DRTs have been developed, e. g. – – – 5. 0: Grid point data - simple packing 5. 1: Matrix values at grid point - simple packing 5. 2: Grid point data - complex packing 5. 3: Grid point data - complex packing and spatial differencing 5. 50: Spectral data - simple packing 5. 51: Spherical harmonics data - complex packing • Section 5 also gives total number of data values to be found in Section 7 WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

A data representation template example DRT 5. 0 - Grid point data - simple packing IEEE floating point numbers in 32 bits: seeeeeee emmmmmmmm WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Simple and complex packing • Simple packing – Y x 10 D = R + (X 1 + X 2) x 2 E – Y is original value, D is decimal scale, R is reference, E is binary scale, X 1 is 0, and X 2 is encoded value – In template R, E, D and bits per value are stored – Field values follow sequentially • Complex packing (intended to decrease message size) – Y x 10 D = R + (X 1 + X 2) x 2 E – R, E, D are as for simple packing – X 1 is reference for group, X 2 is scaled value in group (X 1 removed) – Data are split into groups with similar values per group WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Use of D and E D 0 -1 1 0 -1 -2 0 E 0 -3 4 1 -2 -5 5 2 -1 -4 3 Increment 1. 000 1. 250 1. 600 2. 000 2. 500 3. 125 3. 200 4. 000 5. 000 6. 250 8. 000 WORLD METEOROLOGICAL ORGANIZATION D D-1 Increment x 10 Here R = 0, X 1 = 0, X 2 = 1 Increment = Y = original value D -1 -2 0 -1 -2 -3 -1 E 0 -3 4 1 -2 -5 5 2 -1 -4 3 RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Increment 10. 000 12. 500 16. 000 20. 000 25. 000 31. 250 32. 000 40. 000 50. 000 62. 500 80. 000

R, E and D selection example Temperature on balcony: -15. 0 to 35. 0, ± 0. 2 • Set D to give increments in range 1 - 10 – D = 1, i. e scale by 10 – data are now: -150 to 350, ± 2 • Set R to minimum – R = -150. 0 (IEEE floating point) – data are now: 0 to 500, ± 2 • Set E to give required precision – E = 1, i. e. data are in increments of 2 • Set bit width to give required range – Required range = scaled range / increment – Required range = 500 / 2 = 250 so 8 bits R = -150, E = 1, D = 1, bit width = 8 WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Spatial differencing • Saves space for data which vary smoothly (order 1), or data whose variation varies smoothly (order 2), et c. • For order 1, data values, f, define new values, g: – g 1 = f 1, g 2 = f 2 - f 1, …, gn = fn - fn-1 – 1, 2, 3, 5, 7, 8, 9, 11 becomes 1, 1, 1, 2, 2, 1, 1, 2 • For order 2, g values are replaced by h: – h 1 = f 1, h 2 = f 2, h 3 = g 3 - g 2, …, hn = gn - gn-1 – 1, 2, 4, 10, 17, 27, 40, 57 becomes 1, 2, 1, 4, 1, 3, 3, 4 • Minimum value is subtracted to keep values positive Order 1 Complex packing 4 groups WORLD METEOROLOGICAL ORGANIZATION Order 2 Complex packing 3 groups RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Section 6: Bit map section – – Octets 1 - 4: Octet 5: Octet 6: Octets 7 - xx: Length of section Section number (6) Bit map indicator (CT 6. 0) Bit map (if present) Code Table 6. 0: Bit map section Code figure 0 1 - 253 254 255 WORLD METEOROLOGICAL ORGANIZATION Meaning Bit map is present and follows in this section Use bit map predefined by originating centre Re-use previously defined bit map Missing (no bit map present) RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Bit map case study With bit map No data No bit map (2000 x 1) = 4, 000 bits 2000 x 8 2000 = 32, 000 bits 2000 Data (8 bits per pixel) data ( x 1000 x 8) 25, 130, 000 bits total 29, 130, 000 bits Conclusion: bit map may not always save space, but if more than two bits per data point (typically true) it usually will … consider cloud top height data, space corners, et c. WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Updates not requiring code change • Type of changes concerned – Addition of code and or flag table entries • Agreement and implementation – “Fast Track” can be used – Table entries can be approved for pre-operational use within a few months of consideration and successful validation – Full formal CBS approval follows, available around 2 years later • Application updates – – Encoder needs new code / flag table available Encoder software needs updating to insert field Decoder needs new code / flag table available Down-stream application (beyond decoder) needs to be updated to process new field – Decoder software can remain unchanged WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Updates requiring code change • Type of changes concerned – Changes to code structure (new edition), or new GDT, PDT or DRT • Agreement and implementation – ET/DRC considers a request and proposes update – Multilateral validation of proposed update – Full formal CBS approval follows, available around 2 to 4 years later • Application updates – – Encoder needs new software and /or template available Encoder software needs updating accordingly Decoder needs new software and /or template available Down-stream application (beyond decoder) needs to be updated to process new data • Long lead time necessary … consider PUMA example WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Interfacing to third party software • Use of third party generic software simplifies implementation • NOAA provides GRIB 2 software for free download (look at http: //www. nws. noaa. gov/tdl/iwt) • ECMWF provides GRIBEX software - will be / is being updated for GRIB 2 (look at http: //www. ecmwf. int/products/data/software/grib 2. html) • Typically software package is set of functions compiled into a library • User’s application sets up required parameters and calls function from library, which are referenced at run time • Specific (not generic) encoders and decoders are quite simple but less flexible WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Some GRIB 2 data from EUMETSAT • Cloud mask, CLM, every 15 minutes, ~3 km resolution, via EUMETCast or archive request. • Cloud top height, CTH, every 60 minutes, ~9 km resolution, via EUMETCast or archive request. • Cloud analysis image, CLAI, every 180 minutes, ~9 km resolution, via EUMETCast or archive request. • Clear sky reflectance map, CRM, twice per week, ~3 km resolution, via EUMETCast or archive request. • Multi-sensor precipitation estimate, MPE, every 30 minutes, ~5 km resolution, via web site. • Fire detection, FIR, every 15 minutes, ~3 km resolution, via anonymous FTP (evolving algorithm). WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Case study A, precipitation data WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Case study A, precipitation data • GRIB 2 encoded file • Specific decoder program for GRIB 2 precipitation data • Decoder output • Dump of GRIB 2 file WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Case study B, cloud mask data WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Case study B, cloud mask data • Original data to be encoded • GRIB 2 encoded file • Specific en/decoder program for EUMETSAT cloud mask data • Decoder output • Dump of GRIB 2 file WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Case study C, cloud analysis image data GRIB 2 Cloud analysis image WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Case study C, cloud analysis image data • GRIB 2 encoded file • Specific en/decoder program for EUMETSAT cloud mask data • Bit map used WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Case study D, cloud top height data GRIB 2 Cloud top height WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Case study D, cloud top height data • GRIB 2 encoded file • Specific en/decoder program for EUMETSAT cloud mask data • Bit map used • Two repeats of Sections 4, 5, 6 and 7 WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

Sources of GRIB 2 information The information used in preparing this presentation is based on: Guide to WMO Table Driven Code Form Used for the Representation and Exchange of Regularly Spaced Data in Binary Form: FM 92 -XII GRIB Written by Dr Cliff Dey & WMO Manual on Codes, WMO Publication No. 306, Vol. 1, Part B WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005