BUFR CREX and GRIB WMOs Table Driven Data
BUFR, CREX and GRIB: WMO’s Table Driven Data Representation Forms Simon Elliott EUMETSAT simon. elliott@eumetsat. int WORLD METEOROLOGICAL ORGANIZATION CBS OPAG-ISS Expert Team on the Assessment of Data Representation Systems (ET-ADRS) Washington, USA, 23 - 25 April, 2008
Data exchange and codes • a code: “A system of words arbitrarily used for other words, to secure brevity and secrecy” • a cipher: “A secret manner of writing … intelligible only to those possessing the key” • to cipher: “To express by occult characters” Purpose is global exchange of data in an unambiguous and efficient way. WMO “codes” are data representation forms (DRFs) or systems. WORLD METEOROLOGICAL ORGANIZATION CBS OPAG-ISS Expert Team on the Assessment of Data Representation Systems (ET-ADRS) Washington, USA, 23 - 25 April, 2008
Overview of formats · Self Description · Code Structures · BUFR and CREX Tables · Features Common to BUFR and CREX · Differences Between BUFR and CREX WORLD METEOROLOGICAL ORGANIZATION CBS OPAG-ISS Expert Team on the Assessment of Data Representation Systems (ET-ADRS) Washington, USA, 23 - 25 April, 2008
Issues behind move to BUFR/GRIB • Increasing volume and complexity of observations • Changing requirements but fixed codes • Change implies long time scale and costs: equipment manufacturers, training et c. • Changes take about 2 years WORLD METEOROLOGICAL ORGANIZATION CBS OPAG-ISS Expert Team on the Assessment of Data Representation Systems (ET-ADRS) Washington, USA, 23 - 25 April, 2008
Table driven codes • BUFR - Binary universal form for the representation of meteorological data. – Observations, satellite products • GRIB - Processed data in the form of gridpoint values expressed in binary form. – Forecast, analysis, climatology, satellite images and products • CREX - Character form for the representation and exchange of data. – Like simple BUFR but human readable ASCII WORLD METEOROLOGICAL ORGANIZATION CBS OPAG-ISS Expert Team on the Assessment of Data Representation Systems (ET-ADRS) Washington, USA, 23 - 25 April, 2008
General structure Table driven codes generally have this structure WORLD METEOROLOGICAL ORGANIZATION • Identification: GRIB/BUFR/CREX • Header: Date, time, originator, table versions. . . • Optional section: Metadata (potentially XML), private data … • Data description: What sort of data follows • Actual data: here • Closure: “ 7777” CBS OPAG-ISS Expert Team on the Assessment of Data Representation Systems (ET-ADRS) Washington, USA, 23 - 25 April, 2008
Some features • Data are self defining: type, content, edition, section lengths et c. • Header data are at fixed offsets for quick access without decoding • Binary data (GRIB and BUFR) are not really human readable • Optional section can contain anything; use of XML for metadata under continuing discussion WORLD METEOROLOGICAL ORGANIZATION CBS OPAG-ISS Expert Team on the Assessment of Data Representation Systems (ET-ADRS) Washington, USA, 23 - 25 April, 2008
Consider BUFR • Table B - Element descriptors – Start with ‘ 0’, temperature, pressure, et c. • Table C - Operator descriptors – Start with ‘ 2’, change of scale, qc follows, et c. • Table D - Sequence descriptors – Start with ‘ 3’, wind sequence, high accuracy location sequence, et c. • Code and flag tables – Type of cloud, channel combination, et c. WORLD METEOROLOGICAL ORGANIZATION CBS OPAG-ISS Expert Team on the Assessment of Data Representation Systems (ET-ADRS) Washington, USA, 23 - 25 April, 2008
Table B Example: 0 -20 -082 Element descriptor Amount of segment cloud free Entry in class Class 20, observed phenomena WORLD METEOROLOGICAL ORGANIZATION % 0 7 Reference value Unit Element name 0 Scale CBS OPAG-ISS Expert Team on the Assessment of Data Representation Systems (ET-ADRS) Washington, USA, 23 - 25 April, 2008 Data width (bits)
Scale, reference and width Encoding: value = (obs x [10^”scale”]) - “reference” in “width” bits e. g. : Let scale=1, reference=-50, width=7 then, if obs=1. 3, encoded value is 63 allows obs from -5. 0 to +7. 6, steps of 0. 1 Decoding: obs = (value + “reference”) ÷ [10^”scale”] e. g. : Let scale=-2, reference=5, width=4 then, if value=14, decoded value is 1900 WORLD METEOROLOGICAL ORGANIZATION CBS OPAG-ISS Expert Team on the Assessment of Data Representation Systems (ET-ADRS) Washington, USA, 23 - 25 April, 2008
Table C Operator descriptors for more complex tasks, e. g. – Change width – Change scale – Change reference – Quality control follows – First order statistics follow – Define “data present” bit map – Re-use “data present” bit map – Cancel “data present” bit map WORLD METEOROLOGICAL ORGANIZATION CBS OPAG-ISS Expert Team on the Assessment of Data Representation Systems (ET-ADRS) Washington, USA, 23 - 25 April, 2008
Table D • Not strictly required, but … • Important if data are archived and accessed directly in BUFR • Saves space in Section 3 (list of descriptors) … about 1. 5% for EUMETSAT winds • One table D entry represents an agreed list of table B and/or C and/or D descriptors • 3 -01 -011 = 0 -04 -001 Year 0 -04 -002 Month 0 -04 -003 Day WORLD METEOROLOGICAL ORGANIZATION CBS OPAG-ISS Expert Team on the Assessment of Data Representation Systems (ET-ADRS) Washington, USA, 23 - 25 April, 2008
Differences between BUFR and CREX • BUFR is binary, better for telecoms and computers • BUFR has a compression scheme • BUFR mostly uses SI units (flight level is in meters) • CREX is ASCII, easier for humans to read and write • CREX has no explicit compression scheme • CREX allows other units (temperature can be Celsius or Kelvin) In general, BUFR and CREX are quite similar, the difference being that CREX is easier for human encoders/decoders where computers are not available or telecoms are inadequate. . . WORLD METEOROLOGICAL ORGANIZATION CBS OPAG-ISS Expert Team on the Assessment of Data Representation Systems (ET-ADRS) Washington, USA, 23 - 25 April, 2008
Characteristics of user communities Real time NMHSs: Data are needed in time forecast cut-off Systems are in place for the management of data using standard WMO DRFs The data can be monitored/assimilated/archived as is without translating to an alternative format The processing history of the data is important (originator, quality control, statistics, corrections, et c. ) Users outside NMHSs (universities, commercial users, NGOs et c. ): Data are not generally needed in real time Systems are generally not in place for the management of data using standard WMO DRFs The data need to be translated to an alternative format before use The processing history of the data is not important WORLD METEOROLOGICAL ORGANIZATION CBS OPAG-ISS Expert Team on the Assessment of Data Representation Systems (ET-ADRS) Washington, USA, 23 - 25 April, 2008
Strengths Real time NMHS exchange • • • DRF design driven by way data are used- i. e. association of quality control, updates, statistics, corrections One format (BUFR) covers all types of data (although GRIB is more efficient for gridded data) GRIB and BUFR have compressions schemes built in Central management through WMO ensures standardization Tables and templates are separated from the data making exchange more efficient Encoding sequence makes decoding fast Users outside NMHSs • • • WORLD METEOROLOGICAL ORGANIZATION BUFR and GRIB are compact and so many observations can be stored in one file making the data easier to archive Because BUFR/GRIB data contain or refer to necessary metadata such as geolocation or units, no additional guidelines or documentation are required One format (BUFR) covers all types of data (although GRIB is more efficient for gridded data) CREX allows other units (temperature can be Celsius or Kelvin) Software for encoding and decoding is freely available (e. g. from NOAA and ECMWF) Text format of CREX facilitates human interpretation of data and transmission CBS OPAG-ISS Expert Team on the Assessment of Data Representation Systems (ET-ADRS) Washington, USA, 23 - 25 April, 2008
Weaknesses Real time NMHS exchange • • • Generic encoding and decoding software is complex to integrate into an operational system New requirements for representation of data products take a long time to become adopted fully by WMO Complex packing (available in GRIB) can be very efficient but require clever algorithms and significant memory/disk space to establish the right way to split up the data for compression BUFR Editions up to and including 4 do not readily support the representation of data with a large dynamic range of values Local tables and templates employed by data providers are required in order to decode the data WORLD METEOROLOGICAL ORGANIZATION Users outside NMHSs • • • Generic encoding and decoding software is complex to implement Some decoders are specific to a certain type of product and cannot be used for any others Before any data can be encoded or decoded, the user has to have access to the tables and templates which were used for the encoding Applications for the analysis and display of data in BUFR and CREX are difficult to get with a limited budget The non NMHS community as a whole has limited exposure to BUFR/GRIB/CREX. This can lead to a lack of available experience, and reluctance to adapt to and adopt these formats. CBS OPAG-ISS Expert Team on the Assessment of Data Representation Systems (ET-ADRS) Washington, USA, 23 - 25 April, 2008
Opportunities Real time NMHS exchange • • From Edition 5 onwards, BUFR will allow the encoding of IEEE floating point numbers. This will allow the representation of data with a large dynamic range such as chemical constituents GRIB supports various forms of compression including JPEG 2000 and PNG BUFR could easily be used in other disciplines such as oceanography, volcanology, et c. by the development of appropriate Master Tables GRIB could be more widely for climatological data WORLD METEOROLOGICAL ORGANIZATION Users outside NMHSs • • BUFR could easily be used in other disciplines such as oceanography, volcanology, et c. by the development of appropriate Master Tables GRIB could be more widely for climatological data CBS OPAG-ISS Expert Team on the Assessment of Data Representation Systems (ET-ADRS) Washington, USA, 23 - 25 April, 2008
Threats Real time NMHS exchange • • • Continuing long delay in modifying or adding descriptors or templates could drive the user community to other more adaptive formats A proliferation of local table entries reduces the benefit of standardization and can deter use Ready access to the internet for data transfer removes the possibility for constraining the DRS used to those supported by the GTS. WORLD METEOROLOGICAL ORGANIZATION Users outside NMHSs • A dichotomy could develop if BUFR and GRIB are not accepted by the non-NMHS community and simultaneously maintained for real time exchange. This would necessitate development and support of appropriate tools for interfacing between the two communities CBS OPAG-ISS Expert Team on the Assessment of Data Representation Systems (ET-ADRS) Washington, USA, 23 - 25 April, 2008
Conclusion ü WMO table driven DRFs (DRSs) are standardized, flexible, and support compression ü BUFR was developed in direct response to the way meteorological data are managed for numerical weather prediction ü Software for encoding and decoding BUFR/CREX and GRIB is freely available ~ Applications for displaying and analysing BUFR/CREX and GRIB data are not freely available, but do exist ~ Updates of the official DRFs is too slow to keep up with users requirements • Thanks to Aydın Gürol Ertürk of the Turkish State Meteorological Service for his input relating to the view of users outside the NMHSs WORLD METEOROLOGICAL ORGANIZATION CBS OPAG-ISS Expert Team on the Assessment of Data Representation Systems (ET-ADRS) Washington, USA, 23 - 25 April, 2008
- Slides: 19