TIGGE GRIB to Net CDF converter Doug Schuster

TIGGE, GRIB to Net. CDF converter Doug Schuster (NCAR/ECMWF)

Topic Outline l l l Introduction Ensemble Net. CDF File Structure Converter Highlights Challenges Future Plans

GRIB to Net. CDF Conversion GRIB File Net. CDF File t, EGRR, 1 Metadata d, ECMF, 2 t, EGRR, 2 t (1, 2, 3, 4) d, EGRR, 1 t, ECMF, 1 d, EGRR, 2 d, ECMF, 1 d (1, 2, 3, 4) Gather metadata and message locations Create Net. CDF file structure Populate Net. CDF parameter arrays (1, 2, 3, 4) represents ensemble member id (Realization)

Ensemble Net. CDF File Structure l Net. CDF File format – Based on available CF conventions – File organization built according to Doblas-Reyes (ENSEMBLES project) proposed Net. CDF file structure – Provides grid/ensemble specific metadata for each member n Data Provider n Forecast type (perturbed, control, deterministic) – Allows for multiple combinations of initialization times and forecast periods within one file. n Pairs of initialization and forecast step

Ensemble Net. CDF File Structure l l Net. CDF Parameter structure (5 dimensions): – Reftime – Realization (Ensemble member id) – Level – Latitude – Longitude “Coordinate” variables are use to describe: – Realization n Provides metadata associated with each ensemble grid. – Reftime n Allows for multiple initialization times and forecast periods to be contained within one file

Ensemble Net. CDF File Structure l Coordinate variables for Realization – Provide ensemble member metadata n n Institution Forecast Type Realization 1 Param(Realization) Grid 1 Grid 2 Grid 3 Grid 4 Institution(Realization) ECMF EGRR Forecast. Type(Realization) cf -At fixed Reftime and Level 2 pf 3 cf 4 pf

Ensemble Net. CDF File Structure l Coordinate variable for Reftime – Provide Forecast Times for each Initialization n Leadtime Reftime 20071012, 12 z 20071013, 00 z Param(Reftime) Leadtime(Reftime) Grid 1 Grid 2 Grid 3 24 12 18 -At fixed Realization and Level

Ensemble Net. CDF File Structure l CDL Examples (reftime and coordinate variable) int reftime(reftime) ; reftime: data_type = "long" ; reftime: units = "hours since 1950 -01 -01 00: 00" ; reftime: standard_name = "forecast_reference_time" ; reftime: long_name = "Time of model initialization" ; int leadtime(reftime) ; leadtime: data_type = "int" ; leadtime: units = "hours" ; leadtime: standard_name = "forecast_period" ; leadtime: long_name = "hours since forecast_reference_time" ;

Ensemble Net. CDF File Structure l CDL Examples (realization and coordinate variables) int realization(realization) ; realization: data_type="int" realization: units = "1" ; realization: standard_name = "realization" ; realization: long_name = "Number of the simulation in the ensemble" ; char forecast_type(realization, string 2) ; forecast_type: data_type = "char" ; forecast_type: standard_name = "forecast_type" ; forecast_type: long_name = ”Forecast type" ; char institution(realization, string 4) ; institution: data_type = "char" ; institution: standard_name = "institution" ; institution: long_name = "Institution responsible for the forecast system" ;

Ensemble Net. CDF File Structure l CDL Examples (Ensemble parameter example) short t(reftime, realization, level, latitude, longitude) ; t: data_type = "short" ; t: units = "K" ; t: standard_name = "air_temperature" ; t: coordinates = "leadtime institution forecast_type" ; t: scale_factor = 0. 00190138198339349 ; t: add_offset = 253. 616560226903 ; t: _Fill. Value = -32767 s ;

Conversion Tool Highlights l Tool User options – Pack to 16 bit shorts. – Select grids based on forecast type n n n l Perturbed Control Deterministic Control and Deterministic All 3 forecast types Large file support with Net. CDF 3. 6. 0 or later.

Conversion Tool Highlights l Tool Limitations – – First version TIGGE data Regular Lat/Lon grids Requires Multiple Outside Libraries n Net. CDF n GRIB API n Jasper (JPEG 2000) – Hard Coded Cell Methods – Slow

Conversion Tool Highlights l Tool Performance – GRIB-2 Simple Packing to Net. CDF 32 BIT n GRIB-2 size x ~2 – GRIB-2 Simple Packing to Net. CDF 16 BIT n Similar size – GRIB-2 JPEG 2000 to Net. CDF 32 BIT n GRIB-2 size x ~8 – GRIB-2 JPEG 2000 to Net. CDF 16 BIT n GRIB-2 size x ~4

Conversion Tool Highlights ECMWF sl params, 2 x 2 Degree Global Grid, simple packing, converted on 3. 4 GHz processor Linux Box

Challenges l Parameter names – Map available GRIB-2 names to CF names. n Product Discipline, Parameter Category, Parameter Number – Map dynamically generated names and GRIB-2 encoding to CF parameter names n Abstract Definitions n Not a simple Discipline/Category/Number combination. – Require statistical processing and level information. Will generally be CF name + Cell method – Get all centers to use this style of parameter encoding. n NCEP only uses the “Discipline/Category/Number” parameter encoding scheme. – GRIB API doesn’t recognize hard coded parameters that include statistical processing (e. g. maximum temperature). n

Challenges l User Tools – Compatibility with Net. CDF ensemble file structure. Deal with the ensemble dimension. n Properly ingest, compute valid times. n Read Coordinate variables to provide ensemble member metadata. n Net. CDF 3. 6. 0 or later required for large file support. n

Future Plans l l l Implement on TIGGE portals to provide option for Net. CDF format data download. Generalize converter tool to encompass all types of GRIB data. Add GRIB API “Definitions” files for CF parameter names, and cell methods. Support Additional GRID types Explore Net. CDF 4 – API with greater functionality and flexibility. Add user options.