GRIB 2 vs Net CDF Evaluation of the
GRIB 2 vs Net. CDF: Evaluation of the Technical Aspects Bruce Wright (IT Architect) © Crown copyright Met Office
Contents This presentation covers the following areas • Data Format Overview • GRIB vs Net. CDF • Summary © Crown copyright Met Office
Data Format Overview © Crown copyright Met Office
GRIB • Developed by WMO for the exchange of gridded data • Allowing detailed description of a huge variety of grids, parameters, processes, represented by codes that reference external tables • Portable, implemented as octets (groups of 8 bits): Section Name Length Notes 0 Indicator 16 bytes "GRIB" initially 1 Identification 21 bytes 2* Local Use 3 ** Grid definition 52 bytes? 4 *** Product definition 14 -50 bytes? 5 *** Data representation typically 21 bytes Dependent on compression 6 *** Bit-map 6 bytes + bitmap Optional section 7 *** Data 5 bytes + data 8 End 4 bytes © Crown copyright Met Office Optional section - may contain anything "7777" *, *** - can be nested
Net. CDF • Developed by Unidata to facilitate the access and sharing of array-orientated data in a form that was self-describing and portable • Simple, but flexible, data model based around variables, dimensions and attributes. • Climate and Forecast (CF) convention (developed for sharing climate model and NWP forecast data) offers a widely-used standard for metadata (esp. through standard names) • Users interact with the data and metadata in a file through an API that allows things like querying the number of variables (data objects) in a file, reading or writing slices of data to the variables, defining the variables in a file after creation, etc. • Unidata provide a library of routines for interacting with Net. CDF files © Crown copyright Met Office
GRIB vs Net. CDF © Crown copyright Met Office
Interoperability GRIB Net. CDF Standard WMO standard Used by NMSs and other areas of operational meteorology (e. g. Aviation) De facto standard Used by wider fluid earth sciences community (esp. research & academia) Machine Independent Yes Self. Describing No – interpretation requires: Yes – low-level definition of dimensions, variables and - WMO-agreed tables attributes - Local tables COTS Support Limited Mainly WMO-sponsored More extensive ERSI Arc. View 9. 2 OGC web service standard Transmission (& Archiving) Direct Access – whole file required © Crown copyright Met Office Sequential – record-based access
Metadata GRIB Net. CDF (CF) Comprehensive Yes Standard range of descriptors agreed out of considerable experience Less so than GRIB for operational meteorology …but potential to be extremely comprehensive Flexible Yes, but… Use of local tables can be ‘clumsy’ Highly Use of global and variable attributes Human Readable No Yes Self. Describing No – refers to external tables Yes Extensible Yes, but… WMO process slow Local table limitations Highly CF managed through active community mailing list © Crown copyright Met Office
Interface GRIB Net. CDF (CF) Standard Interface No, but… ECMWF GRIB API NCEP software… Yes (from Unidata) Net. CDF is really an interface, rather than a format Languages ECMWF – C, FORTRAN 77, Java NCEP – C, FORTRAN 90, (Tcl/Tk GUI, MS-Windows degrib) C, C++, FORTRAN 77, FORTRAN 90 Java Bindings to Perl, python, ruby, IDL, Tcl, (PV-Wave), MATLAB, R, Open. DX Platforms ECMWF has tested on Linux, AIX, HP-UX, IRIX Most platforms Installation & Use ? Easy Linux install, learning curve but not severe © Crown copyright Met Office
Compression GRIB Net. CDF (CF) Packing Good Simple packing Complex, with different algorithms Crude packing only Lossless Compression Proposals to include LZW, Weather-Huffman, JPEG 2000 Fraunhofer Institute developed compression (factor 2. 5 on 16 bit data) No for Net. CDF 3, but Cornell developed extension Net. CDF 4 (HDF 5) supports various Licencing Issues None known, except for specialist solutions Possible limitations for some algorithms © Crown copyright Met Office
General GRIB Net. CDF Dimensions 2 D fields only GRIB 2 allows these to be grouped for higher dimensions Small Reads Whole file must be parsed to Efficient, as ‘header’ index file provides direct access Initial Write Sequential format allow efficient streaming © Crown copyright Met Office N-dimensional e. g. 3 spatial dimensions and time in a single array File has to be constructed using API
Summary © Crown copyright Met Office
Summary • CF-compliant Net. CDF (CF-Net. CDF) • Net. CDF interface developed by Unidata to facilitate the access and sharing of array-orientated data in a form that was self-describing and portable (the format being machine-independent) • Highly flexible data management solution for multi-dimensional gridded data • Very widely used in atmospheric and oceanographic sciences community • CF (Climate-Forecast) metadata convention offers, arguably, the best metadata standard available within this community • GRIB 2 • Developed as WMO standard to provide an efficient, machineindependent format for the exchange of gridded data by National Met Services • No standard interface, although several have been developed • Metadata is code-based, needing to cross-reference external tables, with a highly specified metadata ‘vocabulary’ and layout and no indexing of the data © Crown copyright Met Office
- Slides: 13