Recent Work in Progress John Caron June 3

Recent Work in Progress John Caron, June 3, 2003 • THREDDS development – Dynamic Catalogs: DQC, Resolvers – IDD Data Server – ADDE Cataloger • Net. CDF development – Net. CDF Markup Language (Nc. ML) – More efficient Java I/O (NIO) – Net. CDF/DODS/HDF 5 Data Models

THREDDS Catalogs HTTP Server Catalog. Ref. xml Catalog Generator Catalog. xml Catalog. Ref. xml Data Server DODS, ADDE, FTP, HTTP Client Application Datasets hostname. edu

Dynamic Catalogs = Services HTTP Tomcat Server Catalog. xml Catalog Generator Catalog Service Catalog. Ref. xml Query Resolver Service DQC. xml Resolver Service URI URL Data Server DODS, ADDE, FTP, HTTP Datasets hostname. edu Client Application

Dataset Query Capability (DQC) • XML document. • Describes what the user can ask for as a set of orthogonal “selections”. • On the client, a “query URL” is formed based on the user’s choices, and sent to the server. • The “query resolver” server finds which datasets satisfy the query and returns a list of real dataset URLs. • The DQC describes the queries that the server is capable of responding to.

Resolver Services • Logical Dataset, eg “latest ETA model run” • Dataset with Service type “Resolver” • On the client, the URI of the logical dataset is sent to the server • The server finds what is available and returns a list of real dataset URLs.

ADDE Cataloger HTTP Tomcat Server Catalog. xml Catalog Service ADDE Catalog. Ref. xml Cataloger Query Resolver Service DQC. xml Client Xxxxx Xxxx Application ADDE Data Server hostname. edu Datasets IDD

Summary IDD Data Server Get as much of the IDD Data feeds available via THREDDS as possible. – NCEP model data (catgen) (DODS) – Level 3 NEXRAD (custom server/DQC) (ADDE) – SSEC/Unidata Satellite data (ADDE Cataloger) (ADDE) – Text Data: Metars, Surface Obs, etc (DQC/custom server), returns text or XML. – Profiler Data (custom server/DQC) (ADDE)

Net. CDF 3 Net. CDF File Open. DAP Dataset Local file Nc. ML Dataset XML Open. DAP HTTP protocol Net. CDF-3 library Virtual dataset API Client Application protocol

Net. CDF Markup Language XML representation of net. CDF metadata, uses XML Schema • Core: existing net. CDF data model • Coordinate System: general and georeferencing coordinate system • Dataset: redefine, aggregate, subset • Luca Cinquini (NCAR/SCD/ESG), John Caron, Ethan Davis, Bob Drach (LLNL), Stefano Nativi (Florence), Russ Rew

Nc. ML Coordinate Systems Convention Parser • ATDRadar • AWIPS • COARDS • CF • CSM • GDV • NUWG • WRF • Zebra Net. CDF File Open. DAP Dataset Netcdf Dataset Nc. ML Dataset XML

Geo. Grids, Geo. Tiffs, Geowhiz! Net. CDF File Open. DAP Dataset Convention Parser Netcdf Dataset Geo. Grid factory Vis. AD / IDV Geo. Grid Dataset WCS Server Geo. Tiff Writer Strange land of GIS Geo. Tiff File Open. GIS WCS

Nc. ML Dataset : “virtual view” Net. CDF File Open. DAP Dataset Nc. ML Dataset XML Parser Java-net. CDF 2. 1 Client Application Net. CDF Dataset

Nc. ML Dataset • Use Nc. ML like CDL, to declare the contents of a net. CDF file. • Add, delete or rename Variables, Attributes, and Dimensions • Subset Variables • Reorder a Variable’s dimensions • Aggregate multiple net. CDF files, a la DODS Aggregation Server • Nc. ML Dataset is a “virtual view” or can make copy to a local net. CDF file.

2: Nc. ML Datasets on a Server Catalog. xml DODS Agg/Netcdf Server DODS, ADDE, FTP, HTTP Dataset XML Parser Client Application Nc. ML Dataset XML Datasets hostname. edu

3: Nc. ML Datasets via Catalogs Catalog. xml Net. CDF File Open. DAP Dataset Nc. ML Dataset XML Catalog/Dataset XML Parser Java-net. CDF v 2. 1. 1 Client Application

NIO • Rewrite ucar. nc 2 I/O layer using java. nio package (currently using ucar. netcdf) • Uses memory mapping, bulk I/O transfer • Prototype has 7 x speedup on large files. • Requires JDK 1. 4+ • HTTP access must be rewritten

NIO vs current Java NIO Current old/new First access small (3. 9 Mb) large (240 Mb) 281 671 3334 28221 2. 4 8. 5 Average next 5 accesses small large 54 290 2239 16367 5. 4 7. 3 • Time in millisecs to sequentially read entire file • Wintel 2 GHz, 1 GB main memory • Java 1. 4. 2 -client

NIO vs optimized C NIO C C/NIO First access small (3. 9 Mb) large (240 Mb) 281 370 3334 19348 1. 3 5. 8 Average next 5 accesses small large 54 2239 . 44. 43 • Java 1. 4. 2 –client vs. VC 6. 0 /O 2 24 953

Net. CDF Data Model Netcdf. File Dimension Variable Attribute Data. Type • byte • char • short • int • float • double Attribute

Open. DAP Data Model Dataset Base. Type Attribute Base. Type • primitive (8) array Dimension • string • array Base. Type • grid Attribute • structure / Attribute • sequence Attribute Base. Type

HDF 5 Data Model Groups File directory structure inside HDF file. Datatype • Fixed point • floating point • date/time • string • bit field Dataset Datatype Data. Space Attribute • Opaque Data storage • Compound • Compact • Reference • External • Enumeration • Layout • Variable length • Array • Indexed • Striped

Possible Extensions to net. CDF data model • Add new data types: – Strings: variable length arrays of bytes, plus an encoding attribute. – Structures: collections of any other element types, allow nested structures. – Vector: a variable length 1 D array of any type. • Allow reusable structure definition = user defined data type. • Allow unnamed, undeclared dimensions = anonymous dimensions. • Allow multiple unlimited dimensions (outer dimension only) • Compression. Push scale/offset into library, allow variable bit sizes. • Explicit support for coordinate variables/axes.

New Net. CDF Data Model Netcdf. File Dimension Attribute Variable Structure Data. Type • byte • short • int • long • float • double • String • Structure • Vector Attribute Data. Type Vector • Length Data. Type

Net. CDF 4 Net. CDF V. 1 and 2 File HDF 5 File Open. DAP Dataset Open. DAP Nc. ML Dataset XML Local file or 4. 0 HTTP protocol Net. CDF 4 library Virtual dataset API Client Application