Weathertop Consulting LLC Serverside OPe NDAP Analysis A

Weathertop Consulting, LLC Server-side OPe. NDAP Analysis - A General Approach Utilizing Legacy Applications through TDS Roland Schweitzer Weathertop Consulting, LLC Steve Hankin and Ansley Manke NOAA/PMEL

Highlights n Server-side analysis n Motivation LAS as OPe. NDAP Client and Server n Evolution of the server implementation n Community “call to action” n Summary n Weathertop Consulting, LLC

Server-side Analysis In general server-side analysis is a computation made by an OPe. NDAP server at the request of a client. n The specification of the computation is transmitted to the server via the OPe. NDAP URL. n Weathertop Consulting, LLC

Motivation n We are interested in server-side analysis for use with the Live Access Server (LAS). n n n Primarily as a way to implement comparisons between data defined on different grids We want our implementation to leverage the analysis capabilities of legacy applications like Ferret and Gr. ADS. We want to use our experience running legacy applications (like Ferret) from within a Java runtime environment. Weathertop Consulting, LLC

The Live Access Server (LAS) A highly configurable Web server designed to provide flexible access to geo-referenced scientific data Weathertop Consulting, LLC

LAS Product Server Metadata (XML) client SQL Backend Service Ferret Backend Service Ferre t product Local net. CDF data Local RDBMS metadata product request XML (REST) back end request (SOAP) DRDS Backend Service OPe. NDAP server net. CDF data DRDS server Remote RDBMS LAS Architecture Weathertop Consulting, LLC

LAS Product Server Metadata (XML) user SQL Backend Service Ferret DRDS Backend Service product Suppose the variables are on different grids? net. CDF data 1 metadata product request XML (REST) back end request (SOAP) OPe. NDAP server net. CDF data 2 Comparing OPe. NDAP datasets Weathertop Consulting, LLC

LAS as an OPe. NDAP Server Data on grids which are available via LAS are guaranteed to be geo-referenced and at least COARDS compliant. n We can often “repair” (including regridding) the data and/or metadata by associating a script of Ferret commands with the data source in the LAS configuration. n Wouldn’t it be nice to make these “repaired” data available via OPe. NDAP? n Weathertop Consulting, LLC

The Ferret Data Server FDS made this possible. n FDS provides an OPe. NDAP view of the data being served by LAS and makes any transformations specified by the associated script before serving the data. n FDS also implements server-side analysis (including the ability to pass in external data sources). n Weathertop Consulting, LLC

A GDS Digression The Gr. ADS Data (DODS) Server is the first implementation of this concept. n In fact, FDS used the Anagram framework upon which GDS is built. n Both GDS and FDS use the Java Runtime environment to invoke the associated legacy app (Gr. ADS or Ferret) to do the heavy lifting. n Weathertop Consulting, LLC

FDS Capabilities n FDS took advantage of several characteristics of the underlying engine (Ferret). n n New "virtual" data variables can be defined Can build the metadata (net. CDF header described by dimensions, coordinate variables and the structure of data variables) without performing any heavy calculations for both data read from files and “virtual” data variables Only performs calculations when the data are requested Only calculates the minimal set needed to fulfill the current request Weathertop Consulting, LLC

FDS Evolution n n Keep these advantages and evolve the implementation. The Java net. CDF library allows new data container formats to be plugged-in by implementing the I/O Service Provider interface. Once “plugged-in” clients using nj 22 have access to the data from this container. We implemented a Ferret I/O Service provider which can read Ferret command scripts and direct Ferret to perform the calculations as needed to satisfy data requests. Weathertop Consulting, LLC

Example % more data/simple 4. jnl use levitus_climatology let/d=levitus_climatology temp_20 =temp[d=levitus_climatology, z=0: 20@sum] set var/title="surface heat content"/units="deg C“ temp_20[d=levitus_climatology] Weathertop Consulting, LLC

Example dncdump -c http: //porter. pmel. noaa. gov: 8920/thredds/dods. C/mydata/simple 4. jnl float TEMP(ZAXLEVITR, YAXLEVITR, XAXLEVITR) ; TEMP: units = "DEG C" ; TEMP: long_name = "TEMPERATURE" ; TEMP: _Fill. Value = -1. e+10 f ; TEMP: missing_value = -1. e+10 f ; TEMP: dataset = "levitus_climatology. cdf" ; float TEMP_20(YAXLEVITR, XAXLEVITR) ; TEMP_20: dataset = "levitus_climatology" ; TEMP_20: direction = "IJ" ; TEMP_20: units = "deg C" ; TEMP_20: long_name = "surface heat content" ; TEMP_20: missing_value = "-1. 00000 E+34" ; TEMP_20: virtual = "true" ; Weathertop Consulting, LLC

IOSP* Application Netcdf. File The low-level part of the Net. CDF-Java version 2. 2 architecture I/O service provider Net. CDF-3 Net. CDF-4 HDF 5 NIDS is. Valid GRIB open GINI Nexrad DMSP Java Runtime read. Data Ferret Gr. ADS *Stolen directly from John Caron with only this measly acknowledgement. Weathertop Consulting, LLC

The THREDDS Data Server TDS is an OPe. NDAP server. n nj 22 I/O Service Providers can be plugged in to TDS. n The combination of the Ferret I/O Service Provider and TDS (aka F-TDS) serves via OPe. NDAP data which are represented by Ferret command scripts (both data read from disk by Ferret and virtual data computed on-the-fly by Ferret). n Weathertop Consulting, LLC

F-TDS and Server-Side Analysis n n A Data. Source handler can also be plugged in to TDS which allows custom handling of OPe. NDAP requests based on the contents of the HTTPServlet. Request Object (and by implication the URL). We built such a Data. Source handler which recognizes URL with embedded analysis expressions. The three groups of input to the server-side analysis were copied from GDS. The three sets are: n n Data sources (e. g. OPe. NDAP URLs). Analysis “commands” which are implementation specific. Weathertop Consulting, LLC

An Analysis URL http: //machine: port/thredds/dods. C/ _expr_ {dataset 1, dataset 2, . . . } {expression 1; expression 2; . . . } Part of the original GDS specification, not {region} necessary and not often used with FDS. URLsuffix? constraint Weathertop Consulting, LLC

FDS-Specific Example http: //host. gov: 9090/thredds/dods. C/data/coads _expr_ {http: //host. gov: 9090/thredds/dods. C/data/levitus} {DIF=SST[d=1]-TEMP[d=2, g=SST[d=1]]}. asc? DIF Weathertop Consulting, LLC

Community Action n Can we define a implementationindepedent syntax for server-side analysis requests? Some common operations (averaging, differencing, linear interpolation) with standard name and syntax n Server-specific (native) operations n n What mechanism? n An encoded XML string? n We n use this technique in LAS with good success A simple command language? Weathertop Consulting, LLC

XML Analysis Expression <datasets> <dataset id=1>"my_local_dset. nc"</dataset> <dataset id=2>"http: //remote_dset. nc"</dataset> </datasets> <operation> <name>DIFF</name> <arg pos=1 dset=1>sst</arg> <arg pos=2 dset=2>sst</arg> </operation> Weathertop Consulting, LLC

With “native” Operation <datasets> <dataset id=1>my_local_dset. nc</dataset> <dataset id=2>http: //remote_dset. nc</dataset> </datasets> <native_operation> LET diff = sst[d=2]-sst[d=1] </native_operation> Weathertop Consulting, LLC

Summary Server-side analysis is critical for the future LAS. n Net. CDF Java 2. 2 and the THREDDS Data Server is a great platform for implementing this type of analysis with a legacy analysis application (like Ferret). n A community-developed server-side analysis framework would make it easier to get the advantages of server-side analysis from other servers. n Weathertop Consulting, LLC

Acknowledgments n From COLA n n Jennifer Adams Brian Doty Joe Wielgosz n Former TMAP n n Richard Rogers Yonghua Wei From Unidata n n John Caron Ethan Davis Weathertop Consulting, LLC