An OPe NDAP server for CORIOLIS data centre

  • Slides: 16
Download presentation
An OPe. NDAP server for CORIOLIS data centre • Context and functions • Architecture

An OPe. NDAP server for CORIOLIS data centre • Context and functions • Architecture and technical description • Planned improvements T. Loubrieu, A. Forest (IFREMER)

DAP 4 COR : An OPe. NDAP server for CORIOLIS data centre CORIOLIS is

DAP 4 COR : An OPe. NDAP server for CORIOLIS data centre CORIOLIS is an operational data centre for ocean in-situ measurements. As such CORIOLIS : Collects measurements for various observation systems (ARGO profilers, measurement from research ships, XBT from Volontary observation ships, moorings, …) Apply various quality controls on datasets. Disseminates datasets as asked by users (ocean forecasting systems, ARGO, Carbocean, …) The datasets are fully managed in an Oracle RDBMS until they are disseminated as net. CDF files in ftp directories or requested through a GUI web interface.

DAP 4 COR : external usage requirements Open a standard interface on CORIOLIS database

DAP 4 COR : external usage requirements Open a standard interface on CORIOLIS database : Enable interactive subsetting of the datasets (even with scripts) Provide an oceanography oriented, high level, interface so that data management and low level information are hidden to the end-user. Promote CORIOLIS data centre interoperability Use DAPPER ocean in-situ vertical profile DAP model. DAP 4 COR is not intended to be distributed and installed outside of CORIOLIS data centre.

DAP 4 COR : internal requirements Demonstrate a robust and highly capable federating service

DAP 4 COR : internal requirements Demonstrate a robust and highly capable federating service for dissemination : Improve reactivity in the development of new dissemination tools. Facilitate the maintenance of the multiple dissemination tools by making them independant from the low level RDBMS model. Avoid datasets redundancy (Oracle RDBMS / net. CDF files) to ease data update management. Better resources management by using a unique RDBMS pool of connexion. Use J 2 EE architecture (java + tomcat) because of system infrastructure and available skills.

DAP 4 COR : functions DAP 4 COR is Web Application enabling : OPe.

DAP 4 COR : functions DAP 4 COR is Web Application enabling : OPe. NDAP request interpretation for ocean in-situ profile dataset subsetting OPe. NDAP, DAPPER-like response generation from RDBMS profile data. DAP 4 COR Dataset configuration (on SQL criteria on the RDBMS native storage). DAP 4 COR command line client for saving extraction as Net. CDF ARGO profiles file.

DAP 4 COR : request syntax example http: //www. ifremer. fr/dap 4 cor/ server

DAP 4 COR : request syntax example http: //www. ifremer. fr/dap 4 cor/ server base URL dap 4 cor-coriolis dataset . separator [dds|das|ascii|dods] output extension ? separator location. profile. PRES projected variable , separator location. profile. TEMP projected variable & separator location. TIME>1136505600000. 0 selection criteria & separator location. TIME<1138579200000. 0 selection criteria & separator location. attributes. PLATFORM_NUMBER="DACF" selection criteria

DAP 4 COR : user’s subsetting criterias available Platform code ARGO profiling float Geographical

DAP 4 COR : user’s subsetting criterias available Platform code ARGO profiling float Geographical area Time Vertical reference : depth, pressure Available measurements Measurement values T/S diagram On a small ocean box Dissolved oxygen Profiles for 2 weeks Mediterranean Salinity profiles One month

DAP 4 COR : dataset configuration Main data (location, time, measurements) are stored in

DAP 4 COR : dataset configuration Main data (location, time, measurements) are stored in the same 3 tables. Ancillary data are distributed into numerous exotics tables (CORIOLIS « expert » domain). DA 4 COR Datasets can be configured according to user needs by defining low level SQL criteria on the Oracle DBMS : Subset of platforms (gliders, elephant seals, moorings array…) Instrument types (CTD, XBT, XCTD, …) Quality assesment level (real time, near real time, delayed mode…) Geographical area (ocean basin, WOCE reference section), by using Oracle Spatial functions (GIS). … Thanks to that, a most of the RDBMS model complexity is hidden to the user. The most relevant datasets are configured on server as SQL pre-defined SQL criteria without data redundancy.

DAP 4 COR : dataset configuration examples WOCE reference section, for example AX 7

DAP 4 COR : dataset configuration examples WOCE reference section, for example AX 7 Mediterannean sea Measurements from elephant seal Gliders dataset GTS datasets (BATHY, TESAC messages) Sea Surface dataset (for satellite calibration) Pre-processed profiles : pressure/depth conversion, interpolation on standard vertical levels, parameters conversion… To Be Done …

DAP 4 COR : response format Ocean in-situ vertical profiles as proposed in PMEL’s

DAP 4 COR : response format Ocean in-situ vertical profiles as proposed in PMEL’s DAPPER server : One sequence of profiles For each profile : First level information (time, location + quality control results) Metadata (profile attributes) Sequence of the measured levels (vertical reference value, measured parameters values, quality control information) Maximum number of vertical profiles in each response : 1000

DAP 4 COR : command line client (dap 4 cor 2 net. CDF) Windows/Unix

DAP 4 COR : command line client (dap 4 cor 2 net. CDF) Windows/Unix or Linux portable software. Uses DAP 4 COR opendap request as input. Saves the subset as Net. CDF ARGO profile. Command line tool which can be used as batch (usefull for operational requirements).

DAP 4 COR : system architecture JDAP 1. 1. 7 DAP 4 COR Datasets

DAP 4 COR : system architecture JDAP 1. 1. 7 DAP 4 COR Datasets configuration DAP 4 COR tomcat JDBC connection pool tomcat application server Components we rely on : JDAP 1. 1. 7 Tomcat application server (enabling load balancing) Tomcat JDBC connection pool Oracle RDBMS Oracle spatial functions (GIS) Oracle RDBMS Spatial functions

DAP 4 COR : a 3 layers software architecture DAP request interpretation/data model generation

DAP 4 COR : a 3 layers software architecture DAP request interpretation/data model generation based on java API (JDAP 1. 1. 7) In-situ Vertical Profile business interface Implemented in order to manage : raw profiles Or pre-processed profiles (confidential information hidden, vertical reference conversion, interpolated on standard vertical levels…) To Be Done Design as a « profile reader » so that memory overflow does not occur. : It reads the lower layer / write the upper layer very frequently so that the smallest possible amount of data is kept is that layer memory. JDBC convenient result. Sets

DAP 4 COR : profile business interface Main methods : set. Constraint() : initializes

DAP 4 COR : profile business interface Main methods : set. Constraint() : initializes the OPe. NDAP subsetting criterias (dataset, projected parameters, selection parameters) get. Station. Rough. Number() – get. Station. Precise. Number() : maximum profile quota analysis. Prepare() : initializes the requested profiles subset (before reading) Profiles subset reading methods (mapped on DAPPER sequences order) : next. Station() : move to next vertical profile get. Station. Element. Value() : get global information on the current vertical profile (time, location, platform, …) next. Level() : move to next vertical level for the current profile next. Profile. Level. Element. Value() : get and move to next measurement for the current vertical level. Interface implementation is a parameter of the dataset configuration so that we can define, for example : A dataset of pre-processed profiles with homogeneous vertical references (pressure/depth converted if necessary) …

DAP 4 COR : planned improvements Enable big dataset request (thanks to OPe. NDAP

DAP 4 COR : planned improvements Enable big dataset request (thanks to OPe. NDAP indexed requests ? ) Improve performance especially when only the global profile information (time, location, platform code, …) are requested. Implement alternative profile interface for profile pre-processing (vertical level conversion, interpolation on standard vertical levels…) Provide a new Ope. NDAP interface for other CORIOLIS datasets : time series at a fixed location along track observations Keep on following the DAPPER observation data model convention and improve compliance with it.

DAP 4 COR : links Server URL : http: //www. ifremer. fr/dap 4 cor

DAP 4 COR : links Server URL : http: //www. ifremer. fr/dap 4 cor Documentation and related tools : ftp: //ftp. ifremer. fr/ifremer/coriolis/tools/dap 4 cor Containing : FAQ document Java client which enable saving datasets as Net. CDF ARGO profiles file (dap 4 cor 2 netcdf). Matlab client script examples Python client script example