Overview of the ODP Data Provider Sergey Sukhonosov
- Slides: 19
Overview of the ODP Data Provider Sergey Sukhonosov National Oceanographic Data Centre, Russia Expert training on the Ocean Data Portal technology, Buenos Aires, Argentina, 05 -08 July 2011
Scope Data Provider provides access to data and metadata of the local data systems. When the wrapper is installed in the local data system, the latter becomes a data source for the distributed data system. The Data Provider processes the local data sets and in a semi-automated way generates the discovery metadata. These services are based on the Open. DAP data (point, profile and grid) structures and specific metadata model based on ISO 19115. Expert training on the Ocean Data Portal technology, Buenos Aires, Argentina, 05 -08 July 2011 2
How to become a data provider The data center which agrees to be the ODP data provider should provide: the middleware for communications: application server available for HTTP protocol, installation of the ODP Data Provider software OR use light Data Provider extension, registration of the data source and its discovery metadata, support of the local data system Expert training on the Ocean Data Portal technology, Buenos Aires, Argentina, 05 -08 July 2011 3
How to become a data provider Light Data Provider extension: Allows integration of data from data centres unable to install the Data Provider software. In this case the owner of the Data Provider must create new user with login and password and provide this information to remote user Data centres can use remote Data Provider for providing catalogs of data to ODP distributed system
Supported data storage types Data in relational Database Management System (DBMS): Oracle, MS SQL Server, My. SQL, Postgre. SQL, etc. ; Structured data files with non-hierarchical data formats (CSV, TSV, …) Object data files: documents, images, data which has format not supported by ODP technology Links (web site pages, web-applications, URLs, webservices) Expert training on the Ocean Data Portal technology, Buenos Aires, Argentina, 05 -08 July 2011 5
How to provide the data Data in database inside local network with the Data Provider software Data Provider DBMS files Local network of Data Centre Structured data files: upload to the Data Provider server or specify URL to data files location (FTP, HTTP) FTP, HTTP Internet Expert training on the Ocean Data Portal technology, Buenos Aires, Argentina, 05 -08 July 2011 6
Software architecture To install and operate the ODP Data Provider, it is recommended to use a computer with the following minimum characteristics: CPU 1 GHz or more, 2 GB RAM, 300 Mb hard disk space. It is recommended to use a dedicated computer for Data Provider installation. Data Provider web application JBoss Application server Apache web-server PHP Database access service Di. GIR Dedicated Server running under Windows or Linux OS DBMS Structured and objective data files
Network requirements HTTP and SOAP protocols must be available JBoss AS port must be opened in firewall settings IP-address verification: Data Provider receives requests only from the Integration Server
Software requirements Middleware software J 2 SDK 6 Application server JBoss 4. 2. 3. GA (or later) Web server Apache 2. 2. x (or later) PHP 5. x (or later)
Data exchange protocol Request and response messages: XML document via HTTP or SOAP protocol Transport data file : Net. CDF file for data from DBMS and structured files, data file “as is” for object data files.
Functional requirements The local data administrator should provide: design of resources; data source registration; discovery metadata registration; provision of data
How to prepare metadata (design of resource) • • This work can be handled before or at once after the Data Provider installation and includes the following actions: local data structure analysis assessment of resource contents assesment of data granularity discovery metadata content (title, area coverage, temporal extent, …)
Local data structure analysis Local data storage types: SQL-oriented Database Management System: Plain table structure – 1 parameter in 1 column. Pn P 1 P 2 P 3 . . . P 1 P 2 P 3 … Joining more than couple of tables is not a clever solution – use views if it’s possible
Local data structure analysis Plain structured data files (CSV, TSV, etc. ). List of formats is extendable by adding new data parser; Structured data file can include comment lines, different data value separator; Objective files – no restrictions. Object files must be accessible via HTTP or uploaded to the Data Provider server. //M 4321; M 4322; M 4401; M 4400; M 4050; M 4311; M 4312; M 4303; P 0696_00; P 0735_00; P 0964_00; ///P 0507 _00; P 0001_00; //Forecast from 00 hr 20. 05. 2009. Producing center - NCEP/NOAA. 0. 3125; 999. ; 2009 -05 -20 00: 00; 2009 -05 -20 12: 00; 0; -89. 761; 0. 0000; 0; 0. 6; -9. 1; 0. 01; -37. 3; -36. 6; 0. 3125; 999. ; 2009 -05 -20 00: 00; 2009 -05 -20 12: 00; 0; -89. 761; 0. 3125; 0; 0. 7; -9. 1; 0. 01; -37. 3; -36. 6; 0. 3125; 999. ; 2009 -05 -20 00: 00; 2009 -05 -20 12: 00; 0; -89. 761; 0. 6250; 0; 0. 8; -9. 1; 0. 01; -37. 2; -36. 6; 0. 3125; 999. ; 2009 -05 -20 00: 00; 2009 -05 -20 12: 00; 0; -89. 761; 0. 9375; 0; 0. 8; -9. 1; 0. 01; -37. 2; -36. 6; 0. 3125; 999. ; 2009 -05 -20 00: 00; 2009 -05 -20 12: 00; 0; -89. 761; 1. 2500; 0; 0. 9; -9. 1; 0. 01; -37. 2; -36. 6;
Resource granularity The Data Provider has the possibility to support various types of the data granularity, i. e. it is possible to create discovery metadata, search and deliver full datasets or specified data pieces (logical data units) of datasets - single cruise or data profile, single buoy or single coastal station data - and etc. The resource can be presented as a single unit (called a single resource) or as a set of resource instances (called a serial resource) reflecting the local data granularity of the local data system. The data granularity level is adjusted in the process of the local dataset registration.
Resource life-cycle define the schedule for updating of discovery metadata, check the local data availability using the report submitted by Integration Server, take the needed actions to provide the data source actuality (connection, data files storage availability),
How to generate metadata Use web-interface of the Data Provider to generate, update and maintenance discovery metadata
Case studies Connecting data to the Data Provider (object files, structured data files, SQL-databases, HTTP location) Creating new discovery metadata using existing one as a template Managing users of the “Light” Data Provider Connecting uploaded data via “Light” Data Provider
Thank you for attention! JOIN OCEAN DATA PORTAL! HAVE YOU SHARED YOUR DATA TO OCEAN DATA PORTAL? www. oceandataportal. org http: //odp. oceandataportal. net 19
