OPe NDAP The Opensource Project for a Network

OPe. NDAP (The Open-source Project for a Network Data Access Protocol) BOM Tutorial Introduction October 15, 2007 pfox@opendap. org 1 BOM, Melbourne, VIC 20071015 (Fox)

Distributed Oceanographic Data System (DODS) Conceived in 1993 at a workshop held at URI. Objectives were: – to facilitate access to PI held data as well as data held in national archives and – to allow the data user to analyze data using the application package with which he or she is the most familiar. Basic system designed and implemented in 1993 -1995 by Gallagher and Flierl with NASA and NOAA funding. From 1994 to present it has been extended with NASA, NOPP, NSF and NOAA funding. BOM, Melbourne, VIC 20071015 (Fox) 2

Some Definitions DAP = Data Access Protocol Model used to describe the data; Request syntax and semantics; and Response syntax and semantics. OPe. NDAP The software; Numerous reference implementations; Core/libraries and services (servers and clients). OPe. NDAP Inc. OPe. NDAP is a 501. c(3) non-profit corporation; Formed to maintain, evolve and promote the discipline neutral DAP that was the DODS core infrastructure. BOM, Melbourne, VIC 20071015 (Fox) 3

Some Definitions Syntax The computer representation of a data object - the data types and structures at the computer level; e. g. , T is a floating point array of 20 by 40 elements. Semantics The information about the contents of an object; e. g. , T is sea surface temperature in degrees Celsius for a certain region of the Earth. 4 BOM, Melbourne, VIC 20071015 (Fox)

Considerations with regard to the development of DAP and OPe. NDAP Many data providers Many data formats Many different client types Many different semantic representations of the data Many different security requirements BOM, Melbourne, VIC 20071015 (Fox) 5

Broad Vision A world in which a single data access protocol is used for the exchange of data between network based applications regardless of discipline. A layer above TCP/IP providing for syntactic and semantic consistency not available in existing protocols such as FTP. BOM, Melbourne, VIC 20071015 (Fox) 6

Practical Considerations The broad vision: Is syntactically achievable, but Is not semantically achievable, at least not in the near term. 7 BOM, Melbourne, VIC 20071015 (Fox)

OPe. NDAP Inc. Mission Statement To maintain, evolve and promote a data access protocol (DAP) and reference implementation software (OPe. NDAP) for the syntactically consistent exchange of data over the network. The DAP should provide syntactic interoperability across disciplines and allow for semantic interoperability within disciplines. BOM, Melbourne, VIC 20071015 (Fox) 8

OPe. NDAP Inc. Vision Statement To achieve the mission: OPe. NDAP Inc. will be non-profit. Easier to obtain federal funds. The DAP is more likely to be adopted. OPe. NDAP software will be open source. More likely to be adopted. Need community contributions to software. OPe. NDAP Inc. will mix implementation with research. Implementation - to encourage use. Research - to keep the protocol current. BOM, Melbourne, VIC 20071015 (Fox) 9

OPe. NDAP Inc. Vision Statement (cont) OPe. NDAP Inc. will rely primarily on federal funding. Unlikely to obtain private funding for middleware. Development to be use case driven. Aligned with Vision/Mission. Strategic direction will be sought from an Advisory Board consisting of data system experts and with input from you the community of developers. OPe. NDAP Inc. will seek partners. OPe. NDAP Inc. will utilize community working groups to develop ‘standards’ related to the DAP, OPe. NDAP BOM, Melbourne, VIC 20071015 (Fox) 10

The Data Access Protocol (DAP) The DAP has been designed to be as general as possible without being constrained to a particular discipline or world view. The DAP is a discipline neutral data access protocol; it is being used in astronomy, medicine, earth science, … Provides data format and location, and data organization transparency Is metadata neutral BOM, Melbourne, VIC 20071015 (Fox) 11

DAP comparisons • File-based – Grid. FTP/FTP – HTTP – SRB • Service-based – Open-Geospatial Consortium, WCS, WMS, WFS, … – Virtual Observatory (Astronomy), SIAP, SSAP, STAP, … BOM, Melbourne, VIC 20071015 (Fox) 12

The Astronomy approach; datatypes as a service Limited interoperability VO App 1 VO App 2 VOTable VO App 3 Simple Image Access Protocol Simple OGC: {WFS, WCS, WMS} and SWE {SOS, Spectrum Access SPS, SAS} Protocol use the same approach VO layer Simple Time Access Protocol Lightweight semantics DB 1 DB 2 DB 3 Limited meaning, hard coded Limited extensibility ………… BOM, Melbourne, VIC 20071015 (Fox) 13 DBn

Who is using DAP/ OPe. NDAP? • Science examples – PMEL with their Tsunami inundation modeling – Ocean regional modelers to extract open boundary conditions – Visualization of data sets using MATLAB/IDL/… • Service examples – Live Access Server – Mapserver – OGC services and OPe. NDAP data access (future) – Digital Library Service - metadata and catalogue info BOM, Melbourne, VIC 20071015 (Fox) 14

Data Access Protocol (DAP 2) - Current DAP 2 currently a NASA/ESE ‘Standard’ Current servers implement DAP 2 DAP 3 DAP 2 + XML responses (implemented) BOM, Melbourne, VIC 20071015 (Fox) 15

DAP 4 - Late 2007 (? ) DAP 4 improvements over DAP 3: Additional datatypes Swath Blob - GIF, MPEG, … Additional functionality Check sum Modulo The additional datatypes will enable the DAP to be used in a wider variety of circumstances and are a direct response to users’ requests. BOM, Melbourne, VIC 20071015 (Fox) 16

What DAP means to me • Data access and transport • Response types: DAP objects versus file type – A DAP URL is essentially an HTTP URL with additional restrictions placed on the abs-path component. – DAP 2 -URL = "http: //" host [ ": " port ] [ abs-path] • • abs-path = server-path data-source-id [ ". " ext[ "? " query ] ] server-path = [ "/" token ] data-source-id = [ "/" token ] ext = "das" | "dds" | "dods" – The server-path is the pathname to the server, whereas data-source-id is the pathname to the data. BOM, Melbourne, VIC 20071015 (Fox) 17

OPe. NDAP V 3 Architecture Client Cgi style access Data CGI-style access Uses web server HTTP protocol Several request and response types Reads data files, Databases, et c. , returns info May return DAP 2 objects or other data Client can be application, web browser or specialized server/service BOM, Melbourne, VIC 20071015 (Fox) 18

OPe. NDAP V 4 (Hyrax) Architecture Client OLFS BES Data OPe. NDAP Lightweight Front end Server (OLFS) Receives requests and asks the BES to fill them Uses Java Servlets Does not directly ‘touch’ data Multi-protocol Back End Server (BES) Reads data files, Databases, et c. , returns info May return DAP 2 objects or other data Does not require web server BOM, Melbourne, VIC 20071015 (Fox) 19

Binaries Generated There approximately 80 binaries built on a nightly basis. They are built for the following platforms/operating systems: Linux FC 4 FC 5 Mac. OS-X (universal binaries when possible) Windows XP, win 32 Java 1. 5 (Tomcat 5. 5) IRIX (in four variants), Solaris, AIX, OSF BOM, Melbourne, VIC 20071015 (Fox) 20

Communication Website (http: //www. opendap. org and test. opendap. org) SVN - Code repository (http: //scm. opendap. org: 8090/trac/browser/) Trac - Task/milestone repository (http: //scm. opendap. org: 8090/trac/) Telecons Management - ~ Weekly Developers - Weekly on Monday 11 am MT Twiki -> Media. Wiki, for Management Developers Coming soon http: //docs. opendap. org (documentation on-line in Media. Wiki form) opendap-tech@opendap. org e-mail list opendap@opendap. org for announcements (low vol. ) BOM, Melbourne, VIC 20071015 (Fox) 21

Releases/Support Periodic releases (announced on www. opendap. org and on email lists) Support OPe. NDAP software (support@opendap. org) opendap-tech@opendap. org discussions 22 BOM, Melbourne, VIC 20071015 (Fox)

OPe. NDAP and security • Both administrators and developers must consider security aspects of OPe. NDAP software • First developer telecon each month addresses security • This will be addressed during this week 23 BOM, Melbourne, VIC 20071015 (Fox)

OPe. NDAP and authentication • • ESG II/III Australian Access Federation (AAF). htpasswd Tomcat authentication 24 BOM, Melbourne, VIC 20071015 (Fox)

ESG Security Services 25 BOM, Melbourne, VIC 20071015 (Fox)

ESG Security Services - Client Interaction 26 BOM, Melbourne, VIC 20071015 (Fox)

OPe. NDAP Community Working Groups Modeled after best practices of IETF, W 3, OGC, IEEE, ISO, and others Working Groups: Authentication Security Server-side Functions Virtualization (Aggregation) Server-side processing Geospatial Interoperability Hyrax and *DS (TDS, GDS, FDS, etc. ) Semantics DAPPER net. CDF C++ client Response types Metrics Asynchronous transactions DAP 4 Relational Database access via DAP 27 Wiki (http: //docs. opendap. org/index. php/Working_Groups) BOM, Melbourne, VIC 20071015 (Fox)

OPe. NDAP Community Working Groups Terms of Reference http: //docs. opendap. org/index. php/Terms_of_Reference Abridged version: Each OPe. NDAP Working Group is established to apply members' expertise in their focus area to produce specific deliverables or outcomes Types: Software and Documentation Minimum 3 people 6 month time-frame 28 BOM, Melbourne, VIC 20071015 (Fox)

OPe. NDAP System Elements The OPe. NDAP data access protocol is used by a variety of system elements. Clients Browser Interfaces Data System Integrators (ODC) Servers Processing Servers Aggregating Servers - OPe. NDAP chains 29 Ancillary Information Services BOM, Melbourne, VIC 20071015 (Fox)

Clients make requests and receive responses via the DAP. Clients convert data from the OPe. NDAP data model to the form required in the client application. 30 BOM, Melbourne, VIC 20071015 (Fox)

OPe. NDAP Clients Internet net. CDF Java net. CDF C Ferret Gr. ADS IDV Web Browser Vis. AD NCL Client nc. Browse NCL IDL Client Matlab Client Access Matlab IDL py. DAP Excel Arc. GIS OPe. NDAP Data Connector 31 BOM, Melbourne, VIC 20071015 (Fox)

OCAPI A pure OPe. NDAP C API (OCAPI) for the client-side Applications: DAP-aware ‘commands’ for commercial analysis programs (e. g. , IDL) Scripting tools (e. g. , Perl) BOM, Melbourne, VIC 20071015 (Fox) 32

OPe. NDAP System Elements The OPe. NDAP data access protocol is used by a variety of system elements. Clients Browser Interfaces Data System Integrators (ODC) Servers Processing Servers Aggregating Servers - OPe. NDAP chains 33 Ancillary Information Services BOM, Melbourne, VIC 20071015 (Fox)

Browser interfaces 34 BOM, Melbourne, VIC 20071015 (Fox)

OPen. DAP - a Data System Integrator Data GCMD Connector OPe. NDAP GFDL net. CDF URI HDF GSFC Binary NVOD S ODC Matlab Ferret Vis. AD Gr. ADS IDV Access IDL Excel nc. Browse BOM, Melbourne, VIC 20071015 (Fox) 35

OPe. NDAP System Elements The OPe. NDAP data access protocol is used by a variety of system elements. Clients Browser Interfaces Data System Integrators (ODC) Servers Processing Servers Aggregating Servers - OPe. NDAP chains 36 Ancillary Information Services BOM, Melbourne, VIC 20071015 (Fox)

Servers receive requests and provide responses via the DAP. Servers convert the data from the form in which they are stored to the DAP. Servers provide for subsetting of the data and more. BOM, Melbourne, VIC 20071015 (Fox) 37

OPe. NDAP Servers CDM ESML net. CDF HDF 4 Data General net. CDF Data HDF 5 DSP Tables SQL FITS CDF Flat Binary CEDAR Data Data HDF 5 HDF 4 JGOFS DSP FITS JDBC Free. Form CDF CEDAR Internet 38 BOM, Melbourne, VIC 20071015 (Fox)

OPe. NDAP Servers (specialized processing) py. DAP ESG FDS GDS DAPPER CODAR TDS Data Data General net. CDF OPe. NDAP GRIB BUFR OPe. NDAP net. CDF OPe. NDAP CODAR net. CDF OPe. NDAP Internet BOM, Melbourne, VIC 20071015 (Fox) 39

Servers may also provide other services Directory traversal. Browser-based form to build URL. Ascii or other representations of data. Metadata associated with the data. 40 Server side functions. BOM, Melbourne, VIC 20071015 (Fox)

OPe. NDAP Aggregation Servers py. DAP ESG FDS GDS DAPPER CODAR TDS JGOFS Data Data General net. CDF OPe. NDAP GRIB BUFR OPe. NDAP net. CDF OPe. NDAP CODAR net. CDF OPe. NDAP General Internet BOM, Melbourne, VIC 20071015 (Fox) 41

The Aggregation Server: An Example net. CDF Data Set File DSP Data Set File DSP Aggregation Server Local OPe. NDAP HTML, GIF Matlab Client Matlab BOM, Melbourne, VIC 20071015 (Fox) 42

OPe. NDAP’s Hyrax (‘Server 4’) • Uses a modular architecture to support different application-level protocols – Data access using DAP 2 (DAP 3) – Catalogs using THREDDS – Browsing using HTML and ASCII • Modules for data access – Different file types – Potential for database and scripting • Modules for commands – Commands provide varying operations for different protocols BOM, Melbourne, VIC 20071015 (Fox) 43

OPe. NDAP V 4 (Hyrax) Architecture Client OLFS BES Data OPe. NDAP Lightweight Front end Server (OLFS) Receives requests and asks the BES to fill them Uses Java Servlets Does not directly ‘touch’ data Multi-protocol Back End Server (BES) Reads data files, Databases, et c. , returns info May return DAP 2 objects or other data Does not require web server BOM, Melbourne, VIC 20071015 (Fox) 44

Response to client Grid. FTP DAP 2 HTTP DAP 2 Request Formulation** DAP 2 (Grid. FTP, HTTP) SOAP-DAP (HTTP) BES Request from client OPe. NDAP Lightweight Front end Server THREDDS Info output HTML form ASCII output BOM, Melbourne, VIC 20071015 (Fox) 45

BES Framework BES Commands/ XML Documents Net. CDF 3 PPT* Initialization/ Termination DAP 2 Access Data Catalogs HDF 4 Network Protocol and Process start/stop activities Commands** Free. Form … Data Store Interfaces 46 Data BOM, Melbourne, VIC 20071015 (Fox) Data *PPT is built in (other protocols) **Some commands are built in

Additional material 47 BOM, Melbourne, VIC 20071015 (Fox)

OPe. NDAP relies on projects • To guide user-based requirements for application needs as well as OPe. NDAP core development (use cases) • To provide maintenance and evolution of the core software and documentation • Currently: 7 active projects, covers next ~2 -3 years, 2 pending proposals • …… 48 BOM, Melbourne, VIC 20071015 (Fox)

Success What constitutes success of the OPe. NDAP mission? Adoption of the DAP across a broad range of disciplines with extensive use in several of these. In order to achieve this the DAP must do the following: It must be sufficiently flexible, all encompassing, etc. that it can be used across a broad range of data types. Its implementation must be robust, secure, easy to use, provide for a broad range of services, etc. The funding stream must be robust. Active and engaged developer and user community BOM, Melbourne, VIC 20071015 (Fox) 49

Risks It is still the case that Some other data access protocols are seen as more attractive regardless of whether or not they are, or Other data access protocols are developed because their community is not aware of OPe. NDAP or of what its capabilities are. We will compare and contrast some of these today BOM, Melbourne, VIC 20071015 (Fox) 50

To Succeed OPe. NDAP must make sure that: It’s (server and client) capabilities are well known across a broad range of disciplines. The data model is inclusive. The implementation is robust and meets user’s needs. The DAP coexists with other protocols. It has a robust funding base. It has extensive documentation. BOM, Melbourne, VIC 20071015 (Fox) 51

Robust Funding? What is an appropriate level of activity (funding) for OPe. NDAP? What should OPe. NDAP be doing? Core only? + Clients and servers? + Demonstration projects? Should OPe. NDAP be a small staff (core only) or…? What priority should be assigned to the elements currently being developed? Which of these go beyond the core? DAP 4 AIS - metadata consistency THREDDS - Catalogs Toolkits - Matlab, IDL, … Grid capabilities OTHERS? BOM, Melbourne, VIC 20071015 (Fox) 52

Ancillary Information Service • • Current capability: Attributes only Client-side only Local and remote resources Local resource databases The AIS enables users to augment the metadata for a data source in a controlled way without requiring write access to the original data. By using the DAP, users are also isolated from data format issues. BOM, Melbourne, VIC 20071015 (Fox) 53

AIS enhancements • Remote resource databases • AIS server • AIS for variables These enhancements will greatly expand the usefulness of the AIS: Remote resource databases and an AIS server will enable third-party ‘AIS sites’ which may be sponsored by project offices or institutions, et cetera. AIS for variables will enable adding metadata which are stored as ‘data. ’ 54 BOM, Melbourne, VIC 20071015 (Fox)

Proposed AIS Server Client linked w/DAP Software 0 3 1 AIS Server Data Source 2 AIS Resource 0. Client requests metadata from the AIS server (which appears no different from any other DAP server). 1. The AIS server gets metadata from data source 2. The AIS server gets matching the AIS resource using the AIS database and merges it into the metadata. 3. The AIS server returns resulting the metadata object. BOM, Melbourne, VIC 20071015 (Fox) 55