OPe NDAP 4 Data Server Hyrax James Gallagher

OPe. NDAP 4 Data Server – Hyrax James Gallagher and Nathan Potter 21 Feb 2007 OPe. NDAP Developer’s Workshop Feb 21 -23 2007

But first… • These are Hyrax “Despite the unexceptional appearance of modern hyraxes, the order has a remarkable prehistoric lineage and hyraxes may be the closest living relatives of the elephant. ” Wiki. Pedia. OPe. NDAP Developer’s Workshop Feb 21 -23 2007

Overview • Goals of the Hyrax Project • Introduction to customization • Discussion topic OPe. NDAP Developer’s Workshop Feb 21 -23 2007

Goals of the Hyrax Project • • • Integrate THREDDS catalogs with DAP Improve security of our data server Reduce work to maintain the server Reuse the existing server software Integrate work with/for Earth System Grid (ESG) back into the OPe. NDAP software • Improve the server’s performance OPe. NDAP Developer’s Workshop Feb 21 -23 2007

Integrate THREDDS • Requirements: – Standard representation for catalogs – Separation of the logical from the physical data organization – A no-hands mode » Fully automatic catalogs – Graceful transition from custom to automatic catalogs – Full THREDDS features – not stripped down – Must use Unidata’s THREDDS software • Hyrax 1. 0. 2 has met these. OPe. NDAP Developer’s Workshop Feb 21 -23 2007

Improve the Server’s Security • The original data server design was hard to make ‘hack proof’ – Common Gateway Interface (CGI) – Data handlers were run as a subprocess in a protected shell – Main CGI modules written in Perl • As the server grew in complexity, these became more pressing issues. • Improvements – Perl/CGI » Java/Servlet – Subprocess/shell » Specialized IPC OPe. NDAP Developer’s Workshop Feb 21 -23 2007

Reduce Maintenance and Reuse Existing Software • The Perl software was becoming unwieldy – Reimplementation was needed whether in Perl, Java, … • Considerable effort has gone into writing our data handlers for net. CDF, …, and we could not rewrite them. • Additional code for the handlers had to be minimal (for the most complex handler asciival, it’s size increased by 21%, format handlers were just half that value) OPe. NDAP Developer’s Workshop Feb 21 -23 2007

Integrate ESG Work • ESG II had developed a flexible server for the DAP – Its capabilities far out stripped the CGI server – Several extensions to OPe. NDAP software were needed • The changes/extensions to the OPe. NDAP software were merged back into our copies of the code and both projects resumed working from the same version of the software. OPe. NDAP Developer’s Workshop Feb 21 -23 2007

Improve Performance • The new server architecture eliminates basic design problems. – Fewer subprocess created, issues with Perl, … • Response handlers in the BES pass binary objects, not the objects’ network representation • Better use of HTTP (e. g. , persistent connections now supported) OPe. NDAP Developer’s Workshop Feb 21 -23 2007

Recap: Goals of Hyrax • A higher performance server • Provides true THREDDS catalogs • A high level of data-handling code reused • A more secure design • A more maintainable design that can evolve gracefully OPe. NDAP Developer’s Workshop Feb 21 -23 2007

Introduction to Customization • The OPe. NDAP Data Server was designed to be extended. • Hyrax - can be Extended by: – Writing a handlers for the front end – Writing a handlers for the back end – Directly access the back end OPe. NDAP Developer’s Workshop Feb 21 -23 2007

Hyrax Architecture DAP 2 OLFS BES Commands BES THREDDS HTML XML- encapsulated object Java Servlet Engine Unix Daemon Optional THREDDS catalogs OPe. NDAP Developer’s Workshop Feb 21 -23 2007 File system with data files, SQL Database, …

Adding a new BES Handler • Add support for a new data format or data source (relational data base) • Build a gateway to another data system (such as a WCS server) • Support new response types (such as net. CDF 3 files) • Add new commands • Patrick West will talk about this next. OPe. NDAP Developer’s Workshop Feb 21 -23 2007

Direct access to the BES • Use this to build a connection-oriented system • Build new front end systems • And more… • Jose Garcia will talk about this following Patrick. OPe. NDAP Developer’s Workshop Feb 21 -23 2007

Adding a new OLFS Handler • OLFS – Dispatch activity abstracted – Dispatch figures out what they want – Uses a handler to figure out how to do it OPe. NDAP Developer’s Workshop Feb 21 -23 2007

Inside OLFS Dispatch. Handler OPe. NDAP Developer’s Workshop Feb 21 -23 2007

Inside OLFS Dispatch. Handler GET POST Network Interface OPe. NDAP Developer’s Workshop Feb 21 -23 2007

Inside OLFS Dispatch. Handler GET POST Network Interface Uniform Dispatch Interface OPe. NDAP Developer’s Workshop Feb 21 -23 2007

Inside OLFS Dispatch. Handler GET Http. Dispatch. Handler Thredds. Dispatch. Handler POST Network Interface Soap. Dispatch. Handler Uniform Dispatch Interface OPe. NDAP Developer’s Workshop Feb 21 -23 2007

Inside OLFS Dispatch. Handler GET DAS DDS catalog POST SOAP Network Interface Http. Dispatch. Handler Thredds. Dispatch. Handler Soap. Dispatch. Handler Uniform Dispatch Interface OPe. NDAP Developer’s Workshop Feb 21 -23 2007

Inside OLFS Dispatch. Handler GET DAS DDS catalog POST SOAP Network Interface Http. Dispatch. Handler BES Thredds. Dispatch. Handler Crawlable… Soap. Dispatch. Handler BES Uniform Dispatch Interface OPe. NDAP Developer’s Workshop Feb 21 -23 2007 Various sources of information

Summary • BES Handler: Add support for a new data format, response type, commands or to build a gateway • OLFS Handler: Build support for a new network protocol while (re)using the existing software and/or interact with a new data source • Use the BES directly: Build new front–ends (web portals, LAN-based systems) OPe. NDAP Developer’s Workshop Feb 21 -23 2007

Recap: Hyrax Architecture DAP 2 OLFS BES Commands BES THREDDS HTML XML- encapsulated object Java Servlet Engine Unix Daemon Optional THREDDS catalogs OPe. NDAP Developer’s Workshop Feb 21 -23 2007 File system with data files, SQL Database, …

Discussion Topic: How Could Hyrax Support the WCS Protocol? • A BES Handler could be written to support new response formats (i. e. , the well-known binary formats) • A BES command set could be written for operations unique to WCS • An OLFS Dispatch. Handler could be built to use the new BES commands OPe. NDAP Developer’s Workshop Feb 21 -23 2007