PDS Core Infrastructure Design and Implementation PDS Technical

PDS Core Infrastructure Design and Implementation PDS Technical Session Engineering Node April 6 -7, 2005 http: //pds. jpl. nasa. gov/

Purpose and Method Describe the High-Level System Design and Implementation Plan • Show design approach • Describe software design – Assigning functional requirements to System components • Show components are implemented through successive deliveries of the System – From current System through completion of Core System (2003 -2006) – Trace requirements met with each delivery • Current Status 2

Design Approach • Strive to make the System – • – Modular: built from loosely coupled components – Buy a COTS component or download a free one, rather than building it from scratch – Service based: where each component provides a service, built as independently as possible from other services – Use free and open source software – Design for reuse – Build on work done at JPL, NASA, PDS discipline nodes, and the science community – Open: all services accessible though fully documented APIs – Distributed: able to find data and services from multiple physical locations • These characteristics enable distributed and incremental development Design to minimize development • There are great tools out there – OODT middleware (JPL) – Image Atlas (PDS Imaging Node) – JMARS client (ASU) – DITDOS (PPI) 3

Design Approach (Continued) • Build System in layers – limiting most development costs to top layer PDS Custom Software Middleware: OODT, PDS modules Application Server Container: Tomcat Database: Sybase, My. SQL OS Platform: Commercial, Linux, Java (JVM) Hardware: Network, Storage, and Servers Develop and Integrate Download, Customize, and Integrate Download and Integrate Buy/Download and Integrate Buy and Integrate 4

Software Design: Requirements to Components • All Functional Requirements in the Core System have been assigned to System components • But software design starts at level of architecture – Architecture provides framework for understanding components • Strategy – map architecture to components to Core System functions then to functional requirements – Define the architecture as a set of tiers (layers) – Define the components in each tier – For each Core System function, assign component(s) to implement it – Functional requirements determine how the function is implemented 5

Software Design: The 4 -Tier Architecture Client Tier Web browsers and other netenabled tools, running on the user’s machine Web applications running in a J 2 EE servlet container Service Tier Middleware that transmits and transforms information from the storage tier Storage Tier Information stored in file systems and data bases 6

Storage Tier Components • PDS Repositories – Online disk storage containing PDScompliant “mission archives“ – These are the online data and ancillary files that MRO generates and PDS ingests, distributes, and archives • Catalogs – Databases containing metadata used to locate data products and other system resources – Data product catalogs are typically loaded from index tables delivered with the data – The Data Set Catalog at PDS central node is used to locate data sets and associated resources 7

Storage Tier Components • PDS-MRO repositories and catalogs – Geosciences Node, St. Louis – Atmospheres Node, Las Cruces – NAIF Node, Pasadena – Imaging Node, Pasadena, Flagstaff – Rings Node, Palo Alto – PPI Node, Los Angeles – SBN, College Park • Data replication and data set catalog at JPL, Pasadena PDS Repositories and Catalogs are distributed across the USA 8

Service Tier Components • Product Servers – Provides API for product retrieval – Fetches products from PDS repositories on request – Optionally transforms the product returned (e. g. , packaging it with others in a. zip file or converting it from PDS. img format to. jpg format) • Profile Servers – Provides API for product search – Fetches information matching search criteria from catalogs on request – Information returned as an “XML profile” • Both are part of OODT middleware package, distributed to PDS nodes 9

Web Tier Components • Ingest-Workflow Manager – Executes data ingestion pipeline • Validation Service – Checks mission archives for PDScompliance • Tracking Service – Reports the status of resources submitted to the PDS • Data Set View – Searches for data sets and associated resources (data set search) • Data Set Browsers – Controls the search, display, and retrieval of data products and ancillary files belonging to a specified data set 10

Web Tier Components (Continued) • Query Server (part of OODT) – Searches for any resource or product across the system (global search) • PDS Explorer – Enables direct viewing and downloading from data repositories • Subscription Manager – Enables users to subscribe to be notified when new data (or other resources) become available • Display from Data Set View Volume Packer – Builds PDS-compliant archive volumes from data in repositories • Authentication Server – Determines user access to the System Subscription Manager Sign-Up 11

Client Tier Components • Web Browsers – Netscape 7, Firefox, Internet Explorer 6, Safari, and others • Bulk Download Client – Java client for downloading large collections of products • Server Manager (part of OODT) – Java client for administering distributed OODT components • Anything that can speak HTTP – Matlab, IDL, Python, Java, C, etc. – Web services 12

Client Tier – Examples • Web and Java APIs enable wide variety of custom clients from outside developers – Science domain-specific clients – EPO clients – Repository/Catalog management tools (e. g. , combining Server Manager and PDS Explorer type capabilities) • • Geo. Virgil – Steve Mc. Donald PSI - OLAF Server Manager (OODT) 13

The 4 -Tier Architecture - Summary Client Tier Web Tier Service Tier Storage Tier Ingest-Workflow MGR Validation Service Tracking Service Web Browser Repositories Dataset View Dataset Browsers Query Server Bulk Downloader Profile Servers PDS Explorer Subscription Mgr Volume Packer Server Manager Product Servers Authentication Server Product Catalogs DS Cat Profile Server Data Set Catalog 14

Mapping Components to Functions and Requirements • The next step is to map components to functions and requirements • Component-to-function mapping for search and retrieve 15

Components for Search Client Tier Web Browser Web Tier Service Tier Storage Tier Dataset View Dataset Browsers Query Server Profile Servers Product Catalogs DS Cat Profile Server Data Set Catalog 16

Components for Retrieval Client Tier Web Tier Service Tier Storage Tier Product Servers Repositories Web Browser Dataset Browsers Bulk Downloader PDS Explorer 17

Implementation Plan: 2003 - 2006 • Start with current System as defined by the FY 2003 delivery • Increment the System through yearly software deliveries at end of FY 2004, FY 2005, and FY 2006 System matures with each Development Cycle 18

Key System Capabilities – FY 2003 • Simple distribution – Used to distribute Mars Odyssey data – Supports data set searches for any data set in the PDS Catalog – Supports product searches for data sets that have Data Set Browsers – Supports retrieval of any PDS product in any PDS repository (Most of existing archive loaded at JPL repository) – Bulk downloader for large downloads – Supports browsing repositories with PDS Explorer – Subscription and notification for Odyssey data and PDS docs/software 19

Current Status • Due to budget constraints, the Target System presented includes only Core System requirements – No Extended System requirements are scheduled • All future development work is stopped until directed by MC • Tech Session can recommend that work on selected components should continue 20

Backup Material

Web Services • PDS Core Infrastructure provides – HTTP based APIs – Product and profile retrieval – Not SOAP-based • Lowers the bar for entry • SOAP overly complex; REST a better approach • Requirements – Ability to speak HTTP – For profile manipulation, ability to speak XML • See PDS Client Developer’s Pack – http: //oodt. jpl. nasa. gov/pds-client/ 22

New Product Server • Current OODT uses RMI to retrieve product data – Procedural interface • • Data cannot be streamed Retrieval is by blocks – Slower transfer than HTTP or FTP • REST-style product server – Prototyped as a web application – Discipline nodes run it in an application container (Tomcat) – 100 times faster than RMI-based product server – No server manager • Change invisible to end user clients – Except for speed improvement! 23

Key New Capabilities – FY 2004 • Work on global search requirements – Develop query server and “profile-ofprofile servers” to locate any product in the System – Support more data sets with the Basic Data Set Browser – Complete PDS Explorer by adding display and retrieval capabilities • Begin building ingestion pipeline – Create workflow framework that automates store and catalog functions – Write automated Validation Service for checking PDS labels and index tables (supports store and catalog functions) – Integrate Cassini Tracker into System to provide tracking capability 24

Key New Capabilities – FY 2005 • For Ingestion Pipeline, complete automated Validation Service – Complete Validation Service for all PDS -compliance checking – Enable automated updating of pointing information for Cassini • Continue to round out distribution capabilities – Complete global search capability – Deploy an authentication server for secure access to pre-release data • Work on automated archive writing – Complete specification of archive media for data volume – Write Volume Packer for creating archive volumes from mission archives in a repository 25

Key New Capabilities – FY 2006 • Complete Ingestion Pipeline – Complete ingestion workflow of receipt, validate, store, catalog, track, and notification to submitters and users • Complete automated archive requirements – Test automated archive writing with Odyssey and Cassini archives • Performance and Quality – Definining metrics for retrieval, ingest, and so forth – Identifying bottlenecks, improving software systems 26

Summary • Core PDS Target System presented can handle MRO Era mission archiving • Target System will successfully scale to handle the MRO-era capacity requirements • Removing the automated validation component of the Target System will incur an unacceptable cost • We can meet our budget for building this System 27

What is OODT? • Software framework, initially funded by NASA, for data sharing – – Across heterogeneous, distributed data repositories (location independent) – Across multiple science and engineering disciplines (discipline independent) – With many interconnected nodes (scalable) • Some features of OODT – Reusable software architecture and components (mainly web and service tier), written in Java – Has Java and web services (HTTP) APIs – Has interfaces that enable new components to be plugged in • Number of Mars Odyssey data products distributed through various OODT product servers PDS uses OODT components with customized plug-ins – A PDS-OODT package is installed at every PDS data distribution node 28