NPP Atmosphere PEATE Preliminary Design Review Part 2

NPP Atmosphere PEATE Preliminary Design Review Part 2: Science Processing System Scott Mindock Presented by the Atmosphere PEATE Team Space Science and Engineering Center University of Wisconsin-Madison 10 April 2007 Space Science and Engineering Center University of Wisconsin-Madison

Definitions UML - Unified Modeling Language Use Case - Captures requirements at functional level Activity - Describes the steps of a use case Package - Show software structure and dependencies Class - Show software structure and dependencies Space Science and Engineering Center University of Wisconsin-Madison

Sample Activity Diagram Like Flowchart Ovals = Activity Rectangle = Data Action A and B Decoupled Dot = start Circle = end Space Science and Engineering Center University of Wisconsin-Madison

Presentation Format Key Features and Flow Requirements Space Science and Engineering Center University of Wisconsin-Madison

Design Goals Maintainability - System lifetime spans years Reusability - Subsystems and design patterns Testability - Subsystem dependencies are managed Scalability - Design must scale to larger and smaller systems Space Science and Engineering Center University of Wisconsin-Madison

Design Methodology Learn from others - IDPS, Clusters, Ocean PEATE, Disney Parkwide Leverage existing proven technologies - Use successful open source Prototype key features - Demonstration Projects Model System - UML Activity Model, Java System Decouple Major Systems - Manage dependencies, Define interfaces Space Science and Engineering Center University of Wisconsin-Madison

Atmosphere PEATE Science Processing System (APSPS) Space Science and Engineering Center University of Wisconsin-Madison

Software Functional Grouping APSPS Software Subsystems • Ingest • Data Management System • Algorithm Rule Management • Computation Resource Grid Software Development Support • Configuration Management Science Investigation Support • Product Creation Space Science and Engineering Center University of Wisconsin-Madison

APSPS, Atmosphere PEATE Science Processing System DMS: Data Management System CRG: Computational Resource Grid ARM: Algorithm Rule Manage ING : Ingest System Space Science and Engineering Center University of Wisconsin-Madison Stores Data Processes Data Applies Product Rules to Data Brings Data into System

Cloud EDR Assessment Scenario Space Science and Engineering Center University of Wisconsin-Madison

Cloud EDR Assessment Scenario Space Science and Engineering Center University of Wisconsin-Madison

ING Supports pull ingests Supports push ingests Verification functions localized Catalog entry function localized Notices missing or late data Space Science and Engineering Center University of Wisconsin-Madison

ING Requirements Space Science and Engineering Center University of Wisconsin-Madison

APSPS, Atmosphere PEATE Science Processing System DMS - Data Management System CRG - Computational Resource Grid ARM - Algorithm Rule Manager ING - Ingest System Space Science and Engineering Center University of Wisconsin-Madison

Cloud EDR Assessment Scenario Space Science and Engineering Center University of Wisconsin-Madison

DMS Provides well-defined interface DMS is autonomous Provides storage Provides catalog Space Science and Engineering Center University of Wisconsin-Madison

DMS Executive Purges old data Watchdogs Catalog Server Watchdogs Delivery Servers Advertises catalog Space Science and Engineering Center University of Wisconsin-Madison

DMS Executive Requirements Space Science and Engineering Center University of Wisconsin-Madison

DMS Catalog of files in DMS Health of DMS Information stored in database Apache Server Space Science and Engineering Center University of Wisconsin-Madison

DMS Delivery Supports raw access protocols (e. g. FTP, RSYNC, HTTP) Support intelligent access (e. g. Open. DAP, ADDE) Space Science and Engineering Center University of Wisconsin-Madison

APSPS, Atmosphere PEATE Science Processing System DMS - Data Management System CRG - Computational Resource Grid ARM - Algorithm Rule Manager ING - Ingest System Space Science and Engineering Center University of Wisconsin-Madison

Cloud EDR Assessment Scenario Space Science and Engineering Center University of Wisconsin-Madison

CRG Provides well-defined interface Accepts job request Provides Job Status Monitor Job State Scalable Testable Space Science and Engineering Center University of Wisconsin-Madison

CRG Host One or more per node Retrieves data Retrieves algorithm Runs Algorithm Saves results Space Science and Engineering Center University of Wisconsin-Madison

CRG Host Requirements Space Science and Engineering Center University of Wisconsin-Madison

APSPS, Atmosphere PEATE Science Processing System DMS - Data Management System CRG - Computational Resource Grid ARM - Algorithm Rule Manager ING - Ingest System Space Science and Engineering Center University of Wisconsin-Madison

Cloud EDR Assessment Scenario Space Science and Engineering Center University of Wisconsin-Madison

ARM Assigns jobs to CRG Monitors data in DMS Monitors the status of jobs in CRG Applies rules to data Generates notifications Volatile logic lives here Space Science and Engineering Center University of Wisconsin-Madison

ARM Manager Match Rules and Algorithms to available data. Allow users to start processing. Monitor job progress. Implements volatile logic. Space Science and Engineering Center University of Wisconsin-Madison

ARM Manager Requirements Space Science and Engineering Center University of Wisconsin-Madison

ARM Monitors the DMS Finds new data. Finds deleted data. Uses metadata encoded in filenames. Provides scalability. Space Science and Engineering Center University of Wisconsin-Madison

ARM Monitor Requirements Space Science and Engineering Center University of Wisconsin-Madison

APSPS, Atmosphere PEATE Science Processing System DMS - Data Management System CRG - Computational Resource Grid ARM - Algorithm Rule Manager ING - Ingest System Space Science and Engineering Center University of Wisconsin-Madison

Cloud EDR Assessment Scenario Space Science and Engineering Center University of Wisconsin-Madison

Configuration Management Subversion Bugzilla Apache Tomcat SSEC - Document Management Process Space Science and Engineering Center University of Wisconsin-Madison

Configuration Management Provides versioned access to software Provides defect-tracking facilities Used for source code Used for test scripts Used for binaries All open source Space Science and Engineering Center University of Wisconsin-Madison

CM Requirements Space Science and Engineering Center University of Wisconsin-Madison

Lunch Content Space Science and Engineering Center University of Wisconsin-Madison

Product Generation Support Algorithm Ingest - Importing, Developing, Wrapping Algorithm Qualification - Preparing for APSPS LEOCAT - Low Earth Orbit Cloud Algorithm Test bed Space Science and Engineering Center University of Wisconsin-Madison

Algorithm Lifecycle Algorithm can come from anywhere. Once qualified, the algorithm can be applied from ARM. Space Science and Engineering Center University of Wisconsin-Madison

Algorithm Ingest Algorithm entered into subversion Product created in bugzilla Algorithm is ported and wrapped Tests are created Space Science and Engineering Center University of Wisconsin-Madison

Algorithm Ingest Requirements Space Science and Engineering Center University of Wisconsin-Madison

Algorithm Qualification Write a script to execute algorithm Script manages execution environment Algorithm name, inputs and outputs entered into ARM Space Science and Engineering Center University of Wisconsin-Madison

Algorithm Qualification Requirements Space Science and Engineering Center University of Wisconsin-Madison

LEOCAT Reads in all data. Executes algorithm. Writes data. Space Science and Engineering Center University of Wisconsin-Madison

Cloud EDR Assessment Scenario Space Science and Engineering Center University of Wisconsin-Madison