PanSTARRS PS 1 Published Science Products Subsystem Presentation

Pan-STARRS PS 1 Published Science Products Subsystem Presentation to the PS 1 Science Council August 1, 2007

What is PSPS? • Responsible for managing the catalogs of digital data • PS 1 PSPS will not receive image files, which are retained by IPP • Three significant PS 1 I/O threads: – Ingest of detections and initial celestial object data from IPP – Ingest of moving object data from MOPS – User queries of detection/object data records

What is PSPS? • Web Based Interface (WBI) – the “link” with the human • Data Retrieval Layer (DRL) – the “gatekeeper” of the data collections • PS 1 data collection managers Human Other S/W Client WBI – Object Data Manager (ODM) – Solar System Data Manager (SSDM) • Provide the connection protocol for other (future/PS 4) data collection managers; e. g. , DRL ODM SSDM IPP MOPS Other DM – – “Postage stamp” cutouts Complete Metadatabase Cumulative sky image server Filtered transient database (and other special clients)

PSPS Components Overview/Terminology • DRL: Data Retrieval Layer – Software clients, not humans, are PDCs – Connects to DMs • PDC: Published Data Client – WBI: Web Based Interface – External PDCs (non. PSPS) • DM: Data Manager (generic) – ODM: Object Data Manager – SSDM: Solar System Data Manager

PSPS Development Status • DRL - a Request For Proposals has been issued to software developers to code the DRL designed by SAIC. This layer includes APIs to connect to the web clients and the databases. • ODM - a cooperative agreement is being developed with Johns Hopkins University’s Department of Physics & Astronomy to develop the ODM, leveraging their experience from the Sloan Digital Sky Survey database work. • SSDM - will be a working clone of the MOPS science client database (and a hot spare for the MOPS system).

PSPS Development Status • WBI - Web clients to access the ODM and SSDM will include those already developed for the MOPS, the “Gator” clone developed at If. A, and a port of the SDSS “Cas. Jobs” client. These will use the new DRL API being developed in the lead item above. • End-to-end testing of the PSPS structure can be accomplished using the DRL, the ported MOPS web client, and a MOPS clone on the backend. This can be done while the ODM is still under development.

The Object Data Manager • The ODM is the major component of the PSPS, both in terms of size and complexity. It’s more than a simple archive. • The ODM will hold & provide user access to: – – Catalogs of all individual focal plane (P 2) detections. Catalogs of detections from all stacked images. Catalogs of all derived objects. Catalogs of high-significance detections in difference images (when they become available). – “Blobs” of low-significance detections from difference images. – Sufficient metadata to allow the user to determine the provenance of any observation.

ODM - Not Your Traditional Astronomical Database! • Unlike SDSS or 2 MASS, we are not waiting until the project is over to generate the database, we’ll publish data as we go! • Data releases? The concept doesn’t apply here! We will probably keep monthly snapshots of the object catalog as the project proceeds. • Our logical data structure will allow the user to track how an object’s properties change as new (better) information is added over time. (It’s possible but not necessarily easy!)

ODM Prototyping Goals The prototyping effort now underway at JHU is intended to demonstrate: – Data ingest (primarily detection to object correlation) – Scalability (physical data schema) - aka partitioning – Publishing (moving data from ingest pipeline to query side storage) in a way that has minimal impact on queries

Prototype ODM Structure

Existing Components (from SDSS) The prototype will utilize the following existing SDSS components: – – Data Loading Pipeline (sql. Loader) Self-extracting Documentation & Diagnostics SQL Query Workbench (Cas. Jobs) Spatial Library (Spherical/HTM)

Functionality Under Development New components for the prototype include: – – Data Transformation Layer (input to loader) Simulated Data (SDSS data & simulated galactic plane) Sample Queries (verify query performance) Cross-Match Functionality (detection-object correlation) – Data Partitioning Procedures (partition across mutimode cluster for parallel data access)

PS 1 Logical Data Schema

PS 1 Data Tables & Sizes

User Interfaces • The DRL authenticates “users” on a per machine basis. • Our initial implementation will be via a secure web server providing access to the following clients: – A port of Cas. Jobs from SDSS to access the ODM – A “Gator” like menu driven tool to access the ODM – Perl tools (developed by MOPS) to access the SSDM • Machine access (PDCs) will be configured to attach to the DRL directly.

System Expansion • Addition of other data collections, e. g. , value added products are accommodated within our PSPS design. – The basic PSPS design provides well-defined APIs to the outside (WBI and PDCs) and inside (databases) – Data collections need not be Relational Database Management Systems (RDMS), but must obey the DRL-DM API – Databases need not all be the same type, e. g. , ODM will use MSSQL and SSDM will be built on My. SQL. • Although not part of the original PSPS design, we can provide intra-database communications (below the DRL) via well-defined mechanisms (e. g. , ODBC, JDBC) to allow queries that cross the data collections hosted by the PSPS. These would be limited to read operations.

Development Schedule • • Award DRL development contract - August 2007 ODM Prototyping through end of September 2007 Critical Design Review - end of October 2007 Hire PSPS Software Engineers (If. A & JHU) - October 2007 • Complete DRL development and perform integration & end-to-end tests using MOPS DB and web interface - April 2008 • Complete integration of the ODM from JHU into the PSPS & full subsystem testing of the system - August 2008