NOAO Mosaic Pipeline Technical Presentation December 16 2002

NOAO Mosaic Pipeline Technical Presentation December 16, 2002 NOAO Mosaic Pipeline Co. DR 1

Outline of Technical Presentation • • • Introduction Contexts Capabilities Architecture Implementation December 16, 2002 NOAO Mosaic Pipeline Co. DR 2

Presentation Goals Convince you that: • we understand the – – problem requirements resources components • and that the project – is feasible – has a solution for the primary application – has a flexible design for expansion and wider application December 16, 2002 NOAO Mosaic Pipeline Co. DR 3

Guiding Principles • • Modest project Part of Data Products Program (NOAO) Mosaic Imaging Data Dedicated pipeline December 16, 2002 NOAO Mosaic Pipeline Co. DR 4

Principles: Modest Project • Reuse as much software as possible • Keep it simple software December 16, 2002 NOAO Mosaic Pipeline Co. DR 5

Principles: DPP • • MDHS: Mosaic Data Handling System IRAF: Image Reduction and Analysis Facility NSA: NOAO Science Archive DTS: Data Transport System • OPUS: AURA sister institution (STSc. I) • GONG: AURA sister institution (NSO) December 16, 2002 NOAO Mosaic Pipeline Co. DR 6

Principles: (NOAO) Mosaic Data • Use experience of Mosaic Survey Teams • Need to deal with specific peculiarities – Crosstalk, pupil reflections • Allow for high performance per exposure (for real-time telescope context) by capitalizing on the inherent data parallel nature of mosaic imaging data December 16, 2002 NOAO Mosaic Pipeline Co. DR 7

Principles: Dedicated Pipeline • Network of similar computers • No competition with general users December 16, 2002 NOAO Mosaic Pipeline Co. DR 8

What does this project encompass? Algorithms, interfaces, and software for: • • Pipeline infrastructure CCD mosaic data reduction Data quality assessment Image differencing Catalog production Database entry and querying Source merging/classification December 16, 2002 • • Archive ingest and retrieval Alerts Monitoring Data transport High performance computing Parallel computing More … NOAO Mosaic Pipeline Co. DR 9

Contexts In what contexts will the pipeline run? Can we design a pipeline to satisfy multiple contexts? December 16, 2002 NOAO Mosaic Pipeline Co. DR 10

Contexts • NOAO – Telescope/operational context – Archive/NVO context • Community – NOAO Mosaic surveys and observers – Other mosaic instruments December 16, 2002 NOAO Mosaic Pipeline Co. DR 11

Priorities 1. NOAO Archive 2. NOAO Mosaic observers − telescope − downtown − home institution 3. NOAO Mosaic observers at home 4. Community December 16, 2002 NOAO Mosaic Pipeline Co. DR 12

NOAO Contexts • Downtown center fed from telescope • Mountain at telescope • Archive on-the-fly reprocessing December 16, 2002 NOAO Mosaic Pipeline Co. DR 13

Pipeline Locations Pipeline Tucson Archive La Serena Archive Kitt Peak December 16, 2002 Pipeline Cerro Tololo NOAO Mosaic Pipeline Co. DR Pipeline 14

Context: Downtown Pipeline Data Spool and Transport Observer DCA Pipeline User @ telescope, downtown, home December 16, 2002 DSC Archive NOAO Mosaic Pipeline Co. DR 15 DTS

Context: Mountain Pipeline Data Spool and Transport DCA Pipeline Archive User @ telescope December 16, 2002 NOAO Mosaic Pipeline Co. DR 16 DTS

Context: Archive Pipeline User @ home December 16, 2002 Archive NOAO Mosaic Pipeline Co. DR 17 DTS

Context: User Pipeline User @ home December 16, 2002 Pipeline @ home NOAO Mosaic Pipeline Co. DR 18 More

Proposed Context • Downtown pipeline for NOAO archive • Observer may subscribe to data products – At telescope, downtown, home – Images, catalogs, alerts, … • Observer may connect to DQ monitors • Pipeline software available at telescope with minimal support • DQ task/monitors may run at telescope December 16, 2002 NOAO Mosaic Pipeline Co. DR 19

Data Requirements The pipeline design is dependent on the information available about the input data. • Basically we require data with the current NOAO Mosaic readout format that includes: – identification of exposure type (object, etc) – description of regions (data, overscan) – an approximate world coordinate system December 16, 2002 NOAO Mosaic Pipeline Co. DR 21

Data Requirements There may be additional information that the pipeline will use if present. • Associations: type, ID, total and index SEQUENCE = ‘zero 2002 -12 -16 T 043244. 20. 3’ SEQUENCE = ‘dither 2002 -12 -16 T 043244. 5. 2’ If not present heuristics will be used based on a requirement that data enters in time order December 16, 2002 NOAO Mosaic Pipeline Co. DR 22

Capabilities • Major Features and Goals • Data Products – Basic – Advanced • Data Quality Assessment • Instrumental Calibration December 16, 2002 NOAO Mosaic Pipeline Co. DR 27

Capabilities • • • Calibrate mosaic exposures Update instrumental calibrations Identify potential bad data (data quality assessment) Monitor trends and maintain database Stack dither sets Catalog and classify objects and artifacts Get and subtract reference image and detect sources Identify interesting sources Automatically provide data products to subscribers Keep up with observing given sufficient CPU resources December 16, 2002 NOAO Mosaic Pipeline Co. DR 28

Major Features and Goals • • • Data products for NOAO archive and NVO node Data products for observers (by subscription) Pipeline for NOAO and mosaic community Basic CCD mosaic calibrations Advanced time-domain data products Real-time data quality assessment and monitoring High performance, data parallel system LSST testbed Fairly generic pipeline infrastructure (NEWFIRM, …) Automated operation Thorough processing history and data documentation December 16, 2002 NOAO Mosaic Pipeline Co. DR 29

Data Products: Basic • • • Instrument calibrated mosaic exposures Rough photometric zero point Astrometric calibrations Data quality evaluations Updated calibrations Bad pixel, saturated, bleed trail masks Object catalogs Object masks Observing logs Processing information – logs – graphs December 16, 2002 NOAO Mosaic Pipeline Co. DR 30

Data Products: Advanced • • Dither stacks Exposure masks Field Catalogs Difference image detections – Relative to dither stack – Relative to archive or catalog reference • • Light curves Variable object detections Unusual object alerts Moving object trajectories December 16, 2002 NOAO Mosaic Pipeline Co. DR 31

Data Quality Assessment • • • Instrument Telemetry Crosstalk Overscan Bias, flat Noise Focus / Distortions Sky • • • Seeing (PSF) Sky brightness Approx. zero point Twilight Moon up / distance Data quality measures are monitored against preset and user limits as well as adaptive time series limits. Some quantities include mean, sigma, and spatial variations. December 16, 2002 NOAO Mosaic Pipeline Co. DR 32

Instrumental Calibrations • • • Crosstalk [1] CCD defects [2, 4, 5] Saturated pixels [2, 4, 5] Bleed trails [2, 4, 5] Cosmic rays [2, 4, 5] WCS update [3] • • • Overscan [2] Bias [2] Flat field [2] Pupil pattern [3] Fringing [3] Approx. zero point [3] 1. Requires image data from full mosaic (non-parallel) 2. Each image element independent of others (parallel) 3. Global calculation on measurements images (parallel and non-parallel) 4. Interpolate in data 5. Flag in mask December 16, 2002 NOAO Mosaic Pipeline Co. DR 33

Instrumental Calibrations Two-pass calibration for telescope context: 1. Nighttime pass for immediate and nearly complete calibrated exposures 2. Daytime pass for calibration update from the full night’s data set December 16, 2002 NOAO Mosaic Pipeline Co. DR 34

Nighttime Pass • Perform standard CCD calibrations: – Use afternoon master bias – Use most recent flat field • Apply pupil and fringe correction – Use most recent pupil and fringe templates • Apply global coordinate calibration December 16, 2002 NOAO Mosaic Pipeline Co. DR 35

Daytime Pass • Determine if night’s data is suitable for deriving updates to library calibrations • Derive new pupil, fringe, and sky flat calibrations • Evaluate changes and significance of new calibrations • Update library calibrations for next night • Update night’s exposures with new calibrations • Combine afternoon biases into new master bias • Combine afternoon dome flats if no library flat December 16, 2002 NOAO Mosaic Pipeline Co. DR 36

Other Contexts • For archive data will either already have best calibration from library or will be derived by requesting raw data for night • At home or in the community raw data will be queued as at telescope • Documentation and support (data ingest applications) will be provided December 16, 2002 NOAO Mosaic Pipeline Co. DR 37

Data Products Subscription • Capability of the DPP system – Not necessarily specific to the pipeline but requires interfacing with DTS • Allows external software to request notification of new data products • Allows flexibility and broader access – Has implications for the pipeline context December 16, 2002 NOAO Mosaic Pipeline Co. DR 38

Architecture • What is a pipeline? • Mosaic Pipeline Architecture Concept • Pipeline Components – Controls and Monitors – Modules – Calibrations and Database (Rafael Hiriart) – Archive (Robyn Allsman) December 16, 2002 NOAO Mosaic Pipeline Co. DR 39

What is a Pipeline? System to transform input data to output data • • • Automated Composed of processing steps (modules) Steps connected by rules (triggers) Provides monitoring and alerts Error tolerant (continue with next input data) December 16, 2002 NOAO Mosaic Pipeline Co. DR 40

Mosaic Pipeline Architecture Concept • Multiple CPUs but no dependency on N • Multiple types of sub-pipelines by function – One for operations over all mosaic elements – One for operations on individual elements – One for cataloging – One for image differencing • All types on all CPUs: no master! • Sub-pipelines triggered by files December 16, 2002 NOAO Mosaic Pipeline Co. DR 41

Mosaic Pipeline Architecture Concept • All CPUs with identical pipeline software, possibly on common NFS disk • Assign work by minimum data backlog • Transfer data to local CPU disk: not NFS! – Optimize by modules writing to next trigger directory • Controls connected to operator console • Monitors viewed via network by multiple parties December 16, 2002 NOAO Mosaic Pipeline Co. DR 42

Network of Sub-pipelines and CPUs Pipeline CPU CPU MEF MEF SIF SIF CPU CPU SIF SIF MEF MEF: pipeline for operations over all mosaic extensions; eg crosstalk, global WCS correction SIF: pipeline for single CCD images; eg ccdproc, masking December 16, 2002 NOAO Mosaic Pipeline Co. DR 43

Data Flow Concept Last module in one pipeline writes output directly to the data directories of the host for next pipeline, with the host selected by having the minimum number of waiting data files. December 16, 2002 NOAO Mosaic Pipeline Co. DR 44

Data Flow Algorithm • Search list of potential hosts: – Check if host is up – Check number of trigger files – Assign output filename to data directory of host with least number of data files – Network filenames are used: (eg. host!directory/filename • Module runs and writes output files December 16, 2002 NOAO Mosaic Pipeline Co. DR 45

Data Flow Networking • Use a daemon automatically spawned the first time data is transferred to a host • Daemon provides portability across platforms; eg. Unix and VMS December 16, 2002 NOAO Mosaic Pipeline Co. DR 46

Data Flow Networking: Example • Crosstalk input is Obj 123. fits with 2 extensions • Output names are generated from Host. dat: – Host 1 has two waiting files, Host 2 has one, Host 3 is down, Host 4 has none – Host 2!Obj 123. 1, Host 4!Obj 123. 2 • Crosstalk module runs and writes output files directly to the hosts • There are no extra network copy or splitting steps December 16, 2002 NOAO Mosaic Pipeline Co. DR 47

Data Flow Networking: Example Host 1: Obj 456. 1 Obj 321. 2 Host 2!Obj 123. 2 Obj 123 Host 0: Crosstalk Host 2: Obj 567. 2 Obj 123. 2 Host 4: DOWN Host 3!Obj 123. 1 December 16, 2002 Host 3: Obj 123. 1 NOAO Mosaic Pipeline Co. DR 48

Pipeline Components Controls & Monitors Data Source (DTS, user) Pipeline raw data Module products Data Sink (DTS, user) Calibrations & Databases December 16, 2002 NOAO Mosaic Pipeline Co. DR 49

Pipeline Modules Pipeline December 16, 2002 Module CLSH CSH API NOAO Mosaic Pipeline Co. DR 50

Data Parallel Modules Some algorithms may need to be (re-)implemented specifically for a data parallel pipeline. One type is where measurements are made across the mosaic for a global calibration. Rather than requiring all pieces to be in one pipeline arrange for measurements made in parallel to be collected for the global calibration and then apply the global calibration to the pieces in parallel. December 16, 2002 NOAO Mosaic Pipeline Co. DR 51

Data Parallel Modules WCS Example • Catalog objects in each CCD in parallel • Bring catalogs (not images) together – Only need x/y coordinates of brighter stars • Match sources to ref. catalog (eg. USNO) • Compute global correction (shift, scale, etc. ) • Return correction coefficients to parallel pipelines to be applied to each CCD • Cataloging and correction stages can be separated and run asynchronously with other stages December 16, 2002 NOAO Mosaic Pipeline Co. DR 52

Data Parallel Modules Fringe/Pupil Example • Determine best global scaling of pupil and fringe templates to each exposure and then subtract scaled template – Compute statistics over each CCD in parallel – Combine statistics to get global scale factor – Subtract template with global scale from each CCD in parallel December 16, 2002 NOAO Mosaic Pipeline Co. DR 53

Pipeline Triggers • • • Files: Flags: Timers: File contents: Messages: Resources: trigger on appearance of files trigger on particular set of flags trigger at times or intervals trigger on keywords, etc trigger on messages trigger on resources May be more but one type can mimic others December 16, 2002 NOAO Mosaic Pipeline Co. DR 54

Pipeline Triggers • File triggers useful for initiating a pipeline • Flag triggers useful within a pipeline to communicate success of previous steps • Flag triggers also useful for waiting for completion of parallel steps • Timer triggers useful in telescope pipeline for performing different daytime/nighttime steps December 16, 2002 NOAO Mosaic Pipeline Co. DR 55

File Triggers Pipeline Data Directory obj 123. fits Trigger Directory obj 123. trig Module GO May contain information such as output path December 16, 2002 NOAO Mosaic Pipeline Co. DR 56 More

Flag Triggers and Merging Pipeline December 16, 2002 Data Directory Trigger Directory obj 123 b. fits obj 123 d. fits obj 123 a. fits obj 123 c. fits obj 123 b. trig obj 123 d. trig obj 123 a. trig obj 123 c. trig Module GO NOAO Mosaic Pipeline Co. DR -d -d Module GO 57

Timer Triggers and Two-Passes • Nighttime pipeline runs and leaves data in starting directory for daytime pipeline • Daytime pipeline is triggered at end of night by timer December 16, 2002 NOAO Mosaic Pipeline Co. DR 58

Controls & Monitors Process Manager Status Monitor Obs. Manager Keyword Monitor Pipeline Module To Database December 16, 2002 NOAO Mosaic Pipeline Co. DR 59

Data Manager • Interacts with the pipeline, operator, and potentially other parts of the system such as archives or external applications • Record – New calibrations from pipeline or operator – New parameters from operator – Processing information from pipeline • Responds to queries for – Calibrations – Parameters – Processing history – Documentation and reports for data products December 16, 2002 NOAO Mosaic Pipeline Co. DR 61

Data Manager Architecture December 16, 2002 NOAO Mosaic Pipeline Co. DR 62

What do we want to store in the database? December 16, 2002 NOAO Mosaic Pipeline Co. DR 63

Where is Data Manager? December 16, 2002 NOAO Mosaic Pipeline Co. DR 64

Calibrations • The Data Manager responds to requests from pipeline for current calibration for a particular date, filter, etc. • Updates calibrations produced by pipeline (or externally) for a particular date, filter, etc. Calibration updates may require operator confirmation. • Calibrations include – – Biases and flat fields Pupil and fringe templates Standard star data Astrometry coordinates • Some queries are satisfied through secondary queries to other databases such as USNO, GSC 2, Landolt, etc. December 16, 2002 NOAO Mosaic Pipeline Co. DR 65

Parameters • Responds to requests from pipeline for current parameters for – Pipeline module – Observation date, filter, exposure type, etc. – Position on sky • Updates parameters supplied by operator December 16, 2002 NOAO Mosaic Pipeline Co. DR 66

Processing Information • All information produced by the pipeline is recorded (keyed by a data identifier). This includes all the information provided to the keyword monitor as well as other data processing sources (logs, graphics, etc. ) • Pipeline requests processing information for a pipeline execution packaged as an associated data product for the archive. • The operator can query processing information for diagnostic purposes. December 16, 2002 NOAO Mosaic Pipeline Co. DR 67

Reports • Produces reports for a particular data product • Documentation is created from processing information according to some template and desired format (eg xml, html) December 16, 2002 NOAO Mosaic Pipeline Co. DR 68

Pipeline/Archive Ingest Interface • Desirable traits – Independence of database semantics – Use of self-describing data description standards – Hiding data’s physical location December 16, 2002 NOAO Mosaic Pipeline Co. DR 69

Archive Ingest Who, What, Where Authority Payload Data Receiving Agent Archive Ingest Manager Data Store December 16, 2002 NOAO Mosaic Pipeline Co. DR 70

Strawman Implementation December 16, 2002 NOAO Mosaic Pipeline Co. DR 71

Things We Looked At / Aware Of • • • Macho pipeline SM/SN pipeline Sloan pipeline Pan-Starrs: IMCAT, Vista IRAF: Core, IMRED pipelines, STSDAS, PYRAF, etc MIDAS: Mosaic Imager Data Archive System Linda and descendants Elixar (CFH), Terapix (CFH), Subaru, ESO WFI, INT WFI Condor / PVM / NOAO message bus Opus pipelines: HST. MSSO, GONG Databases: My. SQL, Postgres December 16, 2002 NOAO Mosaic Pipeline Co. DR 72

Software and Systems (Blue Ribbon) • OPUS • IRAF System – CLSH (enhanced), KI, OBM/GUI • IRAF Tasks – MSCRED, ACE • • SM/SN Alard/Lupton Algorithm POSTGRES DTS NSA December 16, 2002 NOAO Mosaic Pipeline Co. DR 73

Software and Systems (Honorable Mention) • PVM • Condor • Other scripting languages and systems – PYRAF and Python – Perl – MLCL December 16, 2002 NOAO Mosaic Pipeline Co. DR 74

Pipeline Modules Pipeline Module CLSH CSH OAPI MSCRED, etc December 16, 2002 NOAO Mosaic Pipeline Co. DR 75

Controls & Monitors Process Manager Status Monitor Opus IRAF GUI Obs. Manager Keyword Monitor Opus IRAF GUI Pipeline Module Opus December 16, 2002 NOAO Mosaic Pipeline Co. DR 76

Switchboard Server Keyword Monitor Status Monitor Database Manager Other types Or instances Switchboard address set by environment variable CPU Pipeline Module Switchboard Server Module Pipeline Module Backup December 16, 2002 NOAO Mosaic Pipeline Co. DR 77

Triggers OPUS provides: • Files: trigger on appearance of files – Data entry pipeline initiation • Flags: trigger on “blackboard” flags – Internal sequencing of modules – Parallel to Global sequencing • Timers: trigger at certain times or intervals – Nighttime/Daytime Two-Pass Control December 16, 2002 NOAO Mosaic Pipeline Co. DR 78

Monitoring IRAF Tasks • IRAF tasks, including scripts, will open a messaging connection and write status and monitor information • Minimal changes will be required to tasks • If a server is not running or disappears the tasks will continue to run with output spooled locally December 16, 2002 NOAO Mosaic Pipeline Co. DR 79

Monitoring IRAF Tasks • Initially the broadcasting will be a socket connection with a server that multiple clients may connect to for rebroadcast • The monitor tasks are IRAF GUI tasks which provide flexibility for changes to the GUI or functionality December 16, 2002 NOAO Mosaic Pipeline Co. DR 80

Monitoring IRAF Tasks • The GUI monitors will include: – Adaptive alarms – Adaptive heartbeat monitoring – Advanced graphics December 16, 2002 NOAO Mosaic Pipeline Co. DR 81

IRAF Keyword Monitor Prototype December 16, 2002 NOAO Mosaic Pipeline Co. DR 82

IRAF Keyword Monitor Prototype December 16, 2002 NOAO Mosaic Pipeline Co. DR 83

IRAF Status Monitor Prototype December 16, 2002 NOAO Mosaic Pipeline Co. DR 84

NOAO Mosaic Pipeline Development Plan 1. Basic Calibration Pipeline 2. Advanced Time-Domain Pipeline December 16, 2002 NOAO Mosaic Pipeline Co. DR 85

1. Basic Calibration Pipeline • • • Basic single exposure calibrations Data quality assessment and monitoring High-performance pipeline infrastructure Simple data transport system Connection to the NOAO Science Archive December 16, 2002 NOAO Mosaic Pipeline Co. DR 86

2. Advanced Time-Domain Pipeline • • • Catalogs Image difference detections Multiple detection ident. and merging Time series Alerts Archiving of new data products December 16, 2002 NOAO Mosaic Pipeline Co. DR 87

Timeline Targets • Test version of basic calibration pipeline – July 2003 • Operational – September 2003 • Test version of time-domain pipeline – July 2004 • Operational – September 2004 December 16, 2002 NOAO Mosaic Pipeline Co. DR 88

Work Breakdown • • • Pipeline Monitors Data Manager Input and Output Data Products Archive December 16, 2002 NOAO Mosaic Pipeline Co. DR 89

Work Breakdown • Pipeline – Define methods for running IRAF tasks in OPUS • Parameters • Error handling • I/O – Define and verify data flow balancing method – Define, develop, and implement DQ methods – Develop data parallel algorithm steps for • WCS • Fringe/pupil removal – Develop data parallel OPUS architecture – Setup development system of at least two machines December 16, 2002 NOAO Mosaic Pipeline Co. DR 90

Work Breakdown • Monitors – Develop status monitor • Experiment with different GUI formats – Develop keyword monitor • Experiment with different GUI formats – Develop switchboard server December 16, 2002 NOAO Mosaic Pipeline Co. DR 91

Work Breakdown • Data Manager – Define interfaces • • – – – Pipeline DBMS NVO/web services External clients Define database structures Define archive data products Design processing reports Design calibration library storage and methods Design and implement manager application • Include GUI monitor and operator interface – Install and configure DBMS December 16, 2002 NOAO Mosaic Pipeline Co. DR 92

Work Breakdown • Input and Output Services – Contribute to DTS – Implement interim data transport, staging, and queuing December 16, 2002 NOAO Mosaic Pipeline Co. DR 93

Work Breakdown • Archive – Contribute to NSA development of automatic ingest – Adjust data product specification to include NSA requirements December 16, 2002 NOAO Mosaic Pipeline Co. DR 94

Work Breakdown • Specify Data Products – File types – Headers – Documentation December 16, 2002 NOAO Mosaic Pipeline Co. DR 95

Implementation Plan • It is important to deliver core functionality quickly • Some technologies are new (to the development team) • Delivery timeframe is short December 16, 2002 NOAO Mosaic Pipeline Co. DR 96

Management Plan Key elements of the management plan are: • • Management/staffing Work Breakdown [covered by FV] Schedule Risk Management December 16, 2002 NOAO Mosaic Pipeline Co. DR 97

Personnel Staff Member Role Alloca tion Dick Shaw Project Manager 5% Schedule development, resource planning Frank Valdes Team Lead 40% Allocation of work, tracking technical progress, lead designer, documentation Chris Smith Project Scientist 10% Definition of requirements, use cases, verification & validation, documentation Rafael Hiriart S/W Engineer 25% Database & infrastructure design & development, use case development Robyn Allsman S/W Sys. Eng. 5% Archive interface definition, archive system updates, data storage planning, consultant F. Pierfederici Scientific Progr. 30% Implementation, testing TBD (U. MD) S/W Engineer 50% Implementation, testing December 16, 2002 Responsibilities NOAO Mosaic Pipeline Co. DR 98

Staffing Profile December 16, 2002 NOAO Mosaic Pipeline Co. DR 99

December 16, 2002 NOAO Mosaic Pipeline Co. DR 100

Risk Management • Heavily matrixed staff – New staff will also off-load other work from team lead • Staff distributed across continents & institutions – Project leadership remains in Tucson – Extended visits by new remote staff – Weekly videoconferences • New staff has limited experience in problem domain – Project leader to work closely with new staff • Use of new/third-party software – Make effective use of expertise from external partners December 16, 2002 NOAO Mosaic Pipeline Co. DR 101