ASTROWISE An Astronomical Widefield Imaging System for Europe

ASTRO-WISE An Astronomical Wide-field Imaging System for Europe Konrad Kuijken, Edwin Valentijn Kapteyn Institute, Groningen 10 June 2002 Towards an International VO Garching bei Munchen 1

Astro-Wise Wide field imaging system for Europe VST + Omega. CAM 6 x WFI 1 o x 1 o 16 k x 16 k pixels 10 June 2002 Towards an International VO Garching bei Munchen 2

Astro-Wise Wide field imaging system for Europe The Problem • Many wide-field imaging instruments, vast amounts of data – E. g. : VST = Southern sky (30 min exp, 300 nights/y) in 3 years. Vast amount of data! 100 Tbyte • Science on these data should be archive-based • Handling of the data is non-trivial – Reduction – Calibration and re-calibration – Image comparisons and combinations – Working with source lists – Visualization • ASTRO-WISE aim: – provide a system for maximal data handling flexibility 10 June 2002 Towards an International VO Garching bei Munchen 3

Astro-Wise Wide field imaging system for Europe Concept solution • Environment which – – Has access to all raw and calibration data Can run and modify reduction/calibration pipelines Can run source extraction algorithms Archives reduced data and source lists, or regenerates these dynamically – Can be federated to link different data centers • This dynamical archive continuously grows as more raw data enter the system, and can be used for – `small’ or large science projects – generating and checking calibration data – exchanging methods and scripts • A key functionality is the link back from source data to the original raw pixel data and calibration files 10 June 2002 Towards an International VO Garching bei Munchen 4

Astro-Wise Wide field imaging system for Europe Example uses • Deep multi-colour fields – No need to take all data in one campaign – Combine data of particular quality, assess results – Select sources, visualize interesting ones, … • 1 -in-1, 000 events spurious or not? • Large homogeneous surveys – Eg weak lensing maps, cluster searches, star counts • Variability (source list- or pixel based) – Proper motions (asteroids, nearby stars) – Flux variations 10 June 2002 Towards an International VO Garching bei Munchen 5

Astro-Wise Wide field imaging system for Europe Example uses • Monitor instrument (calibration files) • Planning observations – View quality of existing data – Build on what already exists, add more filters, more exposure time, better seeing, … • All this can be done on the continuously growing archive 10 June 2002 Towards an International VO Garching bei Munchen 6

Astro-Wise Wide field imaging system for Europe Philosophy • System is not geared to a single data product, but to being a flexible tool. • Not focused on making a single, homogeneous survey with static data release(s) - but could be used for this • Optimise interaction between users and their data Dynamic archive • Geared to optical (IR) wide-field image data 10 June 2002 Towards an International VO Garching bei Munchen 7

Astro-Wise Wide field imaging system for Europe Partners • Distribute the load/environment over National centers, each serving their community • Partners: – – – NL - Groningen (+ Leiden) ESO – EIS I – Napoli + F - Terapix D – USM + UK - VISTA Edinburgh/Cambridge • 1. 5 MEuro grant from EU RTD programme 10 June 2002 Towards an International VO Garching bei Munchen 8

Astro-Wise Wide field imaging system for Europe • • • How? Procedurizing data taking at telescope for both science and calibration data Full integration with data reduction Omega. CAM FDR Data model (classes) defined for data reduction and calibration Design View pipeline as an administrative problem 10 June 2002 Towards an International VO Garching bei Munchen 9

Astro-Wise Wide field imaging system for Europe 10 June 2002 Towards an International VO Garching bei Munchen 10

Astro-Wise Wide field imaging system for Europe 10 June 2002 Towards an International VO Garching bei Munchen 11

Astro-Wise Wide field imaging system for Europe Tbyte source lists- brains make the associations Link -lists 10 June 2002 Towards an International VO Garching bei Munchen as fast as possible 12

Astro-Wise Wide field imaging system for Europe Share the load – – – Hardware: Beowulf processors – 32 (most cases) Hardware: multi Terabyte disks (10 – 100) Data reduction: derive calibration Data reduction: run image pipeline (1 Mpx/s) Archive: storage of images (100’s Tbyte) Archive: storage of source parameters (1 -10 Tbyte) Network speed free parameter: - from 5 Mb/s (24 hours/day provides full replication of all Omega. CAM data) - to 200 Mb/s no replication 10 June 2002 Towards an International VO Garching bei Munchen 13

Astro-Wise Wide field imaging system for Europe Federate • Calibration results • Methods (pipelines) for processing calibration • Raw data • Reduced images • Source lists - catalogues 10 June 2002 Towards an International VO Garching bei Munchen 14

Astro-Wise Wide field imaging system for Europe Concepts • Federation maintained by a single db • Full history tracking of all input that went into result providing on-the fly reprocessing – tell me everything tools • Dynamical archive - Context areas in object attributes – Project: eg Cal, Science, Survey, Personal – Owner: eg pipeline, developer, user Partitions – Strategy: Standard, Deep, Freq (monitoring) Link to AVO – Mode: Stare, Jitter, Dither VO table – Time: time stamping • Standards - classes/data model/ procedures - 00 – inheritance/ persistency - Python scripts/ libraries 10 June 2002 Towards an International VO Garching bei Munchen 15

Astro-Wise Wide field imaging system for Europe DB – engine Oracle contract: – reference – licenses - consultancy Oracle 9 i: - federation through “Advanced replication” to evolve into cross site links (=references= pointers) - Python I/F SQL – OCI- Oracle db - Full oo support + Python to db persistency - Terabyte scalability through “partitioned tables” - Administration tools – Interoperability: STREAMS connects to others: Sy. Base, My. SQL 10 June 2002 Towards an International VO Garching bei Munchen 16

Astro-Wise Wide field imaging system for Europe Intra-operability peer to peer • CVS • “Advanced Replication” evolving to pointers 10 June 2002 Towards an International VO Garching bei Munchen 17

Astro-Wise Wide field imaging system for Europe WP s • WP 1 – pipelines / same as ESO-DMD » Implement and operate • WP 2 – visualization tools » Panorapix – fast Gbyte image display + true colour » Graphic cards – Open. GL - • WP 3 – federated db’s » CVS » Oracle • WP 4 – processing power 4 Beowulf centers • WP 5 – direct access 100 Tbyte storage • WP 6 – coordinate – AVO – CERN-HEP- Datagrid ASTRO-GRID 10 June 2002 Towards an International VO Garching bei Munchen 18

Astro-Wise Wide field imaging system for Europe Schedule • ADD – review Q 2 2002 - OK • Basic operations – Q 4 2003 • Evaluate and prepare for mass production 2004 • Qualify and populate 2005 • Deliver survey system – satellites • Links to LOFAR, JIVE, WHIRCAM, GAIA Personal Observatory 10 June 2002 Towards an International VO Garching bei Munchen 19