EuroVO DCA WP 4 and GAVO and a

Euro-VO DCA WP 4 and GAVO and (a little) Astro. Grid-D Gerard Lemson Euro-VO DCA WP 5 kick-off meeting Trieste, 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

Overview • Euro-VO DCA WP 4 – – – Goal and motivation Theory in VO Theory in IVOA Theory in Euro-VO DCA: WP 4 and WP 5: theory WP 5 and/or grid in WP 4 ? • GAVO (& Astro. Grid-D) – – – GAVO II Theory in GAVO Grid in GAVO Astro. Grid-D (GA)VO and Astro. Grid-D 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

Euro-VO DCA WP 4 • Objectives – This task aims at assessing the inclusion of theory data and services in the Virtual Observatory • Description of work – The task is tackled by the Theory Expert Group, which has yearly meetings and email discussions to establish a framework for the inclusion of Theoretical Astronomy Services in the VObs. The TEG organizes a Workshop. • Deliverables – WP 4 -D 10: Euro-VO Theory Workshop – WP 4 -D 11: TEG Report: Framework for inclusion of Theoretical Astronomy Services in the VObs • Expected results – The final report will describe a framework for inclusion of theoretical data and services in the VObs – Motivation of the European Theoretical astronomy community to participate in the VObs endeavour. 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

Reminder • Broad goal of VObs – Make results of astronomical research, data and applications, more readily available to larger community, and create value-adding services. • Facilitate results– – – communication checking (re)use comparison combination • Organised in IVOA 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

The observational VObs • Most VO efforts concentrate on observational data sets – simple observables: photons detected at a certain time from a certain area on the sky – long history of archiving – pre-existing standards (FITS) – valuable over long time (digitising 80 yr old plates) – long history of comparing different observations: interoperability • Standards observationally biased – common sky: cone search, SIAP, region – common objects: XMatch – data models: characterisation of sky/time/energy (/no polarisation yet) 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

The same object Radio Optical X-Ray John Hibbard http: //www. cv. nrao. edu/~jhibbard/n 4038. html NASA/CXC/SAO/G. Fabbiano et al. 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

Theory in VO • Apply same ideas to theoretical resources (datasets and services). • Plenty of resources available online. • Not standardised: – 2004, whitepaper detailing issues related to standardisation of theory 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

Theory in the VO: issues • Simulations not so simple – complex observables – no standardisation (not even HDF 5) – archiving ad hoc, for local use • Moore’s law makes useful lifetime relatively short: few years later can do better 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

“Moore’s law” for N-body simulations 2006 -11 -30, Courtesy Simon. Triest White Euro-VO DCA WP 5 kick off

Standardisation issues • Current IVOA standards somewhat irrelevant – assume common sky – common objects – comparison direct • In theory – no common sky – no common objects – comparison statistical • requires new data models – content, physics, code 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

Not the same object Toomre & Toomre, 1972 Courtesy Volker Springel 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off Di Matteo, Springel and Hernquist, 2005

So why bother ? • Simulations are interesting: – For many cases only way to see processes in action – Complex observations require sophisticated models for interpretation • Bridging gap in specialisations: not everyone has required expertise to create simulations, though they can analyse them. • Many use cases do not require the latest/greatest – exposure time calculator – survey design 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

Theory in the IVOA • IVOA Interest group on theory – founded May 2004 – “horizontal” – use cases • Current projects: – Theory UCDs/Semantics (Laurie Shaw) – SNAP (Claudio Gheller) – Simulation (meta-) data model (GL) • Meeting on SNAP in Garching, early 2007 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

Euro-VO DCA WP 4 (repeat) • Objectives – This task aims at assessing the inclusion of theory data and services in the Virtual Observatory • Description of work – The task is tackled by the Theory Expert Group, which has yearly meetings and email discussions to establish a framework for the inclusion of Theoretical Astronomy Services in the VObs. The TEG organizes a Workshop. • Deliverables – WP 4 -D 10: Euro-VO Theory Workshop – WP 4 -D 11: TEG Report: Framework for inclusion of Theoretical Astronomy Services in the VObs • Expected results – The final report will describe a framework for inclusion of theoretical data and services in the VObs – Motivation of the European Theoretical astronomy community to participate in the VObs endeavour. 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

WP 4: tasks • Procurement WP 4 personnel: • Draft whitepaper “Framework for inclusion of Theory data and services in the VObs” (End cycle 1) • EURO-VO Theory Workshop (Deliverable 10, Month 17) • Whitepaper ready (Deliverable 11, month 26) • Participation IVOA interop meetings with theory 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

WP 4 first steps • Formation Theoretical astronomy Experts Group (TEG) • • • CNRS ESA ESO INAF INTA MPG NOVA LU Hervé WOZNIAK Pedro OSUNA None Santi CASSISI Miguel CERVIÑO Gerard LEMSON (Chair) TBD (Joop SCHAYE) TBD (? ) Census theory data centers (with WP 2) Cooperation in IVOA theory interest group activities Coordination with WP 5 (which is why I’m here) 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

WP 4 and WP 5: some first ideas • Grid support for WP 4 – Grid must be addressed in WP 4 whitepaper – Grid can facilitate/is required for implementations of prototype for standards (see below) • Theory use cases/requirements for WP 5. – Asynchronous, queued querying of large simulation database • OGSA-DAI? (GAVO+Astro. Grid-D) – SNAP on Millennium raw data • grid services wrapping custom data access modules (GAVO+Astro. Grid-D) – Scientifically useful virtual telescopes – Parameter studies using “small” simulations – Large simulations ? ? 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

Example: Virtual Telescope Observed Simulated Thanks to Volker Springel Thanks to Alexis Finoguenov, Ulrich Briel, Peter Schuecker, MPE) 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

Virtual telescope configuration 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

Virtual telescope configuration • Components: – simulation of source, wrapped with “photon producer” service – spec and implementation for “pre-observation image” (POI) – “foreground” accepting, reemitting POI • ISM/IGM • lenses – “software telescope”, simulating realistic observations – comparison software • Required (VO/Grid) infrastructure supporting this workflow – – – registry for publishing/discovering components computational resources scheduler for executing job streaming support for flowing POI between components storage for results workflow engine? 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

Discussion Suggestions very welcome 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

Intermezzo 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

GAVO and Astro. Grid-D 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

GAVO and Astro. Grid-D • GAVO I – – Funded by German Ministry of Dec 2002 – March 2005 3 months extension 3 FTEs • Astro. Grid-D – 1 of 5 funded “communities” taking part in D-Grid project – O(13) FTEs – Autumn 2005 – 2007/8 • GAVO II – Delayed funding, due to need to distinguish from Astro. Grid-D – June 2006 -July 2008 – 5 FTEs (so far 3, ARI (>~1. 5), MPE (1. 5), TUM (. 5), AIP (<~. 5), Tübingen (1)) 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

GAVO (I + II) projects • Archives: – ROSAT All Sky Survey • cone search: photons, sources • SIAP: fields – ROSAT pointed observations: SIAP • • standardised approach to publish FITS images under SIAP uses FITS metadata repository (SQLServer) SQL based mapping and SIAP implementation easily generalised to other datasets and protocols (SSA) – RAVE – CDFS optical follow up spectroscopy: SSA – Heidelberg catalogues and GAVO archive: SCS/Sky. Node • Data mining – Cross-matcher (also on Grid, TUM) – SED classifier – Cluster finder 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

GAVO (I + II) projects ctd. • Grid – evaluation Globus – Gadget 2 on Globus MPI (AIP + MPE) – cluster finder on the grid • VOQL/Sky. Node – VOQL tiger team (Benjamin Gufler) – Hi. Sbase peer-to-peer approach to multi catalogue crossmatching – Mock catalogues for testing • Theory – IVOA: • • Initialised Theory Interest Group (chair) SNAP Data models Semantics – Millennium database – (proto-) Virtual telescopes (planck, Mo. Ma. F 2, Dolag et al) 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

Astro. Grid-D • Part of D-Grid • Community project • Provide – use cases – middleware – adaptation of existing applications/data to the (D-)Grid 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

Astro. Grid-D partner projects (incomplete) • Astrophysikalisches Institut Potsdam – project management – robotic telescopes on the grid – galaxy merger simulations • MPE: – – cluster finder catalogue cross-matcher (with TUM) adapt simulation visualisation to the Grid access to persistent data (in database) • MPA – Planck Process Coordinator on the Grid • Technischer Universitat München-Informatik – Star. Globe: streaming data querying/combination 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

Astro. Grid-D partner projects, ctd. • Zuse Institut Berlin – Grid Application Toolkit – Grid. Sphere web portal – Information Services (“registry”, RDF+Spar. QL) • Albert Einstein Institute, Berlin – Cactus tool kit – Grid based simulation monitoring • Astronomisches Rechen Zentrum, Heidelberg – Stellar dynamics: Grape on the Grid – Gaia. Grid – Planck Pro. C collaboration 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

GAVO + Astro. Grid-D, synergies • Groups involved in both projects: AIP, ARI, MPE, TUM • VO offers different kind of use cases: – VObs aims to enable external users to use resources – Heterogeneous world: Interoperate with resources outside of Astro. Grid-D (but within IVOA standards) • Grid can help implement VO services 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

GAVO + Astro. Grid-D ctd. • Planned (from GAVO point of view) – Cluster finder (MPE) – SNAP implementation (MPA) – OGSA-DAI (GSM? , DSA? , CAS Jobs) access to Millennium DB (MPA) – Real virtual telescopes (MPA+. . . ) – Cross matching (MPE+TUM) • Possibly – VOEvent and Robotic Telesopes ? – Astro. Grid-D Information Services as IVOA Registry 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

SNAP using grid services • Simple Numerical Access Protocol – – Project of IVOA theory interest group Based on simulation data model (input welcome !) Query interesting simulations Select and retrieve subset • Implementation: SNAP on Millennium snapshots – Custom access code deployed near large datasets – Authenticated access – Standardised publication for use in Grid workflows (Pro. C? Streaming? ) – Data staging and movement 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

Globus container SNAP web service Grid Service get. Box() C-module Information Service IS aware Workflow Engine 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off Millennium snaphots

Millennium website and OGSA-DAI • Dataset – postprocessing products of Millennium simulation (http: //www. mpagarching. pg. de/millennum) – halo/galaxy catalogues – density fields (environment queries) • Functions – support generic SQL queries on large (O(5 Tb)), complex database – support mydb – some graphics script based access 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

Implementation • Microsoft SQLServer database (Sky. Server inspired) – data model for efficient retrieval merger tree histories – spatial indexing • TOMCAT web server – http: //www. g-vo. org/My. Millennium – authentication using RDB – wget based access, IDL+R modules • Graphics – VOPlot applet (small results, browser) – TOPCAT (plugin) • Synchronous queries – JDBC – 7 min time limit • Issues – – I/O bottle necks when > 4 users query same table short maximum query time no asynchronous access simplistic user management (compared say to CAS Jobs) 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

Why Grid interesting? • Need – Asynchronous access • Queuing • Improve result management, data delivery – Bulk uploading into My. DB – More generic authentication (X. 509) • Will try CAS jobs (SDSS/JHU) • Investigate OGSA-DAI – Map portal to OGSA-DAI functionality – Monitoring ? Logging ? Performance ? • Will try Astro. Grid-UK’s DSA • Other suggestions ? 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

2006 -11 -30, Triest Euro-VO DCA WP 5 kick off

Thank you ! 2006 -11 -30, Triest Euro-VO DCA WP 5 kick off