Virtual Observatories Wolfgang Voges MaxPlanckInstitut fr extraterrestrische Physik

Virtual Observatories Wolfgang Voges Max-Planck-Institut für extraterrestrische Physik Garching Workshop ‘‘Astronomie mit Großgeräten‘‘ Am 17. Oktober 2003 in Potsdam Wolfgang Voges 1

Virtual Observatories Overview: Historical roots * What’s happening in the world: IVOA European VO-activities German VO-activities (GAVO) Federation of local data-sets Next-generation search engine Grid Theory in GAVO Outlook * Viewgraphs partly copied from other presentations Workshop ‘‘Astronomie mit Großgeräten‘‘ Am 17. Oktober 2003 in Potsdam Wolfgang Voges 2

Virtual Observatories Historical remarks 1. VO meeting at Caltec in Pasadena (June 2000) 2. Astronomers mostly from the US 3. Very enthusiastic talks 4. Big vision of the future 5. Foundation of the NVO (US) 6. Since then similar meeting in Europe (Garching) 7. Foundation of national European and later 8. Other VOs Wolfgang Voges 3

Data Knowledge ? The exponential growth of data volume (and complexity, quality) driven by the exponential growth in information technology … … But our understanding of the universe increases much more slowly -- Why? F Methodological bottleneck VO is the answer F Human wetware limitations … AI-assisted discovery NGVO? Wolfgang Voges 4

How and Where are Discoveries Made? • Conceptual Discoveries: e. g. , Relativity, QM, Brane World, Inflation … Theoretical, may be inspired by observations • Phenomenological Discoveries: e. g. , Dark Matter, QSOs, GRBs, CMBR, Extrasolar Planets, Obscured Universe … Empirical, inspire theories, can be motivated by them New Technical Capabilities IT/VO Observational Discoveries Phenomenological Discoveries: Pushing along some parameter space axis Making new connections (e. g. , multi- ) Theory (VO) VO useful VO critical! Understanding of complex astrophysical phenomena requires complex, information-rich data (and simulations? ) Wolfgang Voges 5

Why is VO a Good Scientific Prospect? • Technological revolutions as the drivers/enablers of the bursts of scientific growth • Historical examples in astronomy: – 1960’s: the advent of electronics and access to space Quasars, CMBR, x-ray astronomy, pulsars, GRBs, … – 1980’s - 1990’s: computers, digital detectors (CCDs etc. ) Galaxy formation and evolution, extrasolar planets, CMBR fluctuations, dark matter and energy, GRBs, … – 2000’s and beyond: information technology The next golden age of discovery in astronomy? VO is the mechanism to effect this process Wolfgang Voges 6

A Schematic Illustration of the VO-Based Science Primary Data Providers Surveys Observatories Missions Survey and Mission Archives Secondary Data Providers VO Data Services -------Data Mining and Analysis, Target Selection Follow-Up Telescopes and Missions Results Digital libraries Wolfgang Voges VO as an integral part of the whole system … 7

The Changing Style of Observational Astronomy The Old Way: Now: Pointed, heterogeneous Large, homogeneous observations (~ sky surveys (multi MB - GB) -TB, ~ 106 109 sources) Small samples of objects (~ 100 - 103) Archives of pointed observations (~ TB) Wolfgang Voges Future: Multiple, federated sky surveys and archives (~ PB) Virtual Observatory 8

This quantitative change in the information volume and complexity will enable the Science of a Qualitatively Different Nature: • Statistical astronomy done right – Precision cosmology, Galactic structure, stellar astrophysics … – Discovery of significant patterns and multivariate correlations – Poissonian errors unimportant • Systematic exploration of the observable parameter spaces (NB: Energy content = Information content) – Searches for rare or unknown types of objects and phenomena – Low surface brightness universe, the time domain … • Confronting massive numerical simulations with massive data sets Wolfgang Voges 9

Panchromatic Views of the Universe: A More Complete, Less Biased Picture Radio Far-Infrared Visible Dust Map Visible + X-ray Wolfgang Voges Galaxy Density 10 Map

Examples of Possible VO Projects: • A Panchromatic View of AGN and Their Evolution – Cross-matching of surveys, radio to x-ray – Understanding of the selection effects – Obscuration, Type-2 AGN, a complete census Evolution and net energetics, diffuse backgrounds • A Phase-Space Portrait of Our Galaxy – Matching surveys: visible to NIR (stars), FIR to radio (ISM) – A 3 -D picture of stars, gas, and dust, SFR … – Proper motions and gas velocities: a 6 -D phase-space picture Structure, dynamics, and formation of the Galaxy • Galaxy Clusters as Probes of the LSS and its Evolution – Cluster selection using a variety of methods: galaxy overdensity, x-rays, S-Z effect … – Understanding of the selection effects Probing the evolution of the LSS, cosmology Wolfgang Voges 11

Exploration of new domains of the observable parameter space: the Time Domain Faint, Fast Transients (Tyson et al. ) Existing and Forthcoming surveys: Microlensing experiments (OGLE, MACHO …) Solar System patrols, GRB patrols … DPOSS plate overlaps (Mahabal et al. ) QUEST-2 and NEAT at Palomar … and many, many others … The future: LSST ? Wolfgang Voges Megaflares on normal 12 MS stars (DPOSS)

Data Mining in the Image Domain: Can We Discover New Types of Phenomena Using Automated Pattern Recognition? (Every object detection algorithm has its biases and limitations) – Effective parametrization of source morphologies and environments – Multiscale analysis (Also: in the time/lightcurve domain) Wolfgang Voges 13

Exploration of observable parameter spaces and searches for rare or new types of objects Wolfgang Voges 14

Advantages of a Virtual Observatory new, more, better, faster, and easier science comparative analysis of multi-instrument data, permit new approaches to research and multi-wavelength exploration, opening discovery capabilities not otherwise possible This is clearly the primary mandate of all VO efforts minimise redundancy: data collected by a single telescope / instrument can be re-used multiple times by different teams and for different scientific purposes data integrity: data are archived and documented in a controlled and uniform fashion, ensuring long-term scientific usage improving calibrations and creating more higher-level data products to make data more science-ready Wolfgang Voges 15

Advantages of a Virtual Observatory interoperability of archives: - strengthening connections to other archives, catalogues and abstract services for broader research parameter space and links to the literature advancing technologies for computers, networks, data compression, and storage media: - to retrieve and analyse more information more readily at lower cost efficient serving of data to the public: - there will be different levels of end-user from professional astronomers to interested (high-school) students and enthusiastic amateurs – many of whom may undertake projects which are simply unrealisable by large institutes data-mining with new software tools and new catalogues of object properties: - to enable higher-order research based on questions posed in scientific terms Wolfgang Voges 16

Advantages of a Virtual Observatory improving the preparation, development, building of new groundbased and space-based projects improving new observation proposals comparison of real data with simulated data – to provide feedback to new insights, new models, new physics Wolfgang Voges 17

International Virtual Observatory Alliance Korea, Japan, China, Australia, India, Russia, Hungary, Italy, France, Germany, Europe (ESO++), Canada, USA Wolfgang Voges 18

Virtual Observatories What’s happening in the world: IVOA International STANDARDS are needed Registry Data-Models VO-Table VO-Query Uniform Content Descriptors (UCD) Simple Image Access (SID) GRID-standards Tools e. g. data-mining Wolfgang Voges 19

Virtual Observatories European VO-activities In the AVO (euro-vo. org) under the leadership of ESO/ESA the following institutes/groups are collaborating: ESO ESA/STECF University of Edinburgh CDS Strasbourg University Louis Pasteur Centre National de la Reserche Scientifique Delegation Paris The Victoria University of Manchester GAVO (RDS: MPE, AIP, HS, MPA) Wolfgang Voges 20

German Astrophysical Virtual Observatory GAVO-Team: Wolfgang Voges (PI) Hans-Martin Adorf, Gerard Lemson, Achim Bohnet, Joachim Paul Max-Planck-Institut für extraterrestrische Physik, Garching Matthias Steinmetz (Co-I) Harry Enke, Detlef Elstner Astrophysikalisches Institut Potsdam Dieter Reimers, (Co-I) Dieter Engels, Peter Hauschildt Hamburger Sternwarte Simon White, Anthony Banday, Volker Springel Max-Planck-Institut für Astrophysik, Garching Other institutes are most welcome to join >>>www. g-vo. org<<< Wolfgang Voges 21

Why do we need a German AVO? to remain internationally competitive (proposals, data utilisation, quality of science output) to make available VO services to everyone and provide support for the science community and public in Germany to prepare and maintain datasets obtained from German facilities for GAVO and IVO to establish a network, within which the needs of the German science community and public are coordinated to obtain financial support from German agencies for such a national task Wolfgang Voges 22

Activities and responsibilities of partners Wolfgang Voges 23

Activities and responsibilities of partners Main goal is science driven, but it will drive science, too - fast access to all kinds of astronomical and related data - capability to use highly sophisticated software tools for new studies - GAVO will provide interoperability of distributed archives over a high speed network through a set of interface/infrastructure tools - GAVO ultimately will be incorporated into larger IVO federation - Astronomical institutes will require expert data centres of different local character e. g. for providing key data archives, documentations, “simple” analysis-, correlation- and visualisation tools - computer science groups will develop data handling and novel analysis tools and are responsible for their maintenance Wolfgang Voges 24

Activities and responsibilities of partners - university institutes will be able to use GAVO for teaching and will provide a simple gateway to the public and to schools the “service community” will be responsible for designing and developing the interface/infrastructure tools necessary for communication between the users Wolfgang Voges 25

Archive publication through GAVO • • • ROSAT source catalogs published in IVOA compliant manner: – simple cone search – webservices RASS Photons stored in Postgre. SQL database – Spatial index using HEALPix – Cone search, webservices Federation: fast match between ROSAT source catalogues and RASS photons. Published first proposal for unified datamodel to serve as an ontology for the IVOA. Plans: – Extend query capabilities – Publish ROSAT fields and pointed observations – Federate with SDSS mirror at MPA – Federate ROSAT catalogues with external catalogues for classification of X-ray sources (in collaboration with Class. X team). Wolfgang Voges 26

The local GAVO activities Top priority during initial stage of development federation of local key datasets and provision of key applications ROSAT, SDSS, Planck, RAVE development of meta-data standards, especially for simulations development of common query tools for the local archives need ability to query/compare both real sky and simulated data post-processing tools - must be platform-independent installation of visualisation packages existing software provides a strong foundation to allow extension to different types of data and archives Wolfgang Voges 27

The local GAVO activities Technical challenges and requirements archive standards: rules for ingestion, data quality, associated meta -data schema, data attributes archive maintenance/evolution: migration of data with new technology and enhancements in data attributes meta-data requirements/standards for different data-sets (observations, simulation, calibration) federation of archives, interoperability high-speed networking, streaming formats for data distributed processing power – GRID concept seeking active cooperation with industry in many of these areas Wolfgang Voges 28

Next Generation Search Engine • Download Manager – Retrieves data from multiple distributed databases • Matcher – Matches sources based on sky-position (astronomical sources have no unique identifier) • Classifier – Uses multi-wavelength data for identification purposes Wolfgang Voges 29

Next. Gen Search Engine (cntd. ) Wolfgang Voges 30

Download Manager • Features – Tool … • … accesses registry at JHU – User … • … selects distributed catalogues • … specifies one or more sky-locations – Tool … • … queries remote catalogues • … retrieves datasets for further processing Wolfgang Voges 31

Download Manager (cntd. ) Wolfgang Voges 32

Matcher • Essential for data mining • Prototype features • Performs “fuzzy” match between pairs of source lists from different catalogues • Computes probability of real match • Moving matcher into production use – Collaboration with Canadian Virtual Observatory (CVO) – Feeding ROSAT source matches to classifier Wolfgang Voges 33

Matcher (cntd. ) Wolfgang Voges 34

Classification of ROSAT X-ray sources • Class. X: in collaboration with US-VO • Requires data from several large skysurveys • • X-ray: ROSAT (BSC + FSC) Optical: SDSS, USNO B 1. 0 Infrared: 2 MASS Radio: FIRST, NVSS, SUMSS • True multi-wavelength VO-application Wolfgang Voges 35

Correlation of ROSAT and SDSS data Probing the large-scale structure of the universe with clusters of galaxies Project outline: - (ideally on single photon/galaxy basis) (Schuecker, Boehringer, Voges) identify a sample of galaxy clusters using X-ray/optical correlation >>>>>> see next 3 viewgraphs utilise optical multi-colour images (u, g, r, i, z) to derive photometric redshifts quantify completeness and selection limits by comparison to simulated cluster data search for IR correlation and quantify galaxy evolution in clusters determine correlation with radio surveys to identify the frequency of radio galaxies and AGN in clusters, search for radio halos optical correlation to identify AGN in clusters identify correlations with microwave/sub-mm data to search for the Sunyaev-Zeldovich (SZ) effect (distance measurements, velocity determination) Wolfgang Voges 36

Search for clusters of galaxies Maximum likelihood contours based on RASS-3 X-ray photons (upper panel, 1 , 2 . . contours), SDSS galaxies (middle panel, >10 ), and the combined maximum likelihood contours of RASS-3 and SDSS data (lower panel, >10 ). Crosses mark the position of the deepest X-ray clustersamples available sofar (REFLEX-2, X-ray flux limit 1. 8. 10 -12 erg s-1 cm-2). Squares mark the position of the X-ray clusters of the final sample. Wolfgang Voges 37

Search for clusters of galaxies Cumulative X-ray cluster number counts of the RASS/SDSSclusters (histograms) for a log-likelihood minimum of 15 applied to SDSS data (continuous line), for 25 (lower dashed line), and for 5 (upper dashed line). The RASS/SDSS cluster counts are compared with results obtained with other surveys (squares: RDCS, REFLEX-2). No corrections for variations of the angular survey-sensitivity (effective survey area) are applied to the RASS/SDSS and REFLEX-2 data. The figure shows that with the combination of RASS and SDSS data a 10 times deeper X-ray flux limit can be obtained compared to traditional X-ray cluster surveys like REFLEX. Wolfgang Voges 38

Search for clusters of galaxies General remarks: Our first results are quite important as a guideline for future X-ray missions like ROSITA and DUO. For the latter, about 8, 000 X-ray clusters are expected to be detectable with standard methods. The application of the matchedfilter technique allows the extraction of about 30, 000 X-ray clusters with DUO. Such large numbers of Xray clusters are needed for precise tests of the dark energy and alternative gravitational theories. Wolfgang Voges 39

Correlation of ROSAT and SDSS data VO functionality required: - - - federation of relevant datasets including interchange/merging of meta-data identification of candidate cluster members by appropriate query applications to optical and/or X-ray catalogues acquire multi-colour information to determine photometric redshifts identification of candidate radio galaxy cluster members by querying radio catalogues with search criteria (e. g. location) tailored to the derived cluster sample identification of associated SZ by specific queries in existing catalogues; if no candidate SZ cluster can be identified apply suitable search algorithms to Cosmic Microwave Background (CMB) sky surveys to determine effect or limits thereon visualisation of multi-wavelength cluster data deprojection algorithms to allow study of morphology in survey data conversion of simulation data to the space of observable parameters 3 D-interface for visualisation (to schools) Wolfgang Voges 40

Query example Correlation between radio, IR, optical and X-ray sources Search for SDSS QSO´s with 1 < z < 2, which are variable in one of the 4 wavelength-bands - search-engine: which datasets do exist and in which archive? multiple availability? parallel handling on different servers data available for different epochs? comparison of fluxes, light curves, period-search Source catalogues available? - radio: FIRST, NVSS, … IR: IRAS, 2 MASS, … Optical: SDSS, Tycho-2, HST-GSC, USNO-2… X-ray: ROSAT, ASCA, XMM-Newton, Chandra, … Wolfgang Voges 41

Query example - if no catalogue entry exists postage-stamp (pixel-image with/without contour-lines) creation of light curves, fluxes, spectra, etc. by using original-data - high demand of CPU? GRID implementation - search for publications on derived variable SDSS QSO objects Wolfgang Voges 42

Grid Technology for GAVO grid : integration of all GAVO-workstations at MPE and AIP into a cluster Basic services on GAVO-grid: Certification. Authority provides single-sign-on/access-all facility via proxy-ca Resource discovery and runtime information, network-weather for the grid Running distributed applications Running MPI-based applications on the GAVO-cluster Wolfgang Voges 43

Virtual Observatories Theory and GAVO Simulations Comparison of Simulations and observations Wolfgang Voges 44

Simulations in the Virtual Observatory Merging of the Milky Way with the Andromeda galaxy (M 31) (3 Mio particles, cluster of 16 CPU’s, 1 week of CPU time) (30 k particles would need 25 minutes of cluster-CPU time) Wolfgang Voges 45

Simulations in the Virtual Observatory COMA type cluster of galaxies (>1000 galaxies, 10^15 Msolar, 7 Mio particles) 8 CPUs, runtime: 2 days; Gravitation, Hydrodynamics, not included: cooling, star formation (Volker Springel, MPA) Wolfgang Voges 46

The Role of Datasets from Theoretical Astrophysics • Direct Comparisons with Observations – Verification (or not) of Models • Data Mining for Both Observations and Theory New Applications – Buried Physics – • Resource for Education and Outreach Wolfgang Voges 47

Theory and the Virtual Observatory • Size of Datasets Appropriate to VO – Large Scale Simulations, Parameter Space Libraries Imply 10 GB – 10 TB Datasets • Rich Complement to Observational Side • Same/Similar Tools as for Obs. Datasets • Use of VO Infrastructure – Grid Technology, Portals, etc. Wolfgang Voges 48

Conclusions • Theoretical Astrophysics is an Essential Part of the Virtual Observatory Concept Provides Benefits to Theorists – Provides Benefits to Observers – Provides Benefits to Education/Outreach – • Drives New Science Wolfgang Voges 49

GAVO efforts on the TVO • • Published IVOA whitepaper on “Theory in the VO” Leading theory subgroup in IVOA data modeling effort. Chair in IVOA special interest group on theory Plans: – Publish simulation archives at AIP, LMU-Obs. , and. MPA – Collaborate with UPitt on publishing services on theoretical datasets (NSF grant proposal) – Collaboration with Technion Haifa to publish observed and simulated Ly- forest spectra (GIF proposal) – Collaboration in RTN proposal for comparison of simulated and observed X-Ray clusters (Boehringer et al@MPE) Wolfgang Voges 50

Virtual Observatories Outlook Great enthusiasm among astronomical community that VO will work and will make life easier BUT still a lot to be done IVOA is combining all available forces to attack the manyfold problems There is still the need to incorporate other fields like mathematics, informatics, computer science, networks etc. since there is parallel work in progress GRID paradigm, fast data-links, super-computer access, etc. MORE MANPOWER NEEDED Wolfgang Voges 51

The end Wolfgang Voges 52