ESA Gaia Archive Architecture key points A Mora

ESA Gaia Archive: Architecture & key points A. Mora, J. Gonzalez-Núñez, J. Salgado, R. Gutiérrez-Sánchez, J. C. Segovia, J. Duran, B. Merín, C. Arviset ESA-ESAC Gaia SOC and ESDC Gaia Data Workshop Heidelberg 2016 -11 -23 Issue/Revision: 1. 0 Reference: Presentation Reference Status: Issued ESA UNCLASSIFIED - Releasable to the Public

ESA Gaia archive: key points Ø Code to the data (workflows) Ø Gaia Archive contents ● Catalogues (tables) ● Pre-computed cross-match (pivot tables) ● Light curves (one entry per point) Ø Gaia Archive funtionality ● ADQL: SQL dialect for queries reproducibility papers ● TAP+ VO standard: web data query, user tables, sharing, … ● SAMP VO standard: interoperability 3 rd party software ● DR 2+ evolution: ~PB size, parameters, spectra, epoch data, … Ø Learning curve: reproducing DR 1 Brown+ 2016 plots Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 2 ESA UNCLASSIFIED - Releasable to the Public

Code to the data (workflows) Ø There is no correct workflow! if it works, it is good Ø Traditional (bring the data to the code) ● Download tgas_source (~1 GB) and/or gaia_source (~1 TB) ● Ingest it in your local supercomputer ● Ingest additional catalogues. Ingest/do cross matches ● Do science! Ø Gaia archive (towards bringing the code to the data) ● Selection (ADQL query) at Gaia Archive Reproducibility! ● Refinement, also at Gaia Archive: Xmatch, simple computation, comparison with external data, share with colleagues ● Download and do science! Data (size) reduction Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 3 ESA UNCLASSIFIED - Releasable to the Public

Gaia Archive contents: DR 1 Ø Where? ● Main Gaia archive: ESAC ● Partner data centres: AIP, ARI, ASI, BCN, CDS ● Affiliated and other data centres, enthusiasts: tens Ø What? ● tgas_source (~2 M 5 parameter astrometry): all ● gala_source (~1. 1 G positions): most ● Variability (599 cepheids, 2595 RR Lyrae), QSOs (2191): ESAC+ ● Xmatch (2 MASS, PPMXL, SDSS 9 , UCAC 4, URAT 1, WISE): ESAC ● External catalogues (for Xmatch): ESAC+ Ø What else @ESAC? : TAP+ (data base, user space, share), cross-match Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 4 ESA UNCLASSIFIED - Releasable to the Public

Gaia Archive contents: DR 1 Total DR 1: 1. 6 TB Gaia • Gaia DR 1 catalogue • TGAS 1. 1 0. 002 billion rows 0. 0012 1. 2 0. 47 0. 0025 0. 11 0. 029 billion billion External Catalogues • Hipparcos & Hipparcos new red. • IGSL (Initial Gaia Source List) • 2 MASS • Tycho 2 • UCAC 4 • Hubble Source Catalogue v 1. 0 rows rows Crossmatches • Crossmatch tables between Hipparcos, 2 MASS, Tycho 2… and Gaia expressed as neighbourhood and best neighbour, e. g: • All. Wise-Gaia neighbourhood 0. 31 billion rows 11. 59 billion rows Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 5 ESA UNCLASSIFIED - Releasable to the Public

The User Interface http: //archives. esac. esa. int/gaia Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 6 ESA UNCLASSIFIED - Releasable to the Public

TAP+ Open APIs TAP+ I/F Command line tools External Apps Public area • Publicly released catalogues Restricted area • Catalogues during validation or proprietary exploitation Data Validation User Space • User-uploaded data Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 7 ESA UNCLASSIFIED - Releasable to the Public

Gaia Archive: TAP+ Schema Persistent Upload Server Crossmatch Table sharing Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 8 ESA UNCLASSIFIED - Releasable to the Public

VOSpace q Astronomy Drop. Box q VO protocol q Inter. Operable q Accessible by VO Applications q Accessible by REST q Close to data Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 9 ESA UNCLASSIFIED - Releasable to the Public

Visualization q Visualization is a need q Statistics provide holistic views. Big data techniques q Validated static stats q On-the-fly visualization (Lisbon University) q ESASky integration q Virtual Reality prototype Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 10 ESA UNCLASSIFIED - Releasable to the Public

Gaia source: main table Columns Units Data: one entry (row) per object Data model Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 11 ESA UNCLASSIFIED - Releasable to the Public

Variables: light curve One entry per object and epoch: 3194 stars, 233, 181 rows ~Trillion size by DR 5! Table: phot_variable_time_series_gfov Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 12 ESA UNCLASSIFIED - Releasable to the Public

Pre-computed cross-matches Ø Gaia DR 1 is just one data set. Other catalogues needed ● Astrometry, radial velocities, photometry, stellar parameters, … Ø Ambiguity: different resolution, wavelength, epoch, proper motion, … ● Many to many relationship pivot tables needed traceability! Ø Gaia Archive: catalogue copies and pre-computed cross-matches ● Speed: indices. Efficiency: avoids ~100 M rows traffic Ø 2 MASS PSC: Skrutskie et al. 2006. 470, 992, 970 entries Ø All. WISE: Cutri et al. 2013, 747, 634, 026 entries Ø GSC 2. 3: Lasker et al. 2008. 945, 592, 683 entries Ø PPMXL: Roeser et al. 2010. 910, 468, 688 entries Ø SDSS DR 9: Ahn et al. 2012. 469, 029, 929 entries Ø UCAC 4: Zacharias et al. 2013. 113, 728, 883 entries Ø URAT-1: Zacharias et al. 2015. 228, 276, 482 entries Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 13 ESA UNCLASSIFIED - Releasable to the Public

Xmatch: neighbourhood Many to many relationship double entry one row per pair ω Cen core Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 14 ESA UNCLASSIFIED - Releasable to the Public

Xmatch: neighbourhood Many to many relationship double entry one row per pair ω Cen core Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 15 ESA UNCLASSIFIED - Releasable to the Public

Xmatch: best neighbour >0 other Gaia sources share best neighbour ω Cen core Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 16 ESA UNCLASSIFIED - Releasable to the Public

VO standards 3 rd party Ø Topcat (M. Taylor): Data query (TAP), reception (SAMP) & plotting Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 17 ESA UNCLASSIFIED - Releasable to the Public

Brown+ 2016: HR diagram Ø Plots are reproducible! ADQL queries in the paper Ø Uses Gaia parallaxes and fluxes absolute magnitudes Ø Uses pre-computed cross-match and external catalogue: 2 MASS Ø Histograms can be generated in the archive Code to the data DR 2+ Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 18 ESA UNCLASSIFIED - Releasable to the Public

Brown+ 2016: HR diagram select gaia. source_id, gaia. phot_g_mean_mag + 5 * log 10(gaia. parallax)- 10 as g_mag_abs, gaia. phot_g_mean_mag-tmass. ks_m as g_min_ks from gaiadr 1. tgas_source as gaia inner join gaiadr 1. tmass_best_neighbour as xmatch on gaia. source_id = xmatch. source_id inner join gaiadr 1. tmass_original_valid as tmass on tmass_oid = xmatch. tmass_oid where gaia. parallax/gaia. parallax_error >= 5 and ADQL: Learning curve ph_qual = 'AAA' and sqrt(power(2. 5/log(10)*gaia. phot_g_mean_flux_error /gaia. phot_g_mean_flux, 2) ) <= 0. 05 and sqrt(power(2. 5/log(10)*gaia. phot_g_mean_flux_error /gaia. phot_g_mean_flux, 2) + power(tmass. ks_msigcom, 2)) <= 0. 05 Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 19 ESA UNCLASSIFIED - Releasable to the Public

Brown+ 2016: HR diagram select gaia. source_id, ADQL: on the fly computation gaia. phot_g_mean_mag + 5 * log 10(gaia. parallax)- 10 as g_mag_abs, gaia. phot_g_mean_mag-tmass. ks_m as g_min_ks from gaiadr 1. tgas_source as gaia inner join gaiadr 1. tmass_best_neighbour as xmatch on gaia. source_id = xmatch. source_id Gaia + 2 MASS: Pre-computed Xmatch inner join gaiadr 1. tmass_original_valid as tmass on tmass_oid = xmatch. tmass_oid where gaia. parallax/gaia. parallax_error >= 5 and ph_qual = 'AAA' and sqrt(power(2. 5/log(10)*gaia. phot_g_mean_flux_error /gaia. phot_g_mean_flux, 2) ) <= 0. 05 and sqrt(power(2. 5/log(10)*gaia. phot_g_mean_flux_error /gaia. phot_g_mean_flux, 2) + power(tmass. ks_msigcom, 2)) <= 0. 05 Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 20 ESA UNCLASSIFIED - Releasable to the Public ADQL: filters

Brown+ 2016: HR diagram select g_min_ks_index / 10 as g_min_ks, g_mag_abs_index / 10 as g_mag_abs, count(*) as n 2 D histogram in the archive: no billion object download DR 2 from ( select gaia. source_id, floor((gaia. phot_g_mean_mag+5*log 10(gaia. parallax)-10) * 10) as g_mag_abs_index, floor((gaia. phot_g_mean_mag-tmass. ks_m) * 10) as g_min_ks_index from gaiadr 1. tgas_source as gaia inner join gaiadr 1. tmass_best_neighbour as xmatch on gaia. source_id = xmatch. source_id inner join gaiadr 1. tmass_original_valid as tmass on tmass_oid = xmatch. tmass_oid where gaia. parallax/gaia. parallax_error >= 5 and ph_qual = 'AAA' and sqrt(power(2. 5/log(10)*gaia. phot_g_mean_flux_error/gaia. phot_g_mean_flux, 2 )) <= 0. 05 and sqrt(power(2. 5/log(10)*gaia. phot_g_mean_flux_error/gaia. phot_g_mean_flux, 2) + power(tmass. ks_msigcom, 2)) <= 0. 05 )as subquery group by g_min_ks_index, g_mag_abs_index Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 21 ESA UNCLASSIFIED - Releasable to the Public

Brown+ 2016: light curves Ø Plots are reproducible! ADQL queries in the paper Ø Uses Gaia epoch photometry and best fit period Ø Light curves in a relational data base! Best strategy? DR 2+ RR Lyrae: 5284240582308398080 Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 22 ESA UNCLASSIFIED - Releasable to the Public

Brown+ 2016: light curves select curves. observation_time, mod(curves. observation_time, 1 / vars. phot_variable_fundam_freq 1) * vars. phot_variable_fundam_freq 1 as phase, curves. g_magnitude, 2. 5/log(10)* curves. g_flux_error/ curves. g_flux as g_magnitude_error from gaiadr 1. phot_variable_time_series_gfov as curves inner join gaiadr 1. variable_summary as vars on vars. source_id = curves. source_id where vars. source_id = 5284240582308398080 Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 23 ESA UNCLASSIFIED - Releasable to the Public

Brown+ 2016: light curves select curves. observation_time, Light curve reconstruction mod(curves. observation_time, 1 / vars. phot_variable_fundam_freq 1) * vars. phot_variable_fundam_freq 1 as phase, curves. g_magnitude, 2. 5/log(10)* curves. g_flux_error/ curves. g_flux as g_magnitude_error from gaiadr 1. phot_variable_time_series_gfov as curves inner join gaiadr 1. variable_summary as vars on vars. source_id = curves. source_id where vars. source_id = 5284240582308398080 Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 24 ESA UNCLASSIFIED - Releasable to the Public Source selection

The AS Role Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 25 ESA UNCLASSIFIED - Releasable to the Public

Scaling up: Hybrid infrastructures Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 26 ESA UNCLASSIFIED - Releasable to the Public

Ops for Gaia DR 1 Resource limits Sync rows Async rows Sync time Async time DB space Job results Anonymous 3 M 3 M 1 min 30 min none N/A (np) Registered unlimited 1 min* 30 min* 1 GB* (np) not persistent * Can be upgraded on demand Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 27 ESA UNCLASSIFIED - Releasable to the Public 1 GB*

Ops for Gaia DR 1 Job Scheduling Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 28 ESA UNCLASSIFIED - Releasable to the Public

Ops for Gaia DR 1 Archive UI usage: First 24 hours UI Usage • Usage sessions: 12, 005 • # of users: 10, 959 Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 29 ESA UNCLASSIFIED - Releasable to the Public

Ops for Gaia DR 1 File CDN download: First 15 days File download • Total volume downloaded: 73 TB • # download requests: 9 millions Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 30 ESA UNCLASSIFIED - Releasable to the Public

Ops for Gaia DR 1 TAP Interface: First 15 days TAP Interface • Number of queries • Synchronous: 174, 807 • Asynchronous: 95, 090 Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 31 ESA UNCLASSIFIED - Releasable to the Public

Future plans Next data handling challenges: DR 2 (~2017): 5 parameter astrometric solutions ~1 E 9 sources DR 3 (~2018): BP/RP/RVS spectra DR 4 (~2019): Solar System Objects & non-single star catalogues DR 5 (~2022): All epoch and transit data for all sources Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 32 ESA UNCLASSIFIED - Releasable to the Public

Data Volumes Downlink Statistics 25 July 2016 Volume Science data collected 38 TB Stars having passed focal plane 50 billion Astrometry transits 400 billion Photometry transits 100 billion Spectroscopy transits 9 billion Number of stars having passed RVS 3 billion Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 33 ESA UNCLASSIFIED - Releasable to the Public

Data Volumes MDB Statistics Volume Size (as of today) ~200 TB Size (estimation 5 -year ext. ) 1. 4 PB Number of entries (sources, transits…) 1000 billion Number of entries (est. 5 -year ext. ) 6000 billion Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 34 ESA UNCLASSIFIED - Releasable to the Public

Future plans: GAVIP Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 35 ESA UNCLASSIFIED - Releasable to the Public

Documentation and resources Ø Gaia Archive. Includes help and tutorials ● https: //archives. esac. esa. int/gaia Ø Gaia DR 1 papers http: //www. cosmos. esa. int/web/gaia/dr 1#A&A Ø Online documentation (361 pages) ● http: //gaia. esac. esa. int/documentation/GDR 1/index. html Ø Data model documentation ● https: //gaia. esac. esa. int/documentation/GDR 1/datamodel/ Ø ADQL: GAVO short course, UK ROE cookbook ● http: //docs. g-vo. org/adql-gaia/html/index. html ● https: //gaia. ac. uk/science/gaia-data-release-1/adql-cookbook Ø Gaia Helpdesk. https: //support. cosmos. esa. int/gaia/ Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 36 ESA UNCLASSIFIED - Releasable to the Public

Conclusions Ø Archive: http: //gea. esac. esa. int/archive/. Ø Helpdesk: https: //support. cosmos. esa. int/gaia/ Ø Functionality: TAP+ (data base, user space, sharing), cross-match Ø Data: main tables, TGAS, variables, QSO, Xmatch, external catalogues Ø Bring code to the data: select, refine (ADQL, user tables) and download Ø Prepare for DR 2 (1+ G sources). Archive might be the only way forward Ø Add ADQL queries to your papers (Brown et al. 2016) Reproducibility! Ø Use it. User demand is a key driver for future developments Ø Ask us. Via Helpdesk for additional support. Suggestions welcome Ø Want to support Gaia? write papers! (and acknowledge) Alcione Mora et al. | Gaia Archive: Architecture & key points | Gaia data workshop. Heidelberg | 2016 -11 -23 | Slide 37 ESA UNCLASSIFIED - Releasable to the Public