Progetti Europei le proposte presentate nelle call EINFRA12

  • Slides: 60
Download presentation
Progetti Europei: le proposte presentate nelle call EINFRA-12, EINFRA-21 e INFRAIA-02 e qualche accenno

Progetti Europei: le proposte presentate nelle call EINFRA-12, EINFRA-21 e INFRAIA-02 e qualche accenno ai proposal in altre call Luciano Gaido Riunione CCR, Torino, 3 Aprile 2017

Ringraziamenti Doverosi per: • Chi ha contribuito alla preparazione delle proposte presentate per le

Ringraziamenti Doverosi per: • Chi ha contribuito alla preparazione delle proposte presentate per le call che si sono chiuse il 29 marzo scorso: – Davide Salomoni, Giacinto Donvito, Daniele Cesini, Cristina Duma, Alessandro Costantini, Tommaso Boccali, Daniele Spiga, Claudio Grandi, Marco Verlato, Roberto Barbera • Chi ha fornito slide per questa presentazione: – Cristina Duma, Daniele Cesini, Davide Salomoni, Giacinto Donvito, Marco Verlato, Tiziana Ferrari 2

Sommario § § § § Il punto di partenza EOSCpilot (INFRADEV-04) EOSC-HUB (EINFRA-12) DEEP-Hybrid.

Sommario § § § § Il punto di partenza EOSCpilot (INFRADEV-04) EOSC-HUB (EINFRA-12) DEEP-Hybrid. Data. Cloud (EINFRA-21) XDC (EINFRA-21) ICARUS (INFRAIA-02) SCALE-UP Open Data. Cloud (ICT-16) 3

§ § § § Il punto di partenza EOSCpilot (INFRADEV-04) EOSC-hub (EINFRA-12) DEEP-Hybrid. Data.

§ § § § Il punto di partenza EOSCpilot (INFRADEV-04) EOSC-hub (EINFRA-12) DEEP-Hybrid. Data. Cloud (EINFRA-21) XDC (EINFRA-21) ICARUS (INFRAIA-02) SCALE-UP Open Data. Cloud (ICT-16) 4

Il punto di partenza: premessa Il programma della Commissione Europea H 2020 (2014 -2020)

Il punto di partenza: premessa Il programma della Commissione Europea H 2020 (2014 -2020) è strutturato in 3 distinti Work Programme: • WP 2014 -2015 – Concluso: i progetti sono in corso, vedi prossima slide • WP 2016 -2017 – In corso: per ICT, l’INFN è coinvolto in varie proposte già presentate o in corso di elaborazione e in un progetto già approvato • WP 2018 -2020 – il draft del programma è stato definito, siamo nella fase di consultazione degli esperti nazionali 5

Il punto di partenza Progetti in corso in nel Work Programme 2014 -2015: •

Il punto di partenza Progetti in corso in nel Work Programme 2014 -2015: • West-life: – Coordinatore: STFC, durata: 1. 11. 2015 – 30. 10. 2018, resp. naz. M. Verlato, Call: EINFRA-9 -2015, budget totale: 4 M€, budget INFN: 344 K€ • EGI_Engage: – Coordinatore: EGI Foundation (EGI. eu), durata: 1. 3. 2015 – 31. 8. 2017, resp. naz. : L. Gaido, Call: EINFRA-1 -2014, budget totale: 8 M€, budget INFN: 599. 5 K€ • INDIGO-Data. Cloud: – Coordinatore: INFN, durata: 1. 4. 2015 – 30. 9. 2017, PI: D. Salomoni, Call: EINFRA-1 -2014, budget totale: 11. 14 M€, budget INFN: 2. 08 M€ 6

§ § § § Il punto di partenza EOSCpilot (INFRADEV-04) EOSC-hub (EINFRA-12) DEEP-Hybrid. Data.

§ § § § Il punto di partenza EOSCpilot (INFRADEV-04) EOSC-hub (EINFRA-12) DEEP-Hybrid. Data. Cloud (EINFRA-21) XDC (EINFRA-21) ICARUS (INFRAIA-02) SCALE-UP Open Data. Cloud (ICT-16) 7

EOSCpilot Titolo: The European Open Science Cloud for Research Pilot Project • Call: INFRADEV-04

EOSCpilot Titolo: The European Open Science Cloud for Research Pilot Project • Call: INFRADEV-04 -2016 (European Open Science Cloud for Research) • Budget totale richiesto: 9. 8 M€ • Coordinatore: STFC • Partecipanti: 33 beneficiary e 15 linked third parties • Responsabile nazionale: C. Duma • Durata: 24 mesi, iniziato il 1 gennaio 2017 • Budget per INFN: 254. 4 K€ • Sezioni INFN coinvolte: CNAF, PG Prossime slide: grazie a C. Duma 8

EOSCpilot objectives The EOSCpilot represents a first step towards the development of the European

EOSCpilot objectives The EOSCpilot represents a first step towards the development of the European Open Science Cloud. It will: • Design and trial a stakeholder-driven governance framework • Contribute to the development of European open science policy and best practice; • Develop demonstrators of integrated services and infrastructures in a number of scientific domains, showcasing interoperability and its benefits; • Engage with a broad range of stakeholders, crossing borders and communities, to build trust and skills 9

EOSCpilot WP structure 10

EOSCpilot WP structure 10

EOSCpilot expected output • governance framework (rules of engagement, business plan, . . .

EOSCpilot expected output • governance framework (rules of engagement, business plan, . . . ), • 15 science demonstrators • federated service management framework • interoperability demonstrators • identification of training needs • open innovation platform for industry engagement • workshops for engagement of different stakeholders 11

EOSCpilot science demonstrators 5 already selected: • Pan. Cancer Analysis of Whole genomes (EMBL):

EOSCpilot science demonstrators 5 already selected: • Pan. Cancer Analysis of Whole genomes (EMBL): Sensitive genomic data for cancer patient health care • ENVRI Radiative Forcing Integration (ICOS ERIC + ACTRIS + DKRZ + IPSL): Integration of heterogeneous climate data sources • Research with Photons & Neutrons (DESY, ESFR, XFEL, ESS, EMBL, ILL): Exploitation of data from analytical facilities • WLCG (CERN): Large-scale long-term data preservation and reuse of physics data • TEXTCROWD (Univ. of Florence): Collaborative semantic enrichment of text-based datasets –TEXTCROWD 10 more demonstrators being selected through an open call 12

EOSCpilot Interoperability WP (WP 6) Objectives: • Research and Data Interoperability: data & services

EOSCpilot Interoperability WP (WP 6) Objectives: • Research and Data Interoperability: data & services to be findable, accessible, interoperable and reusable (FAIR) • Infrastructure interoperability: facilitate the most adequate infrastructures for the treatment of extensive amounts of data. Demonstrate with multiinfrastructure, multi-community pilots Key Output: • The design of a future EOSC based on federated interoperable services meeting the needs of thematic research domains and wider user base • Tasks: • T 6. 1: gap analysis & interoperability architecture [CNRS lead] • T 6. 2: EOSC Research and Data interoperability [ELIXIR lead] • T 6. 3: Interoperability pilots [INFN lead] 13

EOSCpilot T 6. 3 - Interoperability pilots Validation regarding: • AAI requirements for both

EOSCpilot T 6. 3 - Interoperability pilots Validation regarding: • AAI requirements for both e-infrastructures and scientific communities solutions offered by INDIGO- Data. Cloud, ELIXIR AAI, EUDAT B 2 ACCESS and AARC • Resource brokering solutions - spanning multiple infrastructures and user communities and aimed for high-level resource discoverability and addressability • Accessibility - EOSC local, Grid, HPC and Cloud resources accessible by multiple communities • Data accessibility – through personal resources, scientific portals, CLI • Interoperability - underlying distributed storage systems with the EOSC platform services; services and tools - as those provided by the EUDAT service suite and INDIGO-Data. Cloud toolbox 14

§ § § § Il punto di partenza EOSCpilot (INFRADEV-04) EOSC-hub (EINFRA-12) DEEP-Hybrid. Data.

§ § § § Il punto di partenza EOSCpilot (INFRADEV-04) EOSC-hub (EINFRA-12) DEEP-Hybrid. Data. Cloud (EINFRA-21) XDC (EINFRA-21) ICARUS (INFRAIA-02) SCALE-UP Open Data. Cloud (ICT-16) 15

EOSC-hub Titolo: Integrating and managing services for the European Open Science Cloud • Call:

EOSC-hub Titolo: Integrating and managing services for the European Open Science Cloud • Call: EINFRA-12 -2017 (Data and Distributed Computing einfrastructures for Open Science), topic a: Secure and agile data and distributed computing e-infrastructure • Budget totale richiesto: 30 M€ • Coordinatore: EGI Foundation (EGI. eu) • Partecipanti: 74 beneficiari (più 20 linked third-parties) da 36 paesi diversi • Responsabile nazionale: L. Gaido • Durata: 36 mesi • Budget per INFN: 1. 8 M€ (al secondo posto dopo EGI. eu) 16 • Sezioni INFN coinvolte: BA, CNAF, CT, PD, PG, TO

EOSC-hub Proposal congiunto tra EGI, EUDAT e INDIGO-Data. Cloud per la call EINFRA-12, topic

EOSC-hub Proposal congiunto tra EGI, EUDAT e INDIGO-Data. Cloud per la call EINFRA-12, topic a: (a) Secure and agile data and distributed computing e-infrastructures: fostering the integration of a secure, permanent, on-demand service-driven, privacy-compliant and sustainable e-infrastructure incorporating distributed databases, computing resources and software. The challenge is to integrate at European level the geographically and disciplinary dispersed resources to achieve economies of scale and efficiency gains in providing the best data and computing capacity and services to the research and education communities. This action is interrelated to INFRADEV-04 -2016, “European Open Science Cloud for Research”. 17

EOSC-hub Oltre a background e strutture di gestione diverse, c’è un’altra differenza sostanziale: EGI

EOSC-hub Oltre a background e strutture di gestione diverse, c’è un’altra differenza sostanziale: EGI e EUDAT sono (o sono considerate) infrastrutture mentre INDIGO-Data. Cloud è un progetto di sviluppo software 18

EOSC-hub Scultura di Georges Faures, attualmente sulla terrazza dell’hotel Bologna, in corso Vittorio Emanuele,

EOSC-hub Scultura di Georges Faures, attualmente sulla terrazza dell’hotel Bologna, in corso Vittorio Emanuele, angolo via XX settembre 19

Objectives • Simplify access to a broad portfolio of products, resources and service provided

Objectives • Simplify access to a broad portfolio of products, resources and service provided by major pan-European and international organizations through an open service catalogue – A cornerstone of the EOSC • Remove fragmentation of service provisioning and access to digital services in Europe and beyond – Technical integration between common & thematic services – Service innovation, procurement, provisioning and access • Increase innovation capacity of digital Infrastructures 20

Objectives (cont) • Consolidate digital infrastructures by – Expanding capacities and capabilities – Improving

Objectives (cont) • Consolidate digital infrastructures by – Expanding capacities and capabilities – Improving discoverability, access, interoperability and sharing , across research communities and countries • Extend access to integrated compute, storage, data and software to new user groups including high-education and industry, increase the user base • Expand human capacity – Consolidate/Expand a distributed network of experts and service operators at local/national level 21

Service architecture Community Support services Thematic Service Federation Services AAI, Accounting, Monitoring, Operations, Security

Service architecture Community Support services Thematic Service Federation Services AAI, Accounting, Monitoring, Operations, Security Coord. Thematic Service Added Value Services Compute, Data, Software Management and Preservation Basic Infrastructure Compute and Storage Common services Thematic Service Open Collab. Platforms Application Repository, Configuration Management, Marketplace

Technology Readiness Level Development Phase 1 TRL Drawing board Proof-of. Concept 2 3 4

Technology Readiness Level Development Phase 1 TRL Drawing board Proof-of. Concept 2 3 4 Go/No. Go Service Definition Release naming Convention Service Portfolio Management Documentation Operated by Basic Ideas Preproduction Pilot 5 6 Go/No. Go 7 Production 7 Go/No. Go 9 8 Go/No. Go Define Scope Basic Roadmap RC Roadmap SR Update Roadmap Alfa Beta RC Stable Portfolio Catalog Basic Service Building First version Published Operations 23

Project structure S 24

Project structure S 24

Effort distribution S Effort per Category of services Total % Management and Coordination (WP

Effort distribution S Effort per Category of services Total % Management and Coordination (WP 1) 267 7% Service Planning (WPs 2 -3 -12) 348 9% Service integration and delivery (WPs 4 -10) 763 20% Service Catalogue (WPs 5 -6 -7) 1, 001 26% Service Adoption (WPs 8 -9 -11 -13) 1, 506 39% Grand Total 3, 885 100% 25

Engagement with research communities • Thematic Service Providers – Interested in providing a thematic

Engagement with research communities • Thematic Service Providers – Interested in providing a thematic production service as part of e-Infra-12 and the future EOSC • Early adopters – Interested in piloting common services, using and advancing cross-infrastructure usage for the benefit of their research community and beyond – Organized through “competence centers” bringing together e-Infrastructure providers and relevant research organizations and technology providers/experts Both have already been selected via an open call, but an additional call is expected during the project (about 0. 5 M€ earmarked) 26

Thematic Services S 60 proposals received, 9 selected: • CLARIN (European Research Infrastructure for

Thematic Services S 60 proposals received, 9 selected: • CLARIN (European Research Infrastructure for Language Resources and Technology) • DODAS (CMS) Lead: INFN (vedi present. di D. Spiga) • ECAS (climate studies) • GEOSS (Global Earth Observation System of Systems) • OPENCoasts (On-demand Operational Coastal Circulation Forecast Service) • We. NMR (Worldwide e-Infrastructure for Nuclear Magnetic Resonance and structural biology) • DARIAH (Digital Research Infrastructure for the Arts and Humanities) • Life. Watch • EO Pillar (Earth Observation) 27

Early Adopters/Competence Centers The Competence Centre is driven by well established and mature research

Early Adopters/Competence Centers The Competence Centre is driven by well established and mature research infrastructure or international collaboration requiring advanced and integrated data and computing services. In the Competence Centre early adopters test, adapt, and integrate the digital capabilities they need to pursue they research, with the support of e-Infrastructure and technology experts. The Competence Centre will: • Run Proof of Concepts • Conduct Pilots • Prepare the production environment • Define appropriate business models to sustain the solutions after the end of the project 28

Competence Centers Out of 51 proposals received, 7 Competence Centres have been selected: •

Competence Centers Out of 51 proposals received, 7 Competence Centres have been selected: • Elixir Lead Partner: EMBL-EBI • Fusion Lead Partner: CCFE • Marine Lead Partner: IFREMER • EISCAT_3 D Lead Partner: EISCAT • EPOS-ORFEUS Lead Partner: SURFsara • Radio Astronomy Competence Center (RACC) Lead Partner: ASTRON • ICOS Lead Partner: SNIC • Disaster Mitigation Competence Centre Plus (DMCC+) Lead Partner: ASGC, unfunded 29

Involvement of Industries Organized through Business Pilots: • relevant to EGI participants with a

Involvement of Industries Organized through Business Pilots: • relevant to EGI participants with a national business programme and have activities of pan-European impact/interest • designed to foster innovation between e-Infrastructures and the private sector through building an ecosystem of SMEs, large industries, startups, researchers, accelerators, and investors to become active business partners of e-Infrastructures as customers and/or service providers. These initial pilots will serve as early demonstrators of the project’s Joint Digital Innovation Hub (DIH). Selected via an open call at proposal preparation 30

Business Pilots 6 proposals selected, out of 31 received: • Cyber. HAB (Water body

Business Pilots 6 proposals selected, out of 31 received: • Cyber. HAB (Water body management sector) • Sports Smart Video Analysis (Sports sector) • Bot Mitigation Engine (Business sector) • ACTION Seaport (Local coastal authorities) • Space Weather Data Services for the future DRACO Observatory (Climate sector) • Furniture Enterprise Analytics - Data. Furn (Furniture industry sector) Two transversal activities will support these pilots, an Open. Lab and the commercialization support. 31

Linking programme Goal: • Linking of the EOSC-hub with local/national e-Infrastructures from all European

Linking programme Goal: • Linking of the EOSC-hub with local/national e-Infrastructures from all European regions to: – Contribute to user community engagement through the provisioning of infrastructure services to user communities of the EOSC-hub engagement roadmap, and to participate to direction giving activities involving EOSC-hub stakeholders. – Publish local/national services in the EOSC-hub catalogue in compliance to the EOSC-hub rules of engagement for service providers Budget: • about 500 K€ in task T 4. 1 (4 to 6 PMs) 32

Ruolo dell’INFN/1 • WP 1 (6 PM) per coordinamento WP 10 • WP 2

Ruolo dell’INFN/1 • WP 1 (6 PM) per coordinamento WP 10 • WP 2 (24 PM) per attività di governance, strategy, service portfolio management • WP 4 (16 PM) per contributo a operations coordination (V. Spinoso) • WP 6 (45 PM) per maintenance di alcune componenti grid (CREAM, BDII, ARGUS, VOMS) e INDIGO (IAM, Paa. S, FG, …. ) 33

Ruolo dell’INFN/2 • WP 7 (36 PM) per attività in 2 Thematic Service (DODAS

Ruolo dell’INFN/2 • WP 7 (36 PM) per attività in 2 Thematic Service (DODAS e WENMR) • WP 10 (56 PM) per coordinamento del Technology Committee, coord. task T 10. 1 (technical roadmap), evoluzione service catalogue e supporto in varie aree tematiche (AAI, Paa. S, User Interfaces, Data Solutions) • WP 11 (17 PM) per contributo ad attività di training e coordinamento task T 6. 4 • WP 13 (51 PM) per provisioning di risorse per vari servizi fondi attraverso meccanismo del Virtual Access 34

We. NMR Thematic Service/1 • Inheritance of EGI-Engage Mo. Brain Competence Centre • Involved

We. NMR Thematic Service/1 • Inheritance of EGI-Engage Mo. Brain Competence Centre • Involved partners are the core partners of four past and current EU projects supporting the Structural Biology community: Participants: • University of Utrecht (coordinator) • Magnetic Resonance Center (CERM)/University of Florence • INFN-Padova • … and have also been partners in INDIGO-Data. Cloud, providing use cases to drive the development of novel cloud 35 solutions

We. NMR Thematic Service/2 • 8 grid-enabled application web portals already in production (TRL

We. NMR Thematic Service/2 • 8 grid-enabled application web portals already in production (TRL 9): 6 hosted at University of Utrecht (DISVIS, POWERFIT, HADDOCK, GROMACS, CS-ROSETTA, UNIO) and 2 at CERM (FANTEN, AMPS-NMR) • ~ 20 M normalized CPU-hours/year on EGI HTC platform (SLA in place with EGI) • DISVIS, POWERFIT and AMPS-NMR pioneering the use of HTC GPGPU resources o via udocker tool (INDIGO devel. ) and GPU-enabled CREAM-CE (INFN-Padova/Milano devel. ) • …but also exploring the way to move the workload to cloud (e. g. in 2 INDIGO use-cases) 36

We. NMR Thematic Service/3 • Planned activities encompass: • User support and training, Outreach

We. NMR Thematic Service/3 • Planned activities encompass: • User support and training, Outreach and dissemination • Continuous operation of the various grid- and cloud-enabled web portals, their Consolidation / upgrading and provisioning • Consolidation of the job submission frameworks of the various portals building on DIRAC 4 EGI and INDIGO solutions (e. g. phasing out glite-WMS in favor of DIRAC or Paa. S Orchestrator) • Integration of distributed data storage solutions (from EUDAT and One. Data) • Implementation of AAI solutions developed by EGI and INDIGO • INFN main role: • Maintenance, operations and support of enmr. eu VO related services and users (VOMS servers, LFC, HTC and Fed. Cloud resources, GPU resources) • Supporting the service evolution by integrating (where needed) advanced INDIGO solutions like Paa. S Orchestrator, IAM, One. Data. E. g. INFN is already providing testing One. Data storage to We. NMR users 37

DODAS Thematic Service Vedi presentazione di Daniele Spiga 38

DODAS Thematic Service Vedi presentazione di Daniele Spiga 38

Sommario § § § § Il punto di partenza EOSCpilot (INFRADEV-04) EOSC-HUB (EINFRA-12) DEEP-Hybrid.

Sommario § § § § Il punto di partenza EOSCpilot (INFRADEV-04) EOSC-HUB (EINFRA-12) DEEP-Hybrid. Data. Cloud (EINFRA-21) XDC (EINFRA-21) ICARUS (INFRAIA-02) SCALE-UP Open Data. Cloud (ICT-16) 39

DEEP-Hybrid. Data. Cloud Titolo: Designing and Enabling E-Infrastructures for intensive Processing in a Hybrid

DEEP-Hybrid. Data. Cloud Titolo: Designing and Enabling E-Infrastructures for intensive Processing in a Hybrid Data. Cloud • Call: EINFRA-21 -2017, topic b: Research and Innovation Actions for e-Infrastructure prototypes, subtopic 2: Computing e-infrastructure with extreme large datasets • Budget totale richiesto: 2. 99 M€ • Coordinatore: CSIC • Partecipanti: 10 • Responsabile nazionale: G. Donvito • Durata: 30 mesi • Budget per INFN: 375 K€, al secondo posto dopo CSIC • Sezioni INFN coinvolte: Bari, CNAF, Torino Prossime slide: grazie a G. Donvito 40

DEEP-Hybrid. Data. Cloud Siamo il secondo partner insieme a LIP dopo il coordinatore per

DEEP-Hybrid. Data. Cloud Siamo il secondo partner insieme a LIP dopo il coordinatore per numero di PM e il secondo partner per finanziamento in € 41

DEEP-Hybrid. Data. Cloud: project description 42

DEEP-Hybrid. Data. Cloud: project description 42

DEEP-Hybrid. Data. Cloud: obiettivi • Objective 1: Focus the interest of intensive computing techniques

DEEP-Hybrid. Data. Cloud: obiettivi • Objective 1: Focus the interest of intensive computing techniques on the analysis of very large datasets, considering demanding cases from different research communities, in the context of the future generations of e-infrastructure. • Objective 2: Evolve, up to production level, intensive computing services exploiting specialized hardware components, like GPUs, low-latency interconnects, and others usually accessed as “bare metal” resources. The services, based on open source software, will follow existing standards to guarantee their deployment and orchestration on different platforms. • Objective 3: Integrate the intensive computing services under a Hybrid Cloud approach, assuring interoperability with the existing EOSC platforms and their services. 43

DEEP-Hybrid. Data. Cloud: obiettivi (cont) • Objective 4: Define a “DEEP as a Service”

DEEP-Hybrid. Data. Cloud: obiettivi (cont) • Objective 4: Define a “DEEP as a Service” solution to offer an easy integration path to the developers of final applications. • Objective 5: Analyse the complementarity with other ongoing projects targeting added value services for the cloud, in particular those related to the management of extremely large datasets. • Global Objective: Promote the use of intensive computing services by different research communities and areas, and their support by the corresponding e-infrastructure providers and open source projects. Areas for dissemination via pilot applications: Medical Images, Citizen Science, Earth Observation, Post-processing very large datasets coming from large scale simulations in Lattice QCD, Genomics, Radio. Astronomy 44

INFN in DEEP-Hybrid. Data. Cloud • Sezioni INFN coinvolte: Bari, CNAF e Torino •

INFN in DEEP-Hybrid. Data. Cloud • Sezioni INFN coinvolte: Bari, CNAF e Torino • Persone coinvolte nel proposal: Giacinto Donvito, Alessandro Costantini, Stefano Bagnasco. • Attività dell’INFN: – Use Case: INFN team in Bari collaborates in the application of NN to the study of the hippocampal segmentation (Alzheimer’s disease). (WP 2) – Attività di Testbed e Release&SQA (WP 3) – Exploiting Bare metal resources (WP 4) sia per GPGPU che Lowlatency network – Coordinare l’attività di Hybrid Cloud (WP 5) – Implementazione di soluzioni di “DEEP as a Service” (WP 6) – Liaison con i servizi di INDIGO e con i servizi sviluppati in XDC e in produzione in EOSC-Hub 45

Sommario § § § § Il punto di partenza EOSCpilot (INFRADEV-04) EOSC-HUB (EINFRA-12) DEEP-Hybrid.

Sommario § § § § Il punto di partenza EOSCpilot (INFRADEV-04) EOSC-HUB (EINFRA-12) DEEP-Hybrid. Data. Cloud (EINFRA-21) XDC (EINFRA-21) ICARUS (INFRAIA-02) SCALE-UP Open Data. Cloud (ICT-16) 46

The e. Xtreme Data. Cloud (XDC) Titolo: The e. Xtreme Data. Cloud (XDC) •

The e. Xtreme Data. Cloud (XDC) Titolo: The e. Xtreme Data. Cloud (XDC) • Call: EINFRA-21 -2017, topic b: Research and Innovation Actions for e-Infrastructure prototypes, subtopic 2: Computing e-infrastructure with extreme large datasets • Budget totale richiesto: 3. 1 M€ • Coordinatore: INFN (PI: Daniele Cesini) • Partecipanti: 8 • Durata: 27 mesi • Budget per INFN: 583 K€, al primo posto • Sezioni INFN coinvolte: Bari, CNAF, Padova Prossime slide: grazie a D. Cesini 47

The XDC Project • develop scalable technologies for federating storage resources and managing data

The XDC Project • develop scalable technologies for federating storage resources and managing data in highly distributed computing environments • Will be based on existing tools (TRL 8+) that will be enriched by new functionalities and plugins already available as prototypes (TRL 6) • Target platforms: current and next generation e-Infrastructures deployed in Europe (EOSC, EGI, WLCG and other EINFRA-12 -funded infrastructures) ID Partner Country Represented Community Tools and system that will be developed 1 INFN (Lead) IT HEP/WLCG 2 DESY DE Astroparticle Physics, Research with Photons d. Cache, Orchestrator, Smart Caching mechanisms 3 CERN CH HEP/WLCG EOS, DYNAFED, FTS, Smart Caching mechanisms 4 Cyfronet PL 5 ECRIN [ERIC] Medical data 6 CSIC ES Lifewatch 7 CNRS FR Astro [CTA and LSST] 8 EGI. eu NL EGI communities INDIGO-Orchestrator, Smart caching mechanisms, Access pattern analyzer ONEDATA 48

XDC technical topics (1/2) • Intelligent & Automated Dataset Distribution 1. Orchestration to realize

XDC technical topics (1/2) • Intelligent & Automated Dataset Distribution 1. Orchestration to realize a policy-driven data management 2. Data distribution policies based on Quality of Service (i. e. disks vs tape vs SSD) at infrastructure level (cross-sites) • Data pre-processing during ingestion 1. Automatically run user defined applications and workflows when data are uploaded i. e. for Skimming, indexing, metadata extraction, consistency checks • Data management based on access partners 1. Move to ‘glacier-like’ storage unused data, move to fast storage “hot” data 2. access predictions to improve data availability 49

XDC technical topics (2/2) • Smart caching 1. Extension of a site to remote

XDC technical topics (2/2) • Smart caching 1. Extension of a site to remote location 2. Federated and geographically distributed cache i. e. Federation-wide caches • • • Advanced metadata management with no pre-defined schema 1. Within ONEDATA Encryption Services and Secure Storage 1. Sensitive data management and key storage within ONEDATA Multi-region support in ONEDATA to federate storage system at the Exascale 50

XDC Project Structure n INFN involved in all WPs: n CNAF, BA, PD n

XDC Project Structure n INFN involved in all WPs: n CNAF, BA, PD n INFN leads WP 1 and WP 3 n INFN will represent WLCG in WP 2 n INFN involved in WP 4 to develop: n n n The Paa. S-Orchestrator n policy (Qo. S) driven data management n Pre-processing workflows Smart caching mechanisms Access Pattern Analyzer 51

Sommario § § § § Il punto di partenza EOSCpilot (INFRADEV-04) EOSC-HUB (EINFRA-12) DEEP-Hybrid.

Sommario § § § § Il punto di partenza EOSCpilot (INFRADEV-04) EOSC-HUB (EINFRA-12) DEEP-Hybrid. Data. Cloud (EINFRA-21) XDC (EINFRA-21) ICARUS (INFRAIA-02) SCALE-UP Open Data. Cloud (ICT-16) 52

ICARUS Titolo: Interoperable e. Cosystem of historic. Al sou. Rces for e. Uropean hi.

ICARUS Titolo: Interoperable e. Cosystem of historic. Al sou. Rces for e. Uropean hi. Story (ICARUS) • Call: INFRAIA-02 -2017 (seconda fase) • Budget totale richiesto: 4. 9 M€ • Coordinatore: CNR-OVI • Partecipanti: 16 • Responsabile nazionale: L. Gaido (G. Donvito) • Durata: 48 mesi • Budget per INFN: 493 K€, al secondo posto, dopo il coordinator • Sezioni INFN coinvolte: Bari, CNAF, Torino 53

ICARUS Goal: • reduce the fragmentation of the historical sources digital ecosystem, bringing together

ICARUS Goal: • reduce the fragmentation of the historical sources digital ecosystem, bringing together and integrating in a rich and interoperable Web of Historical Data the existing research datasets, allowing scholars and researchers to exploit new and powerful technologies as an integral component of the historical research methodology The integration activity will enable seamless access of researchers to historical datasets, and provide new Web-based services and tools as well as common interfaces to data repositories. 54

ICARUS: ruolo INFN • Contribuire alla definizione dell’architettura del progetto • Supportare l’integrazione di

ICARUS: ruolo INFN • Contribuire alla definizione dell’architettura del progetto • Supportare l’integrazione di alcune componenti di INDIGO (IAM, Paa. S, Onedata) con il framework per la data analytics (D 4 Science) fornito da CNR-ISTI • Fornire alcune risorse per i servizi del progetto finanziamento tramite Virtual Access • Agire da liaison con la EOSC, grazie alla partecipazione al progetto EOSC-hub 55

Sommario § § § § Il punto di partenza EOSCpilot (INFRADEV-04) EOSC-HUB (EINFRA-12) DEEP-Hybrid.

Sommario § § § § Il punto di partenza EOSCpilot (INFRADEV-04) EOSC-HUB (EINFRA-12) DEEP-Hybrid. Data. Cloud (EINFRA-21) XDC (EINFRA-21) ICARUS (INFRAIA-02) SCALE-UP Open Data. Cloud (ICT-16) 56

SCALE-UP Open Data. Cloud Titolo: titolo provvisorio INDIGO-Data. Extreme • Call: ICT-16 -2017, topic:

SCALE-UP Open Data. Cloud Titolo: titolo provvisorio INDIGO-Data. Extreme • Call: ICT-16 -2017, topic: Big data PPP: research addressing main technology challenges of the data economy • Proposal in preparazione, scadenza call: 25. 4 2017 • Budget indicativo: 5 M€ • Coordinatore: ATOS Coinvolgimento di diverse ditte (Bull, Ontotext, EBIT S. r. l. e probabilmente anche una ditta operante nel campo dei satelliti) Scopo: indirizzare soluzioni software innovative legate a big data partendo da INDIGO-Data. Cloud e dal proposal DEEPHybrid. Data. Cloud. Il consorzio comprende inoltre CSIC, INFN, LIP, UPV, ATOS, PSNC, 57 IISAS (Slovak Academy of Science).

Conclusioni/1 • Le competenze e l’esperienza maturate dall’INFN nei progetti Grid fino al FP

Conclusioni/1 • Le competenze e l’esperienza maturate dall’INFN nei progetti Grid fino al FP 7, e la posizione di rilevanza a livello internazionale sono state mantenute nella prima fase del programma H 2020 (WP 2014 -2015), nonostante il salto tecnologico con il passaggio al Cloud Computing. Ne sono prova la leadership di INDIGO-Data. Cloud e il ruolo rilevante in EGI_Engage e West-life. • Nel WP 2016 -2017 la rilevanza a livello europeo è confermata dal ruolo dell’INFN nei vari proposal presentati importante la capitalizzazione di quanto fatto nei progetti precedenti • è importante anche a livello nazionale, ma serve una strategia comune con gli altri enti di ricerca per fare massa critica • Per l’INFN i finanziamenti dai progetti europei sono molto importanti per ovvie ragioni, ma questo non basta…. 58

Conclusioni/2 • Lo ‘scollamento’ tra quanto fatto nei progetti ICT, in particolare quelli finanziati

Conclusioni/2 • Lo ‘scollamento’ tra quanto fatto nei progetti ICT, in particolare quelli finanziati dalla EC, e le attività ‘core’ dell’INFN è stato avvertito in modo forte nei progetti Grid (da EGI_Inspire in poi) • Il riavvicinamento è cominciato già in INDIGO-Data. Cloud e si è rafforzato in EOSC-hub (ad es. Con il TS DODAS) • Qui le sezioni INFN non direttamente coinvolte nei progetti europei possono avere un ruolo importante: – negli esperimenti INFN – nei resource/data center INFN. E nella Corporate Cloud? Ma questo deve essere supportato da una azione decisa di dissemination (compreso training) ed exploitation Questa dovrebbe essere parte della strategia dell’INFN. 59

Grazie per l’attenzione! Per chi è interessato: dal 9 al 12 Maggio a Catania

Grazie per l’attenzione! Per chi è interessato: dal 9 al 12 Maggio a Catania ci sarà il Summit 2017 di INDIGO organizzato congiuntamente con la conferenza EGI: https: //indico. egi. eu/indico/event/3249 60