EGIIn SPIRE The EGI Federated Cloud architecture and

EGI-In. SPIRE The EGI Federated Cloud, architecture and use cases David Wallom University of Oxford Chair, EGI Federated Cloud EGI-In. SPIRE RI-261323 www. egi. eu

Desire for growth of userbase 20 k researchers Few related use cases EGI-In. SPIRE RI-261323 2 M researchers! Many diverse use cases www. egi. eu

Principles of Federation • Resource integration: Iaa. S Cloud Computing to be integrated into the existing production infrastructure. • Heterogeneous implementation: no mandate on the cloud technology. • Standards and validation: Use recommended and common open standards for interfaces and images • Provider agnosticism: the only condition to federate resources is to expose the chosen interfaces and services. EGI-In. SPIRE RI-261323 www. egi. eu

Value proposition The EGI Federated Cloud is the federation of public and private Clouds, offering Services to researchers initially in Europe, in future sharing and collaborating globally. An Iaa. S cloud visible to the user as a single system able to • Scale to user needs • Integrate multiple different providers to give resilience • Prevent vendor lock-in • Enable resource provision targeted towards the research community Standards based federation of Iaa. S cloud: • Exposes a set of independent cloud services accessible to users utilising a common standards profile • Allows deployment of services across multiple providers and capacity bursting EGI-In. SPIRE RI-261323 www. egi. eu

VM Instance Management through r. OCCI EGI-In. SPIRE RI-261323 www. egi. eu

Contextualisation • Process of installing, configuring and preparing software upon boot time • All uploaded Fedcloud images should support Cloudinit • All supported CMF OCCI implementations extended to support EGI-In. SPIRE RI-261323 www. egi. eu

CDMI Technology and status Value • Cloud Object Storage Service, PUT and GET files into the storage, accessible by cloud appliances and other applications Pioneering open implementation (Stoxy) • Multiplatform • Support for flexible authentication schemas (incl. PAM) • Open. Stack Keystone token compliant • Selected as a supporting technology for FP 7 Strategic project for Open. Data use case • Other implementations from BSC (Emotive), DESY (d. Cache) 9 EGI-In. SPIRE RI-261323 www. egi. eu

Service Discovery & Information Publishing Information system. Each cloud infrastructure exposes a LDAP server publishing information by means of a customised GLUE 2 schema. Each LDAP server is polled by a top-BDII server. ldap: //test 03. egi. cesga. es: 2170 EGI-In. SPIRE RI-261323 www. egi. eu

GOCDB Illustrating Cloud type EGI-In. SPIRE RI-261323 www. egi. eu

Accounting • • Current Usage Record Content Site User: local user/group, Digital Certificate DN VMUUID (VM instance identifier) Start (and End) Time Status Wall and CPU Duration VM source image Cloud Type 156276 VMs launched across 24 clouds EGI-In. SPIRE RI-261323 14 www. egi. eu

Status EGI-In. SPIRE RI-261323 16 www. egi. eu

Federated AAI Inherited from the EGI AAI framework • Scalable with many users • Uniform authentication across multiple service providers • Users manage their community via Virtual Organisations EGI-In. SPIRE RI-261323 www. egi. eu

App. DB – Virtual Appliance Catalogue EGI-In. SPIRE RI-261323 20 www. egi. eu

Brokering EGI-In. SPIRE RI-261323 www. egi. eu

SCREENSHOT OF EGI-In. SPIRE RI-261323 www. egi. eu

Security • Mandatory survey for both resource and technology providers EGI-In. SPIRE RI-261323 www. egi. eu

Extending the EGI Marketplace • • • Rich and diverse set of individual offerings Application/VM Image repository User community choice of provider at all levels Academic and commercial resource & service providers User community from the public and private sector EGI-In. SPIRE RI-261323 www. egi. eu

EGI Cloud Infrastructure User Community Image EGI Application EGI Cloud Service Marketplace Repository DB EGI Collaboration Tools EGI Cloud Infrastructure Platform Instance Mgmt Storage Management OCCI CDMI Stacks Providers. Cloud. Management Framework OVF Open. Nebula, Synnefo, (new one we(Open. Stack, don’t now about and don’t want…) to care about) Service Registry GLUE 2 Information Discovery Help and Support GSI SAM UR Federated AAI Monitoring Accounting Security Coordination Training and Outreach Sustainable Business Models EGI Core Platform EGI-In. SPIRE RI-261323 29 www. egi. eu

Geographical dispersion • 12 countries provide 15 certified resources from the public and private sector – • 2 countries currently integrating – • Czech Republic, Germany, Greece, Hungary, Italy, Macedonia, Poland, Slovakia, Spain, Sweden, Turkey, United Kingdom Croatia, Finland 5 countries interested – Bulgaria, France, Israel*, The Netherlands, Switzerland • + other resource providers in existing member countries • Worldwide partnership/interest – – Australia* (NECTAR) South Africa* (SAGrid) South Korea* (KISTI) United States* (NIST, NSF Centres) * Not shown on map EGI-In. SPIRE RI-261323 www. egi. eu

Federated Cloud Services Virtual e. Laboratory VRE Paa. S ACL mgmt Encryption Key Mgmt Hadoop aa. S DB aa. S Paa. S Secure storage Federated Iaa. S and STaa. S Cloud 17 Feb 2014 EGI-In. SPIRE RI-261323 EYR TC monthly meeting Tier 4: Zero ICT Infrastructures Tier 3: Platform as a Service Tier 2: General-purpose platform services Tier 1: Reliable Infrastructure Cloud 32 www. egi. eu

EGI Fed. Cloud Communities PY 4 • Ecology – Bio. Ve. L: Biodiversity Virtual e-Laboratory • Structural biology – We. NMR: a worldwide e-Infrastructure for NMR and structural biology • Linguistics – CLARIN: ‘British National Corpus’ service (BNCWeb) • Earth Observation – SSEP: European Space Agency’s Supersites Exploitation Platform for volcano and earthquakes monitoring (Collaboration with Helix Nebula) • Software Engineering – SCI-BUS: simulated environments for portal testing • Software Engineering – DIRAC: deploying ready-to-use distributed computing systems • Software Engineering – Catania Science Gateway Framework • Musicology – Peachnote: dynamic analysis of musical scores • Earth Observation – ENVRI: Common Operations of Environmental Research infrastructures (collaboration with EISCAT 3 D) • Geology – VERCE: Virtual Earthquake and seismology Research • Ecology – Life. Watch: E-Science European Infrastructure for Biodiversity and Ecosystem Research • High Energy Physics – CERN ATLAS: ATLAS processing cluster via Helix. Nebula More info: https: //wiki. egi. eu/wiki/Fedcloud-tf: Users EGI-In. SPIRE RI-261323 www. egi. eu

We. NMR e-Infrastructure (Virtual. Cing) Virtual Common Interface for NMR structure generation (Virtual. Cing), developed in We. NMR, has been deployed via the EGI Federated Cloud OCCI client Virtual CING Cing NMR Protein DB Virtual CING Cing To. Po. S Virtual CING Cing … EGI Federated Cloud Infrastructure EGI-In. SPIRE RI-261323 www. egi. eu

Bio. Ve. L • Services are accessed from the Bio. Ve. L Portal, Taverna Workbench and EUBrazil Open. Bio VRE. Portal • Bio. Ve. L Portal is running on the EGI Federated Cloud • The open. Modeller Web Service interface is integrated into the COMPs Workflow Orchestrator, which, using the VENUS-C middleware, dynamically instantiates open. Modeller workflows. • Bio. STIFF and Google Open. Refine services are also integrated in the EGI Federated Cloud, to be used in the ENM and other workflow. EGI Fed. Cloud Open Modeller Web Service Interface (OMWS 2) COMPSs Workflow Orchestrator VENUS-C Cloud Middleware OCCI EGI Fed. Cloud CDMI Bio. STIFF EGI Federated Cloud EGI-In. SPIRE RI-261323 Refine EGI Fed. Cloud www. egi. eu

ESA HN Flagship on the EGI Federated Cloud The European Space Agency deployed part of the ESA Helix Nebula Flagship into the EGI Federeted Cloud: 1. Use Slip. Stream Infrastructure Broker to deploy a custom computing cluster on multiple Federated Cloud sites (CESNET, GRNET) 2. Connect the computing cluster to the ESA G-POD system and run SAR processing for volcano and earthquake monitoring Automatic Cluster Deployment OCCI BLUE BOX EGI Federated Cloud Computing Cluster Globus Cluster Head Working Nodes Job submission 1/20/2022 EGI-In. SPIRE RI-261323 36 www. egi. eu

EISCAT 3 D Po. C EISCAT_3 D is a three-dimensional imaging radar to be located in the northernmost parts of Europe, to study Earth atmosphere and its coupling to space. The EISCAT 3 D Proof of Concept is a Paa. S built on top of the EGI Federated Cloud for Data Access and Dissemination, based on the ENVRI tools, the EGI Federated Cloud Infrastructure and Storage services and the current EISCAT data. EGI-In. SPIRE RI-261323 37 www. egi. eu

CHAIN-REDS Science Gateway (running GNU Octave and R) CHAIN-REDS Science Gateway -CHAIN-REDS RI-306819 1/20/2022 EGI-In. SPIRE RI-261323 38 38 www. egi. eu

Peachnote is a music score search engine and analysis platform. • Analogue to Google Books for music scores • Leverage on the EGI Federated Cloud for Optical Music Recognition (OMR) processing and image/scores storage OMR Feeder PDF Splitter OMR Worker Peachnote Portal … OMR Worker EGI Federated Cloud 1/20/2022 EGI-In. SPIRE RI-261323 39 www. egi. eu

Value added services for User Communities Technology Evolution • Broader support for open standards in Cloud management frameworks (private and public clouds) • New feature additions to foundational tools depending on requests – Accounting, monitoring, service discovery, Application DB • Innovate and deploy Paa. S & Saa. S • Continual improvements on documentation Strengthening the underpinning platform EGI-In. SPIRE RI-261323 40 www. egi. eu

Conclusions • Paving the way for a global federated cloud marketplace – – – Established best practice Illustrating European leadership Open standards, open technology Open membership, open processes Open competition • Launched to production May ‘ 14 • New communities and resource providers @ ~ 1 per week Launch capability Q 4 2014 (planned) 2020 Vision EGI-In. SPIRE RI-261323 – – – 5, 000 cores, 18, 000 cores, 1, 000 cores, 225 TB storage 6000 TB storage 1 EB storage 41 www. egi. eu

Thank you Questions? EGI-In. SPIRE RI-261323 42 www. egi. eu