EGIIn SPIRE EGI Federated Clouds Task Force Incubating

EGI-In. SPIRE EGI Federated Clouds Task Force Incubating the EGI Cloud federation providing a pan-European e-Infrastructure Michel Drescher, EGI. eu Matteo Turilli, Oer. C 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 1 www. egi. eu

Outline • From Grids to Clouds • Federated Infrastructure as a service • The EGI Federated Clouds Task Force 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 2 www. egi. eu

From Grids to Clouds A retrospective glance on Grid architecture and its alignment with Cloud computing 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 3 www. egi. eu

Current Grid Architecture User Interface EGI Grid Middleware Infrastructure 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 4 www. egi. eu

3 -tiered Architecture Client Business Persistence 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 5 www. egi. eu

3 -tiered Architecture Client Business Client Business Persistence 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 6 www. egi. eu

3 -tiered Grid Architecture UI M/W UI Middleware Infrastructure 1/8/2022 EGI-In. SPIRE RI-261323 UI 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 EGI UI 7 www. egi. eu

A future EGI Cloud? Globus g. Lite d. Cache ARC Community Services Globus UNICORE Community Platform Federated Virtual Machine Managers NGI 1/8/2022 EGI-In. SPIRE RI-261323 NGI EIRO Amazon 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 Microsoft Core Software & Support Community Services EGI. eu Coordination Virtual Machines Individual communities get the environments they require 8 www. egi. eu

Virtual Machines A future EGI Cloud? Community Services POTS Services Community Services Saa. S UNICORE Community Platform Paa. S Globus d. Cache VM Mgmt ARC g. Lite Data Information VM Mgmt Accounting Information VM Mgmt NGI Monitoring Data Notification Monitoring Accounting Data Information VM Mgmt NGI Notification Monitoring Accounting Data Information Amazon Notification Monitoring Accounting Notification Iaa. S EGI-wide message bus 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 9 www. egi. eu

Example service delivery VRC Community Service Saa. S NGI Globus Paa. S ARC g. Lite d. Cache NGI Community Platform NGI VM Mgmt Iaa. S Community Services POTS Services Data Information VM Mgmt RP / NGI Monitoring Accounting Data Information VM Mgmt RP / NGI Notification Monitoring Accounting Data Information RP / NGI Notification Monitoring Accounting Notification EGI-wide message bus 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 10 www. egi. eu

Federated Infrastructure as a Service A common infrastructure service underpinning value-added user services 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 11 www. egi. eu

Federated Clouds TF • Explore technical aspects of hybrid Clouds – Elicit user community requirements – Early adopters using the Cloud federation – Gap analysis & technical feedback to technology providers – Recommendations for other EGI areas – Integrate with EGI production infrastructure 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 12 www. egi. eu

Federated Cloud Scenarios Six Scenarios capture core User requirements 1. 2. 3. 4. 5. 6. “Running a pre-defined VM Image” “Running my data and VM in the Infrastructure” “Integrating multiple resource providers” “Accounting across Resource Providers” “Reliability/Availability of Resource Providers” “VM/Resource state change notification” 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 13 www. egi. eu

Federated Cloud Capabilities Six Capabilities define the EGI Cloud Profile Functionality required for uniform & integrated federated Cloud Computing 1. 2. 3. 4. 5. 6. VM Management Data access Information (Discovery) Accounting Monitoring Notification 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 14 www. egi. eu

Capabilities & standards Capability Standard(s) Status Comments VM Management OCCI 1. 1 Confirmed Interop testing underway Data management CDMI 1. 0. 1 OVF 1. 1. 0 Proposed Information Discovery LDAP 3/LDIF, GLUE 2+ Proposed Extensions to GLUE 2 are expected Accounting UR+, APEL based Proposed UR as basis agreed, extensions discussed. APEL Monitoring Nagios, Nagios plugins Confirmed Strict external interface testing. Notification Messaging based Research 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 15 www. egi. eu

The EGI Clouds Federation Task Force Laying down the foundation for the future EGI. eu cloud infrastructure 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 16 www. egi. eu

Goals • Blueprint document: advice/full documentation to resource providers/users on how to engage with the federated virtualised environment. This will be a living document on the EGI Wiki. • Test bed: implementation of the federation blue print by deploying the state-of-the-art of the available technologies. • Engagement: identification of resources centres and user communities willing to commit actively to this Task Force. • Reports: regular reports to the TCB on progress including the identification of issues that need to be considered elsewhere within EGI. • Dissemination: periodic “success stories” through dissemination channels. 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 17 www. egi. eu

Membership SARA INFN/CNAF IN 2 P 3 UK NGI Cloud. Sigma CESGA EGI. eu OGF IPHC GRIF KTH Stratus. Lab CESNET Venus-c SIENA We. NMR EGI-In. SPIRE Cyfronet FZJ STFC GWDG DANTE GWDG GRNET IGI TCD 52 members from 24 institutions across all Europe including 14 Resource Providers, 6 Technology Providers and 3 Users Communities and 2 Liaisons. 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 18 www. egi. eu

Blueprint 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 19 www. egi. eu

Test Bed 15 Resource Providers ■ 7 Open. Nebula ■ 4 Stratus. Lab ■ 3 Open. Stack ■ 1 WNo. De. S ■ 1 Cloud. Sigma • 5 cloud management technologies. • 70% Open. Nebula/Stratus. Lab. • Multiple management interfaces: OCCI, EC 2, custom APIs, Jclouds-compatible APIs. • Multiple storage and data-management technologies. 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 20 www. egi. eu

Thank you Questions? 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 21 www. egi. eu

BACKUP 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 22 www. egi. eu

Scenario 1 “Running a pre-defined VM Image” 1. 2. 3. VM Image known and accessible VM management commands: start, stop, dispose Federated AAI restricts access to the running instance 4. Services offering – – VM-based computing AAI infrastructure 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 23 www. egi. eu

Scenario 2 “Running my data and VM in the Infrastructure” 1. 2. 3. 4. VM Image is generated off-line by the user VM Image upload, snapshot download Data resident with provider Runtime link to data by configuration 5. Services – – Generic storage VM image storage/cache 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 24 www. egi. eu

Scenario 3 “Integrating multiple resource providers” 1. 2. 3. 4. Many resource providers Federated AAI across Resource providers at resource management level Comparable service descriptions Integration at user level 5. Services – Information service 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 25 www. egi. eu

Scenario 4 “Accounting across Resource Providers” 1. 2. 3. Unique user identification across resource providers Resource usage is accounted Shared resource accounting profile 4. Services – Accounting 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 26 www. egi. eu

Scenario 5 “Reliability/Availability of Resource Providers” 1. 2. 3. Provider reliability and availability statistics Historic accounts, and current figures Regular publication of statistics 4. Services – – Monitoring Reporting 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 27 www. egi. eu

Scenario 6 “VM/Resource state change notification” 1. 2. 3. Reactive/proactive management VM Instance state change management Real-time notification constraints 4. Services – Notification/Feedback 1/8/2022 EGI-In. SPIRE RI-261323 7 th IEEE e. Science conference, Stockholm, 7 -12 -2011 28 www. egi. eu