EGIIn SPIRE EGI Federated Cloud Standards Adoption Roadmap

EGI-In. SPIRE EGI Federated Cloud: Standards Adoption Roadmap Michel Drescher, Technical Manager, EGI. eu 10/15/2021 EGI-In. SPIRE RI-261323 ISGC 2012 - 29 February 2012 Standards Adoption track 1 www. egi. eu

EGI Federated Cloud TF • Building on prior experience – Standards critical for open federation • Interoperable cloud standards exist now – Ahead of grids at a comparable time • Federation driven by public sector concerns – EGI’s experience in grids is a unique asset • Re-use existing technology – Builds on a decade of experience from EGI 10/15/2021 EGI-In. SPIRE RI-261323 ISGC 2012 - 29 February 2012 Standards Adoption track 2 www. egi. eu

Task Force Objectives • Integration: – Integrate virtualised resources into EGI • End-user requirements: – Elicit requirements for customer facing services • Technical feedback: – To technology providers on gaps and issues • Early adopters: – Identify and work with interested research groups • Recommendations: – For policies, operations, support and dissemination EGI-In. SPIRE RI-261323 ISGC 2012 - 29 February 2012 Standards Adoption track 3 www. egi. eu

Community Engagement SARA Utrecht DANTE EGI. eu Oer. C Six. Sq CNRS FCTSG IFAE BSC 10/15/2021 EGI-In. SPIRE RI-261323 KTH GWDG TCD STFC TUD FZJ • • Diverse Stakeholders Community Driven 15 Resource Providers 58 individuals 7 Technology Providers 23 institutions 4 User Communities 13 countries 3 Liaisons • • • Multiple Technologies 7 Open. Nebula 3 Stratus. Lab 3 Open. Stack 1 Okeanos 1 WNo. De. S ISGC 2012 - 29 February 2012 Standards Adoption track CESNET Cyfronet Masaryk LMU SRCE INFN GRNET 4 www. egi. eu

Functional Capabilities • 6 Infrastructure Capabilities – VM Management – Data Management – Information Discovery – Accounting – Monitoring – Notification 10/15/2021 EGI-In. SPIRE RI-261323 • 2 Auxiliary Capabilities – Federated AAI – VM Image Sharing ISGC 2012 - 29 February 2012 Standards Adoption track 5 www. egi. eu

EGI’s Cloud Strategy Globus ARC g. Lite d. Cache VM Mgmt Iaa. S Community Services Data Information VM Mgmt Accounting Information VM Mgmt UNICORE Data Information NGI Federated Iaa. S NGI Monitoring Data Globus NGI Notification Monitoring Accounting Notification Monitoring Community Platform Accounting VM Mgmt Data Information Commercial Notification Monitoring Accounting Notification Core Software & Support Community Services EGI. eu Coordination Paa. S Saa. S Personalised environments for individual research communities in the European Research Area EGI-wide message bus 10/15/2021 EGI-In. SPIRE RI-261323 ISGC 2012 - 29 February 2012 Standards Adoption track 6 www. egi. eu

Platform Architecture • • • Cloud Management solutions for virtualised resources Support service(s) for provider selection Support services for Integration automation Loose coupling of system components reduces dependencies Monitored for Infrastructure management Some services may have federation counterparts API Virtual Compute Virtual Storage Web UI Messaging Virtual Network Accounting Notification Fed. Cloud Management (Compute, Storage, Network) Fed. EGI Messaging Infrastructure Monitoring 10/15/2021 EGI-In. SPIRE RI-261323 Information Discovery ISGC 2012 - 29 February 2012 Standards Adoption track Fed. 7 www. egi. eu

Design Principles • Exploit Standards wherever possible – Critical for federation success – Captures functionality without prescribing technology – E. g. OCCI 1. 1, CDMI 1. 0. 1 • Extend standards where applicable – Standard captures most of the needs – Reasonable extensions provide “the last 10%” – E. g. extending UR for Cloud accounting • Re-use existing technology where necessary – Standards may be insufficient or non-existent – Requires consent across all federation members – E. g. VOMS, BDII, ARGUS, APEL • Open Source Technology reduces financial barriers 10/15/2021 EGI-In. SPIRE RI-261323 ISGC 2012 - 29 February 2012 Standards Adoption track 8 www. egi. eu

Interfaces for Federation OCCI 1. 1 X 509 (SAML & XACML) CDMI 1. 0 OVF 1. 1. 0 GLUE 2. 0 (LDAP) Open. Nebula, Open. Stack, Eucluptus Nagios OGF UR UR+ & St. AR Various Active. MQ 10/15/2021 EGI-In. SPIRE RI-261323 ISGC 2012 - 29 February 2012 Standards Adoption track 9 www. egi. eu

Standards roadmap phases • Research A standard does not exist or is not known. If known, individuals research its potential use for the Federated Cloud Infrastructure. • Proposed A standard is proposed to the Task Force for discussion and convergence. Through networking internal and external experts are consulted. • Investigated An investigation looks at the details of the proposed standard, and at product roadmaps for planned support of the respective standard. • Verified Available technology is deployed in the test-bed and the standard support is verified before it is taken further. • Compulsory A respective standard is required to be supported by any deployed technology This does not exclude the support of other interfaces or standards. 10/15/2021 EGI-In. SPIRE RI-261323 ISGC 2012 - 29 February 2012 Standards Adoption track 10 www. egi. eu

1. VM Management • Available standards: – OCCI 1. 1 • Known OS technology – Open. Nebula, Open. Stack, Eucalyptus, … • Status: – OCCI 1. 1 compulsory 10/15/2021 EGI-In. SPIRE RI-261323 ISGC 2012 - 29 February 2012 Standards Adoption track 11 www. egi. eu

2. Data Management • Available standards: – CDMI 1. 0. 1 for storage management – OVF 1. 1. 0 for VM based appliance packaging – Many for storage access • Known OS technology – Open. Nebula, Open. Stack extensions underway • Status: – CDMI 1. 0. 1 investigated/verified – OVF 1. 1. 0 investigated (but likely to become compulsory) – Let the market decide on access protocols • Great flexibility within admin domains • Reduce to well-established protocols across boundaries 10/15/2021 EGI-In. SPIRE RI-261323 ISGC 2012 - 29 February 2012 Standards Adoption track 12 www. egi. eu

3. Information Discovery • Available standards: – GLUE 2+/LDAP/LDIF for hierarchical information modelling – RDF/SPARQL for relational information modelling • Known OS technology – BDII/GOCDB deployed in EGI use LDAP/GLUE 2 • LDAP/LDIF established, GLUE used in Grids only(? ) – Many for RDF/SPARQL • http: //www. w 3. org/wiki/Sparql. Implementations • Status: – LDAP/GLUE 2 based solution investigated – RDF/SPARQL researched 10/15/2021 EGI-In. SPIRE RI-261323 ISGC 2012 - 29 February 2012 Standards Adoption track 13 www. egi. eu

4. Accounting • Available standards: – OGF UR for compute related accounting – OGF St. AR in public comment for Storage related accounting – AMQP for messaging networks • Known OS technology – APEL deployed in EGI, uses UR – Active. MQ, Rabbit. MQ, etc. for messaging networks • Status: – – OGF UR confirmed UR+ for Cloud computing currently defined within EGI AMQP researched De-facto solution for messaging: Active. MQ 10/15/2021 EGI-In. SPIRE RI-261323 ISGC 2012 - 29 February 2012 Standards Adoption track 14 www. egi. eu

5. Monitoring • Available standards: – Unknown. – Nagios de-facto industry standard • Known OS technology – Nagios with commercial support, used world-wide. – Ganglia developed and used within the HEP community • Status: – Nagios for federation level monitoring – Free choice for internal deployments 10/15/2021 EGI-In. SPIRE RI-261323 ISGC 2012 - 29 February 2012 Standards Adoption track 15 www. egi. eu

6. Notification • Available standards: – For end-users (prose based) • E-Mail (SMTP, POP, IMAP), • RSS/ATOM, • XMPP – For application integration (machine-readable content) • RSS/ATOM, • XMPP • Messaging integrations • Known OS technology – Myriads of libraries and clients for Email, RSS, ATOM, XMPP • Status: – All mentioned are researched – Still unclear what exactly notification will entail 10/15/2021 EGI-In. SPIRE RI-261323 ISGC 2012 - 29 February 2012 Standards Adoption track 16 www. egi. eu

7. Federated AAI • Available standards: – X. 509 certificates for authentication federation – SAML, XACML enabling technologies for federated AAIs • Known OS technology – X. 509 is ubiquitous in libraries, tools and processes – XACML 2. 0: VOMS, ARGUS, Apache Axis 2 • Status: – X. 509 compulsory for Platform operators (on top of EGI Clouds) – SAML, XACML proposed for full Platform operator federation – Research Community level federated AAI: Out of scope! 10/15/2021 EGI-In. SPIRE RI-261323 ISGC 2012 - 29 February 2012 Standards Adoption track 17 www. egi. eu

8. VM Image sharing • Available standards: – – – RDF/XML for VM Image metadata SPARQL for VM Image queries UUIDs for VM Image ids HTTP/Web. DAV, … for VM image repository access CDMI for VM repository management • Known OS technology – Large tool-base for RDF/SPARQL – UUIDs, HTTP, Web. DAV ubiquitous • Status: – RDF/XML, SPARQL, UUID verified for VM Image metadata access – HTTP/Web. DAV verified for VM image repository – CDMI proposed for VM Image repository management 10/15/2021 EGI-In. SPIRE RI-261323 ISGC 2012 - 29 February 2012 Standards Adoption track 18 www. egi. eu

Summary Capability Standard Status VM Management OCCI 1. 1 Compulsory Data Management CDMI 1. 0. 1 OVF 1. 1. 0 Investigated/Verified Investigated Information Discovery LDP 3/GLUE 2 RDF/SPARQL Investigated Researched Accounting UR, UR+ St. AR Compulsory, in development Monitoring n/a No standard in sight for us Notification Email, RSS/ATOM, XMPP All researched Federated AAI X. 509, SAML/XACML Compulsory, proposed as next step VM Image sharing RDF/SPARQL HTTP/Web. DAV CDMI Verified Proposed 10/15/2021 EGI-In. SPIRE RI-261323 ISGC 2012 - 29 February 2012 Standards Adoption track 19 www. egi. eu

10/15/2021 EGI-In. SPIRE RI-261323 ISGC 2012 - 29 February 2012 Standards Adoption track 20 www. egi. eu