EGIIn SPIRE EGI Federated Clouds Capabilities 10282021 EGIIn

EGI-In. SPIRE EGI Federated Clouds Capabilities 10/28/2021 EGI-In. SPIRE RI-261323 1 www. egi. eu

Overview • • • VM/Data Management (Michel) Information Discovery (Peter) Accounting (Alison) Monitoring (Emir) Federated AAI (Bjoern) VM Image sharing (Kostas) 10/28/2021 EGI-In. SPIRE RI-261323 2 www. egi. eu

S 1: VM Management • Well-known interfaces exist (e. g. EC 2) – OCCI published just in time, by OGF • OCCI 1. 1 is now mandatory on Federation level – Additional local interfaces are fine – But: there is no such thing as OCCI 0. 8! • OCCI 1. 1 compliance is monitored • OCCI 1. 1 is actively used in the demo • OCCI 1. 1 support is growing in Open. Source – Often backed by commercial development (e. g. Intel/IBM in Open. Stack) EGI-In. SPIRE RI-261323 www. egi. eu

S 2: Data Management • Many interfaces exist – Data management is not data access • CDMI 1. 0. 1 – Cloud Data Management interface – Allows declaring which Data access protocol to use • Implementations exist (but less than for OCCI) – But OCCI and CDMI interoperate very well – These need to be fed upstream to be included in core Cloud Management solutions EGI-In. SPIRE RI-261323 www. egi. eu

S 3: Information Discovery Why a common Information System for the federated clouds? – To provide enough information to the users to choose the cloud service to use First set of relevant information identified: – Cloud provider information: name of the provider, Country – Cloud service information: Overall size of the resource, VM Manager deployed (e. g. XEN, KVM) interfaces (e. g. OCCI), capabilities (e. g. upload VM Images), dynamic information e. g. : available resources – Information about VM types available: Images already available (O. S. , architecture), max memory, max cores • Not about the single instances (status information available through mgt interfaces like OCCI) GLUE 2. 0 – Information model designed to describe Grid services – There are common attributes between Cloud and Grid services – In order to fully support Cloud services GLUE 2 will likely need to be extended EGI-In. SPIRE RI-261323 www. egi. eu

S 3: Information Discovery Achievements: • Mapped of static information into the GLUE 2. 0 schema – GLUE 2 Entities used: Execution. Environment • Domain, Computing. Service, Computing. Endpoint, Cloud service is described using the entities used for a CE. – Tried to do not alter the schema semantic • Just some tweaks, e. g. : CE implementation/Cloud manager, batch system/VM manager – First deployment of the static set of information published through an LDAP server • Future work: – Extend the GLUE 2 schema with Cloud-specific entities • Model the storage and network resources • Propose the extensions to OGF and have them officially included in a future GLUE version – Discuss how the resource information provider should be implemented – LDAP could not be the best protocol for a cloud information system, there are many reason to use it, but. . EGI-In. SPIRE RI-261323 www. egi. eu

S 4: Accounting • Answering these questions: – Which resources need to be accounted for? – What’s their accounting unit? – What’s the metering frequency? • Achievements – Initial definition of a Cloud Usage Record (CUR) • 4 Resource Providers have provided accounting data – Re-uses existing EGI APEL infrastructure • EGI Active. MQ Message Broker • Single message may contain 1000’s of CURs • Messages are encrypted and signed EGI-In. SPIRE RI-261323 www. egi. eu

S 4: Accounting • Next steps – Review integration with Cloud Management services (Open. Nebula, Open. Stack, etc. ) – Which accounting questions need answering • E. g. billing questions, VO orientated questions – How do we want to pre-aggregate the data? EGI-In. SPIRE RI-261323 www. egi. eu

S 5: Monitoring • Plain Nagios “box” – Virtual Machine provided and operated by CESGA – Future integration with EGI SAM infrastructure – https: //test 29. egi. cesga. es/nagios/ • Auth. N requires valid IGTF certificate • Probes – basic HTTP(S) checks • OCCI 0. 8, OCCI 1. 1, Web interface – OCCI probe • OCCI 1. 1 VM Test • Creates VM, waits for machine to start, destroys the machine • Developed by Piotr Kasprzak (GWDG) EGI-In. SPIRE RI-261323 www. egi. eu

S 5: Monitoring EGI-In. SPIRE RI-261323 www. egi. eu

S 7: Federated AAI • Authentication and authorization at infrastructure level – Out of scope: AAI at platform or service level • Assumptions – „Few“ infrastructure operators (~100 s) as compared to platform users (~1000 s) – Common credential mechanisms, i. e. X 509 certificates – Federated policies EGI-In. SPIRE RI-261323 www. egi. eu

S 7: Federated AAI • Policy federation – provider autonomy • Cloud Mgmt stacks are PEPs – need common implementations • PIP – need common information profile EGI-In. SPIRE RI-261323 www. egi. eu

S 7: Federated AAI • Policy federation EGI-In. SPIRE RI-261323 www. egi. eu

S 8: VM Image sharing • One Marketplace service advertising VM Image metadata – http: //marketplace. egi. eu • Many Appliance repositories host actual VM image disks – One instance: https: //appliance-repo. egi. eu/images/ EGI-In. SPIRE RI-261323 www. egi. eu

S 8: VM Image sharing • Next steps – Integrate VM Marketplace with EGI SA 2 activity • SA 2 as Platform packager for existing Grid M/W – M: N relationships • M endorsers sign metadata sets of n VM images – Explore metadata extension for security endorsement levels • Not every endorser endorses on the same level EGI-In. SPIRE RI-261323 www. egi. eu