Exploiting Federated Cloud resources Themis Athanassiadou Technical Outreach

  • Slides: 20
Download presentation
Exploiting Federated Cloud resources. Themis Athanassiadou Technical Outreach User Community Support Team Themis. athanassiadou@egi.

Exploiting Federated Cloud resources. Themis Athanassiadou Technical Outreach User Community Support Team Themis. athanassiadou@egi. eu http: //go. egi. eu/cloud www. egi. eu EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142

Introduction to EGI & EGI Federated Cloud 2/24/2021 2

Introduction to EGI & EGI Federated Cloud 2/24/2021 2

EGI Foundation 2/24/2021 Introduction to EGI 3

EGI Foundation 2/24/2021 Introduction to EGI 3

The current cloud infrastructure Today: • 24 providers from 14 NGIs • 17 Open.

The current cloud infrastructure Today: • 24 providers from 14 NGIs • 17 Open. Stack • 6 Open. Nebula • 1 Synnefo • ~6. 000 cores in total 2/24/2021 4

What is a cloud federation? – A ‘definition’ • Practice of interconnecting cloud service

What is a cloud federation? – A ‘definition’ • Practice of interconnecting cloud service providers. Motivations: – Data locality; Data privacy; Shared investment; Distributed expertise • Multiple cloud sites with some sort of interconnection(s). Examples: – – – Every cloud registered in a single catalogue Single VM image catalogue for users Support for the same image format Automated distribution of VM Images to the federated clouds Single sign-on for users Harmonised operational practices • Cloud configurations, integrated monitoring, accounting, etc. – Integrated support model • Ticketing system, consultancy, training 2/24/2021 5

EGI Federated Cloud • Grid of clouds • Unified user interfaces • Harmonised operational

EGI Federated Cloud • Grid of clouds • Unified user interfaces • Harmonised operational behaviour • Clouds and their interconnections are based on open standards, open technologies • Infrastructure Access AND technology Deploy 2/24/2021 6

Benefits VM and block storage management: Standard-based federation Open. Stack federation Uniform user interfaces

Benefits VM and block storage management: Standard-based federation Open. Stack federation Uniform user interfaces Object storage management (optional): - On every site CDMI - on any site • Open. Stack SWIFT – on OS sites Open. Stack Nova - On OS sites Open. Stack Open. Nebula Synnefo Open. Stack Open. Nebula 2/24/2021 Harmonised operation Cloud registry Information system Virt. Machine marketpl. Usage accounting Access control Open. Stack 7

Resource allocation in Federated Cloud: Virtual Organisations VO 1 (cloud a, b, c) VO

Resource allocation in Federated Cloud: Virtual Organisations VO 1 (cloud a, b, c) VO 2 (cloud b, c, d, e) 1. Generic VOs – e. g. fedcloud. egi. eu Incubator for new users 2. Community-specific VOs – e. g. CHIPSTER, Highthroughtputseq, EISCAT, etc. (SLA, OLAs) 3. Training VO = training. egi. eu Browse VOs at http: //operations-portal. egi. eu/vo/search (both grid and cloud) 2/24/2021 9

The typical user workflow Clouds in your Virtual Organisation (e. g. training. egi. eu)

The typical user workflow Clouds in your Virtual Organisation (e. g. training. egi. eu) Application Portal, Saa. S, etc. . Visual lookup Programmatic lookup (API) OCCI or Nova calls (CMD/API) VM VM VM Virtual/Software Appliances of your Virtual Organisation VM VM VM Storage VM Appliances Marketplace (App. DB) 2/24/2021 10

Example: Chipster VM NFS Server Tools Volume 2/24/2021 Data Volume EGI Fed. Cloud Resource

Example: Chipster VM NFS Server Tools Volume 2/24/2021 Data Volume EGI Fed. Cloud Resource Provider Analysis software contains over 300 analysis tools for NGS, microarray, proteomics and sequence data. Usage Model Scientific Disciplines Deployment in the Fed. Cloud • Web service • Heavy computation and large memory • Manage large datasets • Bioinformatics • Complex deployment through contextualisation • shared block storage exported as NFS up to 1 TB 13

Strategic data replication and cloud access: ELIXIR/EMBL-EBI for life sciences • • Goal: Offload

Strategic data replication and cloud access: ELIXIR/EMBL-EBI for life sciences • • Goal: Offload user traffic from EBI cloud and DBs Solution: • • Setup federated cloud at strategic partners Replication of data and application to partners Users ‘compute locally’ Using from EGI: – – Cloud compute (at sites) Operations services Security services (EGI Check. In) VM Catalogue (App. DB) • Infra setup with 5 scientific applications are under development Further information: https: //wiki. egi. eu/wiki/CC-ELIXIR 2/24/2021 • Service registry • Service monitoring • Federated user access (w. ELIXIR acc. ) • Accounting EMBL-EBI strategic partners Iaa. S cloud Applications EMBL-EBI Mount Replica storage Embassy cloud Iaa. S cloud Curated datasets Datasets Mount Replica storage 14

Next steps 2/24/2021 15

Next steps 2/24/2021 15

The next steps: Roadmap Highlights: – Improve the reliability of federation layer with enhancement

The next steps: Roadmap Highlights: – Improve the reliability of federation layer with enhancement of monitoring and the integration components (AAI, information discovery, accounting, VM marketplace) – Promote Iaa. S Federated Access Tools Layer as interoperation mechanism – Identification of services beyond Iaa. S to expand the portfolio • Check slides from roadmap session – https: //indico. egi. eu/indico/event/3249/session/33 2/24/2021 16

To learn more: EGI Federated Cloud Documentations and Guides: • https: //wiki. egi. eu/wiki/Federated_Cloud_user_support

To learn more: EGI Federated Cloud Documentations and Guides: • https: //wiki. egi. eu/wiki/Federated_Cloud_user_support 2/24/2021 17

Thank you for your attention. Questions? Consult with http: //go. egi. eu/cloud Email support@egi.

Thank you for your attention. Questions? Consult with http: //go. egi. eu/cloud Email support@egi. eu www. egi. eu This work by Parties of the EGI-Engage Consortium is licensed under a Creative Commons Attribution 4. 0 International License.

Getting access to the Fed. Cloud Obtain certificate: Once Renew certificate: Annually Your steplist:

Getting access to the Fed. Cloud Obtain certificate: Once Renew certificate: Annually Your steplist: 1. Obtain certificate from National CA (face-to-face identity check) http: //www. igtf. net OR Terena Certificate Service: (online) https: //www. digicert. com/sso 2. Register at the VO • fedcloud. egi. eu is a good starting point You CA Join VO: Once Use resources • Other VOs: http: //operations- VO manager Register Membership service portal. egi. eu/vo/search 3. VO manager authorizes You User database • Membership DB updated • Identity replicated to resource within 1 day 4. Interact with the resorurces • r. OCCI • API • High-level tool 2/24/2021 Cloud sites DB replication (once a day) VIRTUAL ORGANISATION 19 19

App. DB VMOps Dashboard https: //dashboard. appdb. egi. eu/vmops • Unified resource management on

App. DB VMOps Dashboard https: //dashboard. appdb. egi. eu/vmops • Unified resource management on the Federated Cloud • Complete EGI Check. In integration • Personalized, wizard-like, topology builder: – Manage multiple-VM topologies with associated storage and contextualization – Fine grained control of VMs within topologies 2/24/2021 20

Support services Dedicated technical consultancy for any user or community: support@egi. eu F 2

Support services Dedicated technical consultancy for any user or community: support@egi. eu F 2 F/Web Meetings • Identify suitable setup • Allocate technical experts • Define milestones Continuous tracking and support • Technical integration • Periodic meetings Fedcloud. eg i. eu VO • Resources for prototyping • Enabled on all sites • Usable for 2 x 6 months 2/24/2021 Doc EGI VM Images Migration to production • Step by step guides • Tutorials – CMD line, API • Examples • • Main OS versions Secure, up-to-date Contextualisation Docker • Identifying committed resource providers • Support for VO setup • SLAs, OLAs 21

Support through the NGIs http: //www. egi. eu/about/ngis/ • EGI’s federated support model –

Support through the NGIs http: //www. egi. eu/about/ngis/ • EGI’s federated support model – National support teams (NGIs) – Topic/discipline-specific support teams (see next slide) – EGI. eu UCST – primarily coordination & support for supporters (support@egi. eu) To use resources To federate resources 2/24/2021 22

Which approach to follow? Contextualization Docker Custom Images Medium complexity. • Needs Docker Engine

Which approach to follow? Contextualization Docker Custom Images Medium complexity. • Needs Docker Engine Configuration on creation • Fast VM start-up, separate • application start-up • Works on top of existing • Works on Docker-enabled • images • Debug in local • • Hard to debug if fails environment • Fast to use, just add user • data to VM • Configuration on creation • • • Slow start-up of VM Time consuming, needs virtualization software Static configuration Fast start-up of VM Requires moving large f to sites Easier to debug • EGI. eu maintains core VM images in App. DB. • These can be used as starting point in all three scenarios. 2/24/2021 24