EGI Advanced Computing for Research www egi eu

EGI: Advanced Computing for Research www. egi. eu @EGI_e. Infra Create elastic virtual clusters in the EGI Cloud with the EC 3/IM Giuseppe La Rocca Community Support Team Lead. giuseppe. larocca@egi. eu The work of the EGI Foundation is partly funded by the European Commission under H 2020 Framework Programme Agenda: https: //indico. egi. eu/event/5092/

Today’s Speaker Background: ● MSc in Computer Science Engineering from the University of Catania (Italy) ● 16 years experience in EC-funded projects At EGI Foundation: ● Giuseppe La Rocca works as Community Team Lead. at the EGI Foundation. ● Chief Community Officer of the EGI. eu User Community Board (UCB) group ● WP 11 Manager (Training) of the EOSC-hub project ● Holds a Fit. SM Advanced SOC and SPD, and ISO/IEC 27 K Foundation certifications www. egi. eu @EGI_e. Infra 22/04/2020 2

EGI Webinar Programme 2020 https: //www. egi. eu/webinars/ www. egi. eu @EGI_e. Infra 22/04/2020 3

Video Recording will be available on the EGI channel/agenda after the event www. egi. eu @EGI_e. Infra 22/04/2020 4

Ground Rules Please write your questions in the Q&A window of the Zoom application or raise your hands before to speak www. egi. eu @EGI_e. Infra 22/04/2020 5

Create elastic virtual clusters in the EGI Cloud with the EC 3/IM www. egi. eu @EGI_e. Infra 22/04/2020 6

Outline • The EC 3 (Elastic Cloud Compute Cluster) in a nutshell ▪ Features, Components and Architecture ▪ Different deployment models • EC 3 aa. S web interface • The EC 3 CLI • EC 3 in Docker Hub • Live demo ▪ Launch an elastic cluster in the EGI Cloud • References & Documentations • Q&A www. egi. eu @EGI_e. Infra 22/04/2020 7

Elastic Cloud Computing Cluster (EC 3) is a tool to deploy virtual elastic clusters on top of Infrastructure as a Service (Iaa. S) providers, either public (such as Amazon Web Services, Google Cloud or Microsoft Azure) or on-premises (such as Open. Nebula and Open. Stack) EC 3 has been developed by the Grid and High Performance Computing Group (GRy. CAP) at the Instituto de Instrumentación para Imagen Molecular (I 3 M) from the Universitat Politècnica de València (UPV) www. egi. eu @EGI_e. Infra 22/04/2020 8

EC 3 main features Facilitates the access to cloud computing platforms to non-experienced users • Maintains the “traditional” work environment: • ▪ Clusters configured with a LRMS and scientific applications • Automates elasticity management (based on EC 3/CLUES) • Supports for a wide range of cloud providers: • Supports for a wide list of profiles and templates to configure the clusters: ▪ Automatic configuration of the execution environment ▪ public, federated and on- premises ▪ hybrid infrastructures ▪ Applications: Galaxy, NAMD ▪ Tools & frameworks: ECAS, OSCAR, SLURM, Kubernetes, . . . www. egi. eu @EGI_e. Infra 22/04/2020 9

EC 3 main components • EC 3 deploys and configures virtual elastic clusters. It relies on IM to deploy the machines and on CLUES to automatically manage the elasticity. CLUES IM • Offers a set of predefined templates to configure the resources through Ansible: • Kubernetes, Mesos, SLURM, Torque, SGE, HTCondor, Nomad. The Infrastructure Manager (IM) Webinar on EC 3 Ansible recipes www. egi. eu @EGI_e. Infra 22/04/2020 10

EC 3 overall architecture www. egi. eu @EGI_e. Infra 22/04/2020 11

Different deployment models ● Homogeneous clusters ● Heterogeneous clusters ● Cloud bursting (hybrid clusters) www. egi. eu @EGI_e. Infra 22/04/2020 12

Homogeneous cluster ● An homogeneous cluster is composed by working nodes that have the same characteristics (hardware and software). www. egi. eu @EGI_e. Infra 22/04/2020 13

Heterogeneous cluster ● This model allows that the working nodes comprising the cluster can be of different characteristics (hardware and software) ○ It also allows you to configure several queues and specify from which queue the working node belongs to www. egi. eu @EGI_e. Infra 22/04/2020 14

Cloud bursting (Hybrid clusters) ● It consists on launching nodes in two or more different Cloud providers ○ When a limit is reached and no more nodes can be deployed inside the first Cloud Provider, EC 3 will launch new nodes in the second defined Cloud provider www. egi. eu @EGI_e. Infra 22/04/2020 15

EC 3 aa. S - Web Interface EC 3 aa. S offers user-friendly web interface that allows to easily deploy and configure a virtual elastic cluster on several cloud providers, including EGI Fed. Cloud www. egi. eu @EGI_e. Infra 22/04/2020 16

The web interface ● EC 3 as a Service (EC 3 aa. S), is a web service offered to the community to facilitate the usage of EC 3 to non-experienced users ● Access link (through the EGI Marketplace): ○ https: //marketplace. egi. eu/42 -applications-on-demand-beta www. egi. eu @EGI_e. Infra 22/04/2020 17

The web interface (cont. ) ● A dedicated wizard will guide the user during the configuration process of the cluster 1. ) Choose the Local Resource Management System that can be automatically installed and configured in the elastic virtual cluster 2. ) Select from the list the endpoint of the providers where to deploy the elastic cluster. www. egi. eu @EGI_e. Infra 22/04/2020 18

The web interface (cont. ) ● A dedicated wizard will guide the user during the configuration process of the cluster 3. ) Configure the cluster selecting the base OS images and in the v. CPU cores and RAM to allocate for the front-end and the working nodes. 4. ) Specify the cluster name and the maximum number of nodes of the cluster. www. egi. eu @EGI_e. Infra 22/04/2020 19

The EC 3 command line interface www. egi. eu @EGI_e. Infra 22/04/2020 20

The EC 3 CLI • More powerful client interface than the Web interface: • More control over the cluster (reconfigure, clone, migrate, stop, restart). • Support for hybrid clusters • Support for golden images • The user needs to define an authorization file • Documentation: https: //ec 3. readthedocs. io/en/latest/ec 3. html • EC 3 Client Source Code in Git. Hub: https: //github. com/grycap/ec 3 • EC 3 Client image in Docker Hub: https: //hub. docker. com/r/grycap/ec 3/ www. egi. eu @EGI_e. Infra 22/04/2020 21

The EC 3 CLI Usage: ec 3 [-h] [-v] [-l LOG_FILE] [-ll LOG_LEVEL] [-q] {launch, list, show, templates, ssh, reconfigure, destroy, clone, migrate, stop, restart} Operation: launch a new cluster list launched clusters show print RADL templates list available templates ssh connect to cluster via SSH reconfigure the cluster destroy a launched cluster clone a launched cluster in another Cloud provider migrate a launched cluster to another Cloud provider stop a launched cluster restart a previously stopped cluster www. egi. eu @EGI_e. Infra 22/04/2020 22

Installation of the EC 3 CLI Pre-requisites ● EC 3 requires: Python 2. 6+, PLY, Py. YAML, requests, jsonschema and, ● IM server which is used to launch the virtual machines ○ By default EC 3 uses the public IM server in appsgrycap. i 3 m. upv. es ○ Optionally, it is possible to deploy a local IM server following the instructions of the IM manual Installing (from repo) ● In Debian/Ubuntu based distributions: ]$ sudo apt update ]$ sudo apt install python-pip Installing (from git repo) ]$ git clone https: //github. com/grycap/ec 3 ]$ sudo pip install. /ec 3 ● In Red. Hat based distributions: ]$ sudo yum install epel-release ]$ sudo yum install which python-pip ● Install via pip: ]$ sudo pip install ec 3 -cli www. egi. eu @EGI_e. Infra 22/04/2020 23

Launch a basic cluster (in Amazon EC 2) Authorization file ● Create an auth. txt file providing the Access/Secret keys Operate a simple cluster ● Deploy a TORQUE cluster based on an Ubuntu image: ● Listing deployed clusters: www. egi. eu @EGI_e. Infra 22/04/2020 24

Launch a basic cluster (cont. ) Operate the cluster ● Open a ssh session to the front-end (you may need to install the sshpass library): ● Submit jobs in the queue: www. egi. eu @EGI_e. Infra 22/04/2020 25

EC 3 in Docker Hub www. egi. eu @EGI_e. Infra 22/04/2020 26

Launch a basic cluster with Docker Install the Docker container ● EC 3 has an official Docker container image available in Docker Hub that can be used instead of installing the CLI Operate your cluster ● Launch a cluster: ● Listing, Access and Destroy the cluster: www. egi. eu @EGI_e. Infra 22/04/2020 27

Available EC 3 templates www. egi. eu @EGI_e. Infra 22/04/2020 28

EC 3 images ]$ sudo docker run grycap/ec 3 templates | grep -i images centos-ec 2 images Cent. OS 6. 5 amd 64 on EC 2 ubuntu-azure images Ubuntu 12. 04 amd 64 on Azure ubuntu-ec 2 images Ubuntu 14. 04 amd 64 on EC 2 ubuntu-fbw images Ubuntu 16. 04 amd 64 on Fog. Bow ubuntu-gce images Ubuntu 14. 04 amd 64 on Google Cloud ubuntu-vmrc images Ubuntu 14. 04 to search on the VMRC ubuntu-hybrid-spot-ec 2 images Example of hybrid cluster with regular and spot instances on EC 2 www. egi. eu @EGI_e. Infra 22/04/2020 29

EC 3 main ]$ sudo docker run grycap/ec 3 templates | grep -i main htcondor main Install and configure a cluster HTCondor from distribution repositories kubernetes main Install and configure a cluster using the grycap. kubernetes ansible role mesos main Install and configure a Mesos cluster from mesosphere repo sge main Install and configure a cluster SGE from distribution repositories slurm main Install and configure a cluster using the grycap. slurm ansible role slurm-repo main Install and configure a cluster SLURM from distribution repositories swarm main Install and configure a Docker Swarm cluster using swarm containers torque main Install and configure a cluster TORQUE from distribution repositories www. egi. eu @EGI_e. Infra 22/04/2020 30

EC 3 components ]$ sudo docker run grycap/ec 3 templates | grep -i component blcr component Tool for checkpointing applications bowtie 2 component Fast and sensitive read alignment chronos component An open-source tool to orchestrate job execution in a Mesos cluster ckptman component Tool to automatically checkpoint applications running on Spot instances consul component Service Discovery and Configuration Made Easy docker component An open-source tool to deploy applications inside software containers [. . ] galaxy component Galaxy is an open, web-based platform for data intensive biomedical research hadoop component A framework that allows for the distributed processing of large data sets across kubefaas component Install Faa. S frameworks in a kubernetes cluster [. . ] www. egi. eu @EGI_e. Infra 22/04/2020 31

EC 3 @ work Deploy of an elastic cluster with Torque/MAUI to run simulations with Haddock in a few steps: ● Generate a valid token and prepare the auth. txt file to access EGI resources ● Prepare the RADL template ● Launch the cluster www. egi. eu @EGI_e. Infra 22/04/2020 32

Generate a valid OIDC token https: //aai. egi. eu/fedcloud The EGI AAI Check-In Webinar is now on www. egi. eu @EGI_e. Infra 22/04/2020 33

Configure the cluster settings Get the settings to deploy the cluster in the EGI cloud provider The egicli on Git. Hub. The EGI Fed. Cloud Webinar on www. egi. eu @EGI_e. Infra 22/04/2020 34

RADL template to describe resources The main purpose of the Resource and Application description Language (RADL) is to specify the requirements of the resources where the scientific applications will be executed. It is used to specify: ● hardware (CPU number, CPU architecture, RAM size, etc. ), ● software requirements (applications, libraries, database systems, etc. ). ● The underlying network capabilities required. www. egi. eu @EGI_e. Infra 22/04/2020 35

Prepare the RADL template for the cluster Create a template to a cluster with Cent. OS 7 as based OS in the EGI cloud provider www. egi. eu @EGI_e. Infra 22/04/2020 36

Launch the cluster in the cloud provider www. egi. eu @EGI_e. Infra 22/04/2020 37

More information Video tutorials and demos in You. Tube EC 3 in the EGI Marketplace: https: //marketplace. egi. eu/42 -applications-on-demand-beta EC 3 in the EOSC Marketplace: https: //marketplace. eosc-portal. eu/services/elastic-cloud-compute-cluster-ec 3 EC 3 Ao. D portal: https: //servproject. i 3 m. upv. es/ec 3 -ltos EC 3 official documentation: https: //ec 3. readthedocs. io/en/latest/ EC 3 source code: https: //github. com/grycap/ec 3 EC 3 cluster for HADDOCK: https: //github. com/EGI-Foundation/ec 3 -haddock-cluster www. egi. eu @EGI_e. Infra 22/04/2020 38

Contact the developers Amanda Calatrava amcaar@i 3 m. upv. es Miguel Caballer micafer 1@upv. es Instituto de Instrumentación para Imagen Molecular Universitat Politècnica de València (UPV) - Camino de Vera s/n 46022, Valencia Spain www. egi. eu @EGI_e. Infra 22/04/2020 39

EGI: Advanced Computing for Research www. egi. eu @EGI_e. Infra Thank you for your attention. Questions? This work by the EGI Foundation is licensed under a Creative Commons Attribution 4. 0 International License. The work of the EGI Foundation is partly funded by the European Commission under H 2020 Framework Programme