EGI Applications Database VM Operations Dashboard Marios Chatziangelou

EGI Applications Database VM Operations Dashboard Marios Chatziangelou, et al. <mhaggel@iasa. gr> Institute of Accelerating Systems and Applications (IASA) www. egi. eu EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142

Capabilities A community driven, central service that stores and provides: Ø software solutions (in the form of native software and/or virtual appliances), originated from almost every scientific area/discipline Ø reference of scientific datasets (pilot - under development) Ø the programmers and scientists responsible for them Ø the publications derived from the registered items (SW, VA & datasets) Ø enabling users to deploy and manage Virtual Machines to the EGI Cloud infrastructure through the VMOps dashboard 2

Software Marketplace Registry for Software items: Applications, tools, Workflow frameworks and instances, Science Gateways, Middleware products, …) Offers release management capabilities - unlimited series of releases - light-weight & collaborative, release management process Acts as a repository for binary artifacts ü ü ü unlimited number of repositories per register software generic tarballs, RPM & DEB (32 bit/64 bit) binaries multiple flavor / operating system combinations simplified, web-based, process for uploading the binary artifacts YUM & APT repositories for automatic distribution artifacts populated through the UMD Community Repository 3

Cloud Marketplace (1/2) Registry for virtual appliances (VA) ü a logical container of versioned image file & metadata bundles Registry for software appliances (SA) ü a logical container of VA versions & contextualization scripts bundles VA distribution medium ü distributing endorsed VAs to the resource providers/sites Resource providers catalogue ü list of the VAs which are available by each site/resource provider Virtual Organizations (VO) catalogue ü list of the VAs which are available for each VO member 4

Cloud Marketplace (2/2) Remote Sites • Search for v. Appliances Information System (Top-BDII for Fed. Cloud sites) cloud m/w • Get availability & usage details • Start/Stop/…. an image using rocci tool local info system push info about the VA availability and usage cloudkeeper user: : visitor perform VM operations using the App. DB VMOps subscribe and/or get any update App. DB Image List Store App. DB generate image_list App. DB portal Cloud Marketplace • (Re)publish VO or Project’s accepted list of Virtual Appliances • Publish new v. Appliance version • Select/update/exclude v. Appliances • Insert/update/remove image(s) • Create a new v. Appliance version • Register a v. Appliance App. DB portal VO or Project Manager App. DB portal user: : submitter 5

App. DB VM Operations dashboard The objective (EGI-Engage Do. W) : “The EGI Applications Database (App. DB) will evolve from its current role as catalogue of applications and virtual machines images (VMI) to include a graphical user interface allowing authorized users to perform basic VM management operations” App. DB Portal (main service) VM operations dashboard 6

The App. DB VMops dashboard (1/5) https: //dashboard. appdb. egi. eu Highlighted features for the end-user ü User identification, including VO membership & role(s) ü Provide user views of deployed topologies and VM instances ü Personalized, wizard-like, topology builder and the necessary tools, for: ü ü ü ü Create a new topology with one (or more) instances of a specific VM Attach additional storage to the VM instances Deploy/Un-deploy a topology Start/Stop a topology (= all the VM instances of a topology) Start/Stop a single VM instance Configure VM (cloud-init & ansible) Execute bash script on deployment time 7

The App. DB VMops dashboard (2/5) Wizard-like topology builder ü ü pick a Virtual appliance select the VO for consuming the resources choose a Site specify the resources needed (ram, cpu, disk, …. ) ü make the final adjustments (if any) o o multiple instances additional storage additional users on_deployment script ü and…… launch 8

The App. DB VMops dashboard (3/5) View & manage a topology ü Switch between VMs within a topology ü View details for each VM, like: o o o IPs How to access (ssh command, generated/provided keys, users etc) Attached Storage (incl. mount points) Logs (general & ansible/contextualization) OCCI ID for moving to CLI world ü Manage topology (start/stop/undeploy) or individual VMs (start/stop) 9

The App. DB VMops dashboard (4/5) List of topologies or VM instances ü available filters: o o o active (not undeployed/failed) undeployed by the user (you!) undeployed by the infrastrucure (not by you!) failed all ü All the details needed for each topology or MV instance 10

The App. DB VMops dashboard (5/5) View of the infrastructure availability any point of time 11

The recipe. . What it was actually needed for the App. DB VMOps to be realized? q Information & q Software (development) 12

Information Integrated with the main service [App. DB main service as a source of info] GOCDB Top-BDII Argo Ops Portal Perun EGI Checkin Resource Provider/Site static data Downtimes Which VM and where is available OCCI IDs, resource tpls & other dynamic info Sites monitoring values List of VOs App. DB Portal (main service) A P I Correlated data fetched from the external sources & App. DB DB VMOps Dash VO membership & roles Authentication & Authorization VO membership & roles 13

Software blocks High-level/Context diagram Dashboard (1 inst. ) • User’s interface • User/accounts configuration • Information collector • Auth/Auth. Z VMOps (clustered 4 inst. ) • Action management • Monitoring & State management • Proxy certificate management Infrastructure Manager (clustered 10 inst. ) • OCCI communication layer 14

Next steps ü Implement workflows/functionality to support complex topologies o ability to use different types of VMs within a topology example: 1 centos 7 and 2 VMs of ubuntu 16. 04 in a single topology o interconnected VMs example: the centos holds the database and the ubuntu VMs offers the content management system (CMS), i. e. Word. Press, for load balancing o shared block storages among VMs o spread VMs in different sites (theoretically this could be also possible) ü Advanced functions, like o Snapshotting o DNS on demand for the VM instances ü Support of any VO ü Moving from proxy certificates to OIDC ü Support of native connectors (Open. Stack & Open. Nebula) 15

Technical features & details Highlighted technical features ü ü ü TOSCA compatible Infrastructure Manager [1] as the OCCI communication layer Rest API for integration with other services Single-Sign-On with the portal and the rest of the App. DB sub-services Scalable architecture (one front-end many back-ends for load balancing) Strictly technical details ü Node. JS/HTML 5/CSS/Javascript ü Mongo. DB/Mysql ü RESTful API [1]. https: //appdb. egi. eu/store/software/infrastructure. manager. im 16

Summarizing. . . Portal (main service) VM Operations Dashboard v. Appliance topology VM VM EGI fed. Cloud infrastructure . . . Software Block storage(s) ü Datasets ü App. DB ü ü ü Support of any Virtual Organization Complex topologies ü ü ü Distributed among sites Interconnected Shared block storages Full monitoring of deployed topologies Dynamic DNS support …… 17

Thank you for your attention. Questions? www. egi. eu EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142