EGI Applications Database Marios Chatziangelou et al mhaggeliasa

EGI Applications Database Marios Chatziangelou, et al. <mhaggel@iasa. gr> Institute of Accelerating Systems and Applications (IASA) www. egi. eu EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142

Capabilities A community driven, central service that stores and provides: Ø software solutions (in the form of native software and/or virtual appliances), originated from almost every scientific area/discipline Ø reference of scientific datasets (pilot - under development) Ø the programmers and scientists responsible for them Ø the publications derived from the registered items (SW, VA & datasets) 2

General features (1/2) information retrieval advanced searching mechanism (rated search results) 'faceted search' mechanism for refinements quality of information content tagging, ratting, commenting per registered item contact expertise information problem and comment abuse report centrally managed quality control taxonomy technical classification scientific classification tagging dissemination of information custom RSS/Atom news feeds news e-mail subscription lists user focused communication (messaging, requests, etc) special dissemination tool for sending ad-hoc messages to scientists 'follow' button for receiving all the activity related to a registered item dissemination features customizable through user preferences sharing content with social networks 3

General features (2/2) integration with App. DB RESTfull API, supports operations following a CRUD convention. advanced Auth. N/Auth. Z mechanisms (simple. SAML) flexible API stateless authentication mechanism using integrated with EGI Checkin service Personal Access Tokens (no need for X 509) support for multiple accounts for accessing user’s personal profile API documentation in good shape internally managed Auth. Z, based on allowed actions, or even, by adapting the App. DB Gadget roles and permissions (easy – copy & paste, one line of code – no technical everything is under https channel skills required, you may get it here) Auth. N/Auth. Z and security relations… … between all the entities listed below, are possible: – – – – software virtual appliances datasets persons virtual organizations sites / resource providers organizations projects App. DB already integrated with many EGI services EGI GOCDB for the list of sites, their metadata, including downtimes Top-BDII fetching sites dynamic information EGI Checkin for Auth. N and high level authorization attributes Perun and EGI Operations Portal for VO related details, inc. membership & roles Argo: retrieving the status of the Cloud-enabled sites 4

Indicative Statistics Cloud Marketplace 23 Service Providers 31 Virtual Organizations 5

Software Marketplace Registry for Software items: Applications, tools, Workflow frameworks and instances, Science Gateways, Middleware products, …) Offers release management capabilities - unlimited series of releases - light-weight & collaborative, release management process Acts as a repository for binary artifacts ü ü ü unlimited number of repositories per register software generic tarballs, RPM & DEB (32 bit/64 bit) binaries multiple flavor / operating system combinations simplified, web-based, process for uploading the binary artifacts YUM & APT repositories for automatic distribution artifacts populated through the UMD Community Repository 6

Cloud Marketplace (1/2) Registry for virtual appliances (VA) ü a logical container of versioned image file & metadata bundles Registry for software appliances (SA) ü a logical container of VA versions & contextualization scripts bundles VA distribution medium ü distributing endorsed VAs to the resource providers/sites Resource providers catalogue ü list of the VAs which are available by each site/resource provider Virtual Organizations (VO) catalogue ü list of the VAs which are available for each VO member 7

Cloud Marketplace (2/2) Remote Sites • Search for v. Appliances Information System (Top-BDII for Fed. Cloud sites) cloud m/w • Get availability & usage details • Start/Stop/…. an image using rocci tool local info system push info about the VA availability and usage cloudkeeper user: : visitor perform VM operations using the App. DB VMOps subscribe and/or get any update App. DB Image List Store App. DB generate image_list App. DB portal Cloud Marketplace • (Re)publish VO or Project’s accepted list of Virtual Appliances • Publish new v. Appliance version • Select/update/exclude v. Appliances • Insert/update/remove image(s) • Create a new v. Appliance version • Register a v. Appliance App. DB portal VO or Project Manager App. DB portal user: : submitter 8

App. DB VM Operations dashboard The objective (EGI-Engage Do. W) : “The EGI Applications Database (App. DB) will evolve from its current role as catalogue of applications and virtual machines images (VMI) to include a graphical user interface allowing authorized users to perform basic VM management operations” App. DB Portal (main service) VM operations dashboard 9

The App. DB VMops dashboard (1/4) https: //dashboard. appdb. egi. eu Highlighted features for the end-user ü User identification, including VO membership & role(s) ü Provide user views of deployed topologies and VM instances ü Personalized, wizard-like, topology builder and the necessary tools, for: ü ü ü ü Create a new topology with one (or more) instances of a specific VM Attach additional storage to the VM instances Deploy/Un-deploy a topology Start/Stop a topology (= all the VM instances of a topology) Start/Stop a single VM instance Configure VM (cloud-init & ansible) Execute bash script on deployment time 10

The VMops dashboard (2/4) Highlighted technical features ü ü ü TOSCA compatible Infrastructure Manager [1] as the OCCI communication layer Rest API for integration with other services Single-Sign-On with the portal and the rest of the App. DB sub-services Scalable architecture (one front-end many back-ends for load balancing) Strictly technical details ü Node. JS/HTML 5/CSS/Javascript ü Mongo. DB/Mysql ü RESTful API [1]. https: //appdb. egi. eu/store/software/infrastructure. manager. im 11

The VMops dashboard (3/4) Integrated with the main service – App. DB portal as a source of info …. GOCDB Top-BDII Argo Ops Portal Perun EGI Checkin Resource Provider/Site static data Downtimes Which VM and where is available OCCI IDs, resource tpls & other dynamic info Sites monitoring values List of VOs App. DB Portal (main service) A P I Correlated data fetched from the external sources & App. DB DB VMOps Dash VO membership & roles Authentication & Authorization VO membership & roles 12

The VMops dashboard (4/4) … lets see it in action: https: //dashboard. appdb. egi. eu 13

Where we are heading. . . Portal (main service) VM Operations Dashboard v. Appliance topology VM VM EGI fed. Cloud infrastructure . . . Software Block storage(s) ü Datasets ü App. DB system ü ü ü Support of any Virtual Organization Complex topologies ü ü ü Distributed among sites Interconnected Shared block storages Full monitoring of deployed topologies Dynamic DNS support …… 14

Thank you for your attention. Questions? www. egi. eu EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142