EGI Applications Database Status report Marios Chatziangelou et
EGI Applications Database Status report Marios Chatziangelou, et al. Institute of Accelerating Systems and Applications (IASA) www. egi. eu EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142
Capabilities A community driven, central service that stores and provides: Ø software solutions (in the form of native software and/or virtual appliances), originated from almost every scientific area/discipline Ø reference of scientific datasets (under development) Ø the programmers and scientists responsible for them Ø the publications derived from the registered items (SW, VA & datasets) Acts as distribution medium, for: Ø native software packages (RPMs, DEBs, TARs) exposed through the UMD Community Repository to the public Ø virtual machine images by using the HEPi. X virtualization vmcatcher/vmcaster technology 5/12/15 Insert footer here 2
Latest features • Support for VO-wide image lists • Full integration with the Grid/Cloud information system (BDII) and the GOCDB Service • Extend the App. DB authentication system by supporting federated AAI through edu. GAIN • Software Appliances • Integration with external metadata repositories • Reference datasets (pilot activity) 5/12/15 3
VO wide image lists (1) Ø Editable only by VO managers Ø VO wide image lists always private Ø Site subscriptions using the vmcatcher Ø A personal access token is required Ø All site admins already have or should get a personal access token Simple workflow Ø The user submits one or more VAs , the user notifies the VO manager, the VO manager dicides whether he will include the VA version or not to a VO wide image list Ø The user updates a VA (=> new VA version) , the system notifies the VO manager(s) that alerady includes the VA in their VO-wide image lists. . . ØSites retrieve the new VA version and publish the details into the BDII 5/12/15 4
VO wide image lists (2) Some metrics: 21 Cloud providers already support the App. DB Cloud Marketplace 13 Virtual Organizations and/or Projects are making use of the system 5/12/15 5
Full integration with BDII & GOCDB Sites & Resource Providers - correlated information from GOCDB & top-BDII ü Contact information ü Sitename ü Description ü URIs to GOCDB for more details ü List of offered v. Appliances and VMIs ü Usage details per offered VMIs Site endpoint Template ID OCCI ID 5/12/15 6
Extend authentication system: edu. GAIN Support authentication using federated Identity providers through edu. GAIN. First use case, EBI-EMBL. 1. select the “Federation (edu. GAIN)” option for login 2. 3. pick the EBI-EMBL Id. P and login into the App. DB with their EBI-EMBL credentials Liberates the users from the necessity of either obtaining a new account or for using X 509 certificates. 5/12/15 7
Software Appliances Main purpose: deliver a Software-as-a-Service solution Technically speaking: SAs are Virtual Appliances and Contextualization scripts pairs In practice: SAs is a solution for installing and/or parametrize 'any' software module within a base Virtual Machine, by using a contextualization script ( = is the first script being executed while a virtual machine it is instantiated/started) 5/12/15 8
Integration with external metadata repositories Projects and Organizations – acquired from the open. AIRE Integration with open. AIRE (through the OAI-PMH protocol-interface) Separate App. DB sub-service for managing the high volume of metadata periodically fetched from open. AIRE Over 33 k Projects and over 27. 5 k Organizations Many types of relations can be made, between the registered software, v. Appliance, person and the fetched projects and/or organizations 5/12/15 9
Reference datasets – pilot (1) • In the context of Life Sciences Data Replication VT, App. DB is being extended into a dataset registry • Initial focus is on Life Sciences reference datasets – Integration with the Elixir Tools and Data Services Registry is in the works – Support for other disciplines (e. g. HEP) will be considered in the near future, based on feedback and demand The App. DB dataset registry preview can be accessed from the new “datasets” tab on the development server instance https: //appdb-dev. marie. hellasgrid. gr 5/12/15 10
Reference datasets – pilot (2) • Entries in the dataset list view can be grouped by version • Every entry is tagged as either primary or derived 5/12/15 11
Reference datasets – pilot (3) Primary datasets represent original datasets, as posted by the provider – create/edit access available to Admins, Managers, and Dataset Managers Derived datasets are based on a primary dataset but – only a part of the information is kept (subset based on structure), – or only a part of the data entries are selected (subset based on quantity) – create/edit access to them may be open to all users Key characteristics recorded for each dataset include – name, description, disciplines, homepage link, licensing, and a version list Each dataset version may host one or more locations where data can be accessed Locations may be tagged as master or replica, and contain information about – the format in which the dataset is provided (e. g. FASTA, XML, CSV, etc. ) – the interface / connection type (e. g. REST, SOAP, SQL, etc. ) – the organization or EGI site which provides the location 5/12/15 12
Reference datasets – pilot (4) 5/12/15 13
Next steps Still on requirements gathering phase. Ø Stable requirements: – Finalize the dataset pilot related developments (expected on end of August) – Integrate the App. DB with the Elixir Registry (SW & VA & datasets) – EGI-Engage Do. W: “The EGI Applications Database (App. DB) will evolve from its current role as catalogue of applications and virtual machines (VM) to include a graphical user interface allowing authorized users to perform basic VM management operations”. • Open item: federated AAI Ø Draft requirements: – – – 5/12/15 Support for ‘Containers’ i. e. Docker, technology Developments with regards to the supported VMI formats (i. e. OVA, qcow 2, etc. ) Extensions and/or developments that will needed, towards the EGI Marketplace activity/task Integration with the EGI Accounting repository for fetching VM usage related metrics Performance improvements (i. e. database clustering, light refactoring of the client-side scripting/javascript etc) 14
Thank you for your attention. Questions? www. egi. eu EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142
- Slides: 15