EGIIn SPIRE EGI Applications Database EGIEngage Marios Chatziangelou

EGI-In. SPIRE EGI Applications Database EGI-Engage Marios Chatziangelou, et al. mhaggel@iasa. gr Institute of Accelerating Systems and Applications (IASA) www. iasa. gr 30/05/14 EGI-In. SPIRE RI-261323 1 www. egi. eu

EGI Applications Database A community driven central service that stores and provides: q software solutions in the form of native software products and/or virtual appliances, originating from almost every scientific area/discipline q the programmers and scientists who developed them q publications derived from the registered software solution In addition, acts as a software distribution medium for native software packages as well as virtual images EGI-In. SPIRE RI-261323 www. egi. eu

Benefits q Scientists & developers gain immediate recognition from a wide spectrum of audience q Avoid duplication of effort q Spend less time and effort for developing and/or porting software q Inspires other scientist less familiar with programming q Offers combined information from different thematic areas: Personal – Software – Cloud – Publications q Disseminates the information available, in order to be used by other services and portals (API, SEO, RSS, EMAIL) … and many more EGI-In. SPIRE RI-261323 www. egi. eu

q Highlighted features q Service X integrate with App. DB q App. DB integrate with EGI services q Security q Latest developments q Plans & Ideas for EGI-Engage q Summary & Conclusions EGI-In. SPIRE RI-261323 www. egi. eu

Indicative general features usability accessibility navigation content dissemination of information custom RSS/Atom news feeds news e-mail subscription lists focused user communication (messaging, requests, etc) special dissemination tool for sending ad-hoc messages to scientists 'follow' button for receiving all the activity related to a registered SW item dissemination features customizable through user preferences sharing content with social networks EGI-In. SPIRE RI-261323 information retrieval advanced searching mechanism (rated search results) 'faceted search' mechanism for refinements quality of information application tagging, ratting, commenting per application contact expertise information application problem and comment abuse report centrally managed quality control taxonomy technical classification scientific classification tagging software distribution repository for binary artifacts (lightweight release management process) VA distribution mechanisms www. egi. eu

Usability Accessibility +r world – privacy, only for a subset of VA related metadata (in progress) +w EGI SSO and X 509 support (Federated AAI and edu. GAIN is on the go…) (for more info about Auth/Authz see next slides) Navigation : from browser’s history EGI-In. SPIRE RI-261323 www. egi. eu

Quality of information Community Centrally managed driven - workflow: 1. q q 2. q 3. 4. q q Tagging Set as ‘Moderate’ Rating Contact user(s) Commenting Request for corrective actions Un-moderate or delete Sanitization Contacts EGI-In. SPIRE RI-261323 www. egi. eu

Information retrieval Powerful searching mechanism q Free text search (BNF notation is also supported) q Rated search results (by relevance, by id, by rating, …. . ) q Faceted search (multiple filters: a-z, freshness, discipline, category, status, prog. lang. , vo, country) EGI-In. SPIRE RI-261323 www. egi. eu

Taxonomy: technical classification EGI-In. SPIRE RI-261323 www. egi. eu

Taxonomy: scientific classification See the scientific classification EGI-In. SPIRE RI-261323 www. egi. eu

Dissemination of information (1/2) … through RSS EGI-In. SPIRE RI-261323 www. egi. eu

Dissemination of information (2/2) … through emails EGI-In. SPIRE RI-261323 www. egi. eu

Dissemination tool (1/2) Ability to…. q perform complex queries and locate people associated with a specific: - scientific orientation - software item or a virtual appliance - country, middle-ware, virtual organization - register item (SW or VA) that is of a specific discipline or category - and many, many other combinations q write an informative message q preview the message and inspect the recipients list q send the message q inspect that the message has been send by navigating to the logs tab EGI-In. SPIRE RI-261323 www. egi. eu

Dissemination tool (2/2) EGI-In. SPIRE RI-261323 www. egi. eu

Tune dissemination Ø E-mail settings Ø Dissemination & Communication Ø Notifications Ø API key management EGI-In. SPIRE RI-261323 www. egi. eu

Social Networks One-click far from sharing/disseminate info to social networks Support for: facebook linked. In twitter google+ EGI-In. SPIRE RI-261323 www. egi. eu

q Highlighted features q Service X integrate with App. DB q App. DB integrate with EGI services q Security q Latest developments q Plans & Ideas for EGI-Engage q Summary & Conclusions EGI-In. SPIRE RI-261323 www. egi. eu

Integration with App. DB: General A service can integrate with App. DB with two possible ways: • Inherit the App. DB Gadget (easy – no technical skills required) • Through the App. DB REST API (technical skills required) EGI-In. SPIRE RI-261323 www. egi. eu

Integration with App. DB: gadget (1/2) Web: https: //appdb. egi. eu/gadgets/editor Documentation: https: //wiki. appdb. egi. eu/main: integration: gadget EGI-In. SPIRE RI-261323 www. egi. eu

Integration with App. DB: gadget (2/2) Web: https: //appdb. egi. eu/gadgets/editor Documentation: https: //wiki. appdb. egi. eu/main: integration: gadget EGI-In. SPIRE RI-261323 www. egi. eu

Integration with App. DB: REST API Main aim: Third party registries, portals, databases and clients can retrieve, add or update content in the App. DB Database. Status: Operational since 2012 Supported Operations: CRUDL (Create, Read, Update, Delete, List) Typical use case: The API could used by the grid/cloud developer environments, for interacting with the EGI App. DB Database as part of their development projects. Authenticated access: apikey, username, password Net filters restriction rules: ability to control access from a specific locations per apikey API doc. with examples and sample use cases is available @ EGI Wiki EGI-In. SPIRE RI-261323 www. egi. eu

q Highlighted features q Service X integrate with App. DB q App. DB integrate with EGI services q Security q Latest developments q Plans & Ideas for EGI-Engage q Summary & Conclusions EGI-In. SPIRE RI-261323 www. egi. eu

Integrated with: GOCDB App. DB is already integrated with GOCDB, periodically acquiring data related to fed. Cloud sites: Site: : name Site: : hostname API Site: : endpoints Site: : status (prod, beta) Site: : country …etc EGI-In. SPIRE RI-261323 www. egi. eu

Integrated with: Operations Portal App. DB is already integrated with Operations Portal, periodically acquiring VO and VO membership related data: VO: : name VO: : description VO: : enrolement VO: : AUP …etc API VO: : member: : name VO: : member: : dn VO: : member: : email VO: : member: : role (member, manager, deputy, expert…) …etc EGI-In. SPIRE RI-261323 www. egi. eu

Integrated with: Top-BDII App. DB is already integrated with Top-BDII. Periodically acquiring live VM related metadata with regards to the VMs that are available to the sites/resource providers. Top - BDII ldap Addressing issues like: • Which VM is available in which site? • Under which VO? • Which templates are available by the site? • Which is the unique occi_id that should be populated to the end-user in order to instantiate the VM? EGI-In. SPIRE RI-261323 www. egi. eu

q Highlighted features q Service X integrate with App. DB q App. DB integrate with EGI services q Security q Latest developments q Plans & Ideas for EGI-Engage q Summary & Conclusions EGI-In. SPIRE RI-261323 www. egi. eu

Auth. N & account-profile mapping Auth. N q Technology used: simple. SAMLphp [1] q N Service Providers (SP) – one per web-application q 1 Identity Provider (Id. P) – restricted for *. appdb use only q Single-Sign-On for the *. appdb sub-services q Support for N authentication sources (including ‘hybrid’ x 509 support) q SP ↔ Id. P communication using SAML 2 protocol (Shibboleth is also an option) q Easy to integrate *. appdb SPs with 3 rd-party Id. P(s) Mapping (account ↔ App. DB profile) q N accounts (of any type) ↔ 1 App. DB profile q EGI SSO ↔ x 509 auto-connect functionality q “Connect” or “Create new profile” is up to the user q Secured connection process (send confirmation code with 30’ TTL) EGI-In. SPIRE RI-261323 www. egi. eu

Auth. Z & Privacy (2) New authorization mechanism: Group-based, default permission set (groups: admins, managers, NILs, power users, owners, contacts, users) Adjustable permissions per user per associated item (software or virtual appliance) • • edit, information & publication related info manage, software releases or VA versions respectively access, VA versions private data full control Privacy available for a subset of VA related metadata – easily to be extended to software item metadata as well Easy to use, self-explanatory, GUI Documentation: here EGI-In. SPIRE RI-261323 www. egi. eu

Auth. Z & Privacy (3) Only for Virtual Appliances Set permissions for the Contacts Set permissions for explicit users (no contacts) EGI-In. SPIRE RI-261323 www. egi. eu

Auth. Z & Privacy (4) By policy, system groups have additional permissions to the item EGI-In. SPIRE RI-261323 www. egi. eu

Auth/Auth. Z for the API • ‘Personal Access Tokens’ for Authoritative API calls Documentation: here Personal Access Tokens for API calls using vmcatcher for subscribing at private image lists [details] EGI-In. SPIRE RI-261323 www. egi. eu

q Highlighted features q Service X integrate with App. DB q App. DB integrate with EGI services q Security q Latest developments q Plans & Ideas for EGI-Engage q Summary & Conclusions EGI-In. SPIRE RI-261323 www. egi. eu

Latest developments (1) Multi-content layout – Software Marketplace (apps, tools, m/w products, science GWs and WFs ) – Cloud Marketplace (apps devel/servers/stacks, big data, infra, etc. . ) – People/Researchers registry (coord, sw eng, net eng, sysadmins etc. . ) EGI-In. SPIRE RI-261323 www. egi. eu

Latest developments (2) • Virtual appliances (VAs) – – – VA Registration VMIs Registration per VA VMCatcher compatible Image lists creation Required devels for: • User interface/Portal • Backend (db) • API (xml/json) • Development of a separate sub-service for handling submitted Image lists using vmcaster command line tool, as an alternative to the graphical way through the App. DB portal (documentation). [ https: //vmcaster. appdb. egi. eu/ ] • Deployment of an App. DB dedicated Wiki sub-service [ https: //wiki. appdb. egi. eu/ ] – Documentation in good shape. Suitable for: • Users/visitors or submitters • Resource Providers/Site admins • VO managers EGI-In. SPIRE RI-261323 www. egi. eu

Latest developments (3) VO wide image lists Editable only by VO managers VO wide image lists always ‘private’ – a personal access token required All site admins already have or should get a personal access token Simplified workflow The user submits on or more VAs – a VO manager is able to include or not to a VO wide image list The user updates a VA – a VO manager is able to include the update or not. EGI-In. SPIRE RI-261323 www. egi. eu

q Highlighted features q Service X integrate with App. DB q App. DB integrate with EGI services q Security q Latest developments q Plans & Ideas for EGI-Engage q Summary & Conclusions EGI-In. SPIRE RI-261323 www. egi. eu

Plans & ideas for EGI-Engage (1) 1 Virtual Appliances on demand – Platform as a Service (Paa. S) • offer brokering & contextualization features • introduce user ‘workspace’ where he can start/stop/monitor his Vas • compose VAs combining registered SW and registered base VAs 2 Introduce ‘Site/RPs’, ‘Project’ and ‘Organization’ entities (similar to the VOs we have now) • SW & VAs offered/supported by a specific Site, Project or Organization • SW & VAs management capabilities per entity 3 Advanced software distribution methods • Access (read/write) to a CVMFS stratum 0 head node – (hardware) 4 Integration with publication & data aggregators • as an example Open. Aire. EGI-In. SPIRE RI-261323 www. egi. eu

Plans & ideas for EGI-Engage (2) 5 Extend the authentication methods • Federated AAI, social networks, additional Id. Ps, …. 6 Integration with VAPOR • retrieving cloud-related (only? ) statistical metadata for VO & Sites (and maybe more, needs investigation) 7 Persistent Identifier (PID) for register items • Integrate with PID service(s) – additional cost/fees? 8 Integrate with UMD repository & RT (to check if actually needed) • user: manage m/w product releases • user: propose for UMD inclusion • UMD manager: accept/reject proposal • App. DB System: monitor the UMD workflow EGI-In. SPIRE RI-261323 www. egi. eu

Plans & ideas for EGI-Engage (3) Extended messaging system 9 • monitor messages (automated or user driven) that actually reach the recipient mailbox • retrieve statistical information (i. e. how many times a VA expiration msg has been sent to the VA owner/submitter) • integration with a dedicated email server Wiki & Forum related features 10 • offer Wiki and Forum services per registered SW & VA item • controlled write access EGI-In. SPIRE RI-261323 www. egi. eu

Plans & ideas for EGI-Engage (4) 11 Technical activities • Refactoring the codebase – User Interface (full transition from dojo to j. Query) – Zend uprade 1. x -> 2. x • • • 12 Database clustering (need investigation + hardware) HA-Failover system (need investigation + hardware) Migration of the Community repository database from mysql -> postgres General • Might need to change even the name of the service to something wider – require quite an accountable amount of effort • Quite interested on the ‘EGI. eu Service Catalogue & Marketplace’ EGI-In. SPIRE RI-261323 www. egi. eu

Depend on services… EGI: Operations Portal PERUN GOCDB VAPOR EGI SSO Ldap E-GRANT UMD (community) repository Non-EGI: PID services, social networks, several Id. Ps and many more… EGI-In. SPIRE RI-261323 www. egi. eu

q Highlighted features q Service X integrate with App. DB q App. DB integrate with EGI services q Security q Latest developments q Plans & Ideas for EGI-Engage q Summary & Conclusions EGI-In. SPIRE RI-261323 www. egi. eu

Summarizing • • The service relies on a user-focused, modern and friendly interface. • Offers advanced search mechanisms such as: It offers a unique number of user-oriented features, in terms of dissemination, collaboration, usability, accessibility, quality of information and many, many more…(see previous slides), a fact that makes the App. DB unique of its kind within the EGI Ecosystem. Free text search (BNF notation is also supported) Rated search results (by relevance, by id, by rating, …. . ) Faceted search (details) • The App. DB is already integrated with the most critical EGI services (GOCDB, Operations Portal, Information System) and is about to be integrated with PERUN (current month) • Extending the integration of the current services or integrating with new ones should be consider as a trivial task. • The service already offers a mature, reliable and documented RESTful API, capable of accepting and handling CRUDL operations (currently, the App. DB portal is using and therefore testing the API on every ‘click’) • Build on top of a modern and very flexible Authentication/Authorization system, based on Simple. SAMLphp technology, ready to be connected to any SAML 2 or Shibboleth compliant Id. P. (in the process of supporting edu. GAIN) • The service is currently the Virtual Appliance Marketplace for the fed. Cloud and is also responsible for the distribution of the images to the Resource Providers/Sites. EGI-In. SPIRE RI-261323 www. egi. eu

Concluding (1) Taking into account the current setup of the App. DB, the workplan for PY 5 and the plans for EGIEngage, the service will have the opportunity to act as a Platform as a Service and in parallel will offer access to the correlated information, aggregated by the following entities: Operations Portal GOCDB BDII VAPOR Publications data aggregators E-GRANT PERUN CVMFS UMD EGI-In. SPIRE RI-261323 PID registries Id. Ps www. egi. eu

Concluding (2) … and therefore, will try to solve the following complex relational problem Tip: ‘persons’ and ‘publications’ are not displayed in the diagram in order to keep it simple EGI-In. SPIRE RI-261323 www. egi. eu

Thank you!! Web: https: //appdb. egi. eu Contact: appdb-support@iasa. gr Documentation: https: //wiki. appdb. egi. eu EGI-In. SPIRE RI-261323 www. egi. eu