Arrebol Easy and Efficient Execution of BagofTask Applications

Arrebol: Easy and Efficient Execution of Bag-of-Task Applications on Federated Clouds Federal University of Campina Grande Systems and Computing Department Distributed Systems Laboratory Prof. Francisco Brasileiro, fubica@computacao. ufcg. edu. br Prof. Andrey Brito, andrey@computacao. ufcg. edu. br Open. Stack Summit, Barcelona, October 2016

Outline • • • Motivation Managing Bag-of-Tasks applications The Arrebol service Connecting multiple clouds using Fogbow Short demonstrations Q&A Open. Stack Summit, Barcelona, October 2016

Motivation • Many e. Science applications are structured as a Bag-of-Tasks (Bo. T) • Usually: – they have sporadic demand for very large computing capacities – tasks are independent, thus, their execution can be parallelized in a very easy way Open. Stack Summit, Barcelona, October 2016

Motivation Capacity • Cloud infrastructures are the perfect match for Bo. T applications Costs ≅ Time Open. Stack Summit, Barcelona, October 2016 Capacity • The research cycle could be speeded up, if applications could complete faster Time

Managing the execution of Bo. T applications in the cloud VM 6. Destroy VM 1. VM 4. Monitor execution 2. Stage input files 3. Create Start remote execution 5. Retrieve output data ? USER Open. Stack Summit, Barcelona, October 2016 CLOUD SERVICE

Automating the execution of Bo. T applications in clouds with Arrebol job: label: simulation_experiment_50 requirements: Glue 2 RAM >= 1024 AND Glue 2 CPU >= 1 init: sudo apt-get update ARREBOL sudo apt-get -y install simulator SERVICE final: VM VM rm /tmp/*. out VM task: put /home/fubica/experiments/50/input/file 01 /tmp/file 01. in. VM VM run_simulator –i /tmp/file 01. in –o /tmp/file 01. out 1. User a job/home/fubica/experiments/50/output/file 01 (CLI) get submits /tmp/file 01. out VM VM task: 2. Arrebol service parses the job put /home/fubica/experiments/50/input/file 02 /tmp/file 02. in run_simulator –i /tmp/file 02. in –o /tmp/file 02. out 3. Arrebol service creates VMs get /tmp/file 02. out /home/fubica/experiments/50/output/file 02 4. …Arrebol service manages execution ? ? USER ? ? ? 5. Arrebol service fetches outpus 6. Arrebol service destroys VMs Open. Stack Summit, Barcelona, October 2016 CLOUD SERVICE

Arrebol in action • Demo #1: the Arrebol service Open. Stack Summit, Barcelona, October 2016

Increasing the capacity available • Cloud providers usually impose restrictive resource quotas – Promotes fair share of resources – Facilitates long term capacity planning • Capacity can be extended in several ways: – If using a public cloud • Multi-clouds: accessing other public providers (directly or via a broker service) – If using a private cloud • Cloudbursting on public clouds/multi-clouds • Federating with other private clouds • Cloudbursting + federating Open. Stack Summit, Barcelona, October 2016

Using Fogbow to extend the capacity of the available infrastructure • Fogbow is a suite of open source software projects developed by the Federal University of Campina Grande – Supports access to: • Multiple public clouds • Federation of private clouds • Any combination of the above – Supports the deployment of opportunistic green clouds using conventional desktops Open. Stack Summit, Barcelona, October 2016

Fogbow & Arrebol in action • Demo #2: using a federation of clouds + Open. Stack Summit, Barcelona, October 2016

Fogbow’s architecture Open. Stack Summit, Barcelona, October 2016

Multi-level A&A • A&A is performed at multiple levels – A local request carries federation credentials that are A&A by a Federation Identity Provider – A&A federation credentials are associated to a token that is used to map the federation user to the local cloud credentials that will be used to access the local cloud – The allocation manager performs A&A of a remote request considering both this token and the identity of the requesting allocation manager • Mapping is autonomously defined by each cloud administrator Open. Stack Summit, Barcelona, October 2016

Plugin-based approach • Interoperability plugins make it simple to support different cloud orchestrators • Behavioral plugins allow cloud administrators to customize the federation functionalities that they deploy in their local cloud – E. g. prioritization policies Open. Stack Summit, Barcelona, October 2016

Internet-friendly communication • A messaging service allows membership and allocation managers to communicate with each other locally and remotely, without requiring them to have their endpoints publicly accessible • A reverse tunneling service provides public IP access to all virtual machines created in the private clouds, even if the local cloud offers only private IPs to these virtual machines Open. Stack Summit, Barcelona, October 2016

Success stories so far • Fogbow has been used to federate private clouds in a research project that involved Brazilian and European institutions (http: //eubrazilcloudconnect. eu/) – VOMS was used to implement the federation level A&A – This federation had Open. Stack and Open. Nebula clouds • The Brazilian NREN (RNP) is currently experimenting the use of Fogbow to federate its private clouds, as well as those of some of its clients (http: //portal. naf. lsd. ufcg. edu. br/) – CAFe federation of identity providers (Shibboleth-based) is used to implement authentication at the federation level – This federation has Open. Stack and Cloud. Stack private Iaa. S clouds Open. Stack Summit, Barcelona, October 2016

Q&A • Thanks for your attention! • Useful links: – Arrebol (http: //arrebol. lsd. ufcg. edu. br/) – Fogbow (http: //fogbowcloud. org/) – Distributed Systems Lab at UFCG (http: //lsd. ufcg. edu. br/) Open. Stack Summit, Barcelona, October 2016