INDIGO Data Cloud RIA653549 Paa S activities INFN

INDIGO – Data. Cloud RIA-653549 Paa. S activities @INFN marica. antonacci@ba. infn. it INFN-Bari

Paradigm shifts § From static partitioning to dynamic resource allocation § Improve resource utilization § Improve scalability and fault tolerance § From Monolithic App to Microservices § Easier to develop § Easier to scale § Fault isolation § From VMs to Containers § Run your application (even legacy) everywhere: on public and private clouds INDIGO-Data. Cloud RIA-653549

Docker ecosystem in Openstack Heat Solum Kubernetes Docker Magnum Open. Shift Cloud. Foundr y INDIGO-Data. Cloud RIA-653549

Heroku-inspired frameworks § Deis builds upon Docker and Core. OS to provide a lightweight Paa. S § Build/Release/Run § Similar solutions: § flynn, tsuru INDIGO-Data. Cloud RIA-653549

Mesosphere § Mesosphere offers a layer of software that organizes your machines, VMs, and cloud instances and lets applications draw from a single pool of intelligently- and dynamically-allocated resources, increasing efficiency and reducing operational complexity § Key Concept: Data Center Operating System INDIGO-Data. Cloud RIA-653549

Mesos Two-Level Scheduling INDIGO-Data. Cloud RIA-653549

Mesos Frameworks § Long Running Services § Aurora, Marathon, Singularity, SSSP § Big Data Processing § Cray Chapel, Dpark, Exelixi, Hadoop, Hama, MPI, Spark, Storm § Batch Scheduling § Chronos, Jenkins, Job. Server, Torque § Data Storage § Cassandra, Elastic. Search, Hypertable INDIGO-Data. Cloud RIA-653549

Cluster configuration - example INDIGO-Data. Cloud RIA-653549

Conclusions Preliminary tests § Deis: § Installation is simple, but takes too long (~1 h for 5 nodes cluster) § Poor stability § Debugging not so easy § Supported apps: twelve-factors, only ONE HTTP port § Mesos: § Easy to deploy § Packages available for Debian/Ubuntu and Redhat/Centos § Automatic provisioning available § Easy to scale INDIGO-Data. Cloud RIA-653549

§Backup Slides INDIGO-Data. Cloud RIA-653549

Service Discovery § Mesos-DNS (alpha) allows to connect to an application in the Mesos datacenter knowing only its name § can be made fault-tolerant by launching with a framework like Marathon, that can monitor application health and re-launch it on failures. § Marathon SD allows easy network communication between services § Based on HAProxy TCP/HTTP load balancer plus an assistant script that uses Marathon’s REST API to periodically re-generate a HAProxy configuration file INDIGO-Data. Cloud RIA-653549

Applications in Marathon § JSON app definition curl -X POST -H "Content-Type: application/json" http: //<master>: 8080/v 2/apps -d@app. json § Marathon hands over execution to Mesos. § Mesos executes each task in its own sandbox environment § Marathon knows how to handle application resources INDIGO-Data. Cloud RIA-653549