EGIIn SPIRE Federated Cloud Status David Wallom University

Value proposition The EGI Federated Cloud, a federation of institutional private Clouds, offering Cloud

EGI Cloud Infrastructure EGI Application EGI Cloud Service Marketplace Image Repository DB EGI Collaboration

Cloud Capabilities 1. Manage VM instances • • Consistent operations across multiple different CMF

Cloud Capabilities 7. Federated AAI • User requirements are key, survey ongoing 8. VM

Geographical dispersion 5/14 • 12 countries provide 15 certified resources – Czech Republic, Germany,

Geographical dispersion 9/14 • 12 countries provide 19 certified resources – • 5 countries

Fed. Cloud in Numbers • 15 certified Resource Providers – – – – •

Fed. Cloud in Numbers • 19 certified Resources from 17 Providers – – –

Accounting • • Current Usage Record Content Site User: local user/group, Digital Certificate DN

Accounting Usage Record Content • Site • User: local user/group, Digital Certificate DN •

EGI Fed. Cloud Communities 9/2014 • Ecology – Bio. Ve. L: Biodiversity Virtual e-Laboratory

New EGI Fed. Cloud Communities since launch • Education – Cranfield University distributed systems

Technology Evolution • Broader support for open standards in Cloud management frameworks Value added

Conclusions • 7 new communities have joined the fedcloud activities since launch • Resources

EGI’s Cloud Vision for 2020 To support the digital European Research Area through a

Slides: 17

Download presentation

EGI-In. SPIRE Federated Cloud Status David Wallom University of Oxford Chair, EGI Federated Cloud EGI-In. SPIRE RI-261323 www. egi. eu

Value proposition The EGI Federated Cloud, a federation of institutional private Clouds, offering Cloud Services to researchers in Europe and worldwide A single cloud system able to • Scale to user needs • Integrate multiple different providers to give resilience • Prevent vendor lock-in • Enable resource provision targeted towards the research community Standards based federation of Iaa. S cloud: • Exposes a set of independent cloud services accessible to users utilising a common standards profile • Allows deployment of services across multiple providers and capacity bursting EGI-In. SPIRE RI-261323 www. egi. eu

EGI Cloud Infrastructure EGI Application EGI Cloud Service Marketplace Image Repository DB EGI Collaboration Tools EGI Cloud Infrastructure Platform Instance Mgmt Storage Management OCCI CDMI Cloud Management Stacks OVF (Open. Stack, Open. Nebula, Synnefo, …) Service Registry GLUE 2 GSI SAM UR Information Discovery Federated AAI Monitoring Accounting Help and Support Security Coordination Training and Outreach Sustainable Business Models EGI Core Platform EGI-In. SPIRE RI-261323 3 www. egi. eu

EGI Cloud Infrastructure EGI Application EGI Cloud Service Marketplace Image Repository DB EGI Collaboration Tools EGI Cloud Infrastructure Platform Instance Mgmt Storage Management Cloudinit CDMI OCCI Cloud Management Stacks OVF (Open. Stack, Open. Nebula, Synnefo, …) Service Registry GLUE 2 GSI SAM UR Information Discovery Federated AAI Monitoring Accounting Help and Support Security Coordination Training and Outreach Sustainable Business Models EGI Core Platform EGI-In. SPIRE RI-261323 4 www. egi. eu

Cloud Capabilities 1. Manage VM instances • • Consistent operations across multiple different CMF Inclusion of Contextualisation Integrated with OCCI V 1. 2 More from Boris 3. Data access/transfer interface • • Minimal useful data operation set available More from Christos 4. Cloud service information federation • Standardise GLUE 2 extensions 5. Resource consumption management • • Upgraded to latest SSM and rolling out Need storage accounting 6. Cloud service availability monitoring • EGI-In. SPIRE RI-261323 Continue probe development for new services and new capabilities 5 www. egi. eu

Cloud Capabilities 7. Federated AAI • User requirements are key, survey ongoing 8. VM Image Management • • Production marketplace More from Marios 9. Brokering • Get user communities using these! 10. SLA & Business Models - Sustainability • Part of p 4 u activity 11. Security • EGI-In. SPIRE RI-261323 Surveying technologies and resource providers as part of certification 6 www. egi. eu

Geographical dispersion 5/14 • 12 countries provide 15 certified resources – Czech Republic, Germany, Greece, Hungary, Italy, Macedonia, Poland, Slovakia, Spain, Sweden, Turkey, United Kingdom • 2 countries currently integrating – Croatia, Finland • 5 countries interested – Bulgaria, France, Israel*, The Netherlands, Switzerland • Worldwide interest – South Africa* (SAGrid) – South Korea* (KISTI) * Not shown on map EGI-In. SPIRE RI-261323 www. egi. eu

Geographical dispersion 9/14 • 12 countries provide 19 certified resources – • 5 countries currently integrating – • Croatia, France, Finland, Portugal, South Korea* (KISTI) 5 countries interested – • Czech Republic, Germany, Greece, Hungary, Italy, Macedonia, Poland, Slovakia, Spain, Sweden, Turkey, United Kingdom Bulgaria, Croatia, Israel*, The Netherlands, Switzerland, + more resources + technologies from countries already engaged Worldwide interest – – – Australia* (Ne. CTAR) South Africa* (SAGrid) United States* (NIST, NSF Centres TACC, Arizona) * Not shown on map EGI-In. SPIRE RI-261323 www. egi. eu

Fed. Cloud in Numbers • 15 certified Resource Providers – – – – • 100 Percent IT Ltd, United Kingdom The University of Zaragoza’s Institute for Biocomputation and Physics of Complex Systems (BIFI ), Spain Academic Computer Centre CYFRONET AGH (Cyfronet), Poland The Supercomputing Centre of Galicia (CESGA), Spain The Czech Education and Scientific NETwork (CESNET), Czech Republic Gesellschaft für wissenschaftliche Datenverarbeitung mb. H Göttingen (GWDG), Germany Consejo Superior de Investigaciones Científicas (CSIC) / Instituto de Física de Cantabria (IFCA ), Spain Okeanos by the Greek Research and Technology Network (GRNET), Greece Institute of Informatics, Slovak Academy of Sciences (II SAS), Slovakia Italian National Institute of Nuclear Physics (INFN), Division of Catania, Italy The Royal Institute of Technology, Sweden (KTH), Sweden Italian National Institute of Nuclear Physics (INFN), Division of Bari, Italy The Hungarian Academy of Sciences Institute for Computer Science and Control (MTA SZTAKI ), Hungary Turkish Academic Network and Information Centre (ULAKBIM), Turkey The European Information & Innovation Centre (UKIM), Macedonia 152 individuals participating and contributing to the Federated Cloud group – – EGI-In. SPIRE RI-261323 130 conference calls (Sep 2011 – May 2014) 14’ 800 VM instantiations accounted for in the last year www. egi. eu

Fed. Cloud in Numbers • 19 certified Resources from 17 Providers – – – – – • 100 Percent IT Ltd, United Kingdom The University of Zaragoza’s Institute for Biocomputation and Physics of Complex Systems (BIFI), Spain Barcelona Supercomputing Centre, Spain Academic Computer Centre CYFRONET AGH (Cyfronet), Poland The Supercomputing Centre of Galicia (CESGA), Spain The Czech Education and Scientific NETwork (CESNET), Czech Republic Forschungzentrum Juelich, Germany Gesellschaft für wissenschaftliche Datenverarbeitung mb. H Göttingen (GWDG), Germany Consejo Superior de Investigaciones Científicas (CSIC) / Instituto de Física de Cantabria (IFCA), Spain ~Okeanos by the Greek Research and Technology Network (GRNET), Greece Institute of Informatics, Slovak Academy of Sciences (II SAS), Slovakia Italian National Institute of Nuclear Physics (INFN), Division of Catania, Italy The Royal Institute of Technology, Sweden (KTH), Sweden Italian National Institute of Nuclear Physics (INFN), Division of Bari, Italy Italian National Institute of Nuclear Physics (INFN), Division of Padova, Italy The Hungarian Academy of Sciences Institute for Computer Science and Control (MTA SZTAKI), Hungary Turkish Academic Network and Information Centre (ULAKBIM), Turkey The European Information & Innovation Centre (UKIM), Macedonia Centro Extremeño de Tecnologías Avanzadas CIEMAT (CETA-CIEMAT), Spain 158 individuals participating and contributing to the Federated Cloud group – – – >150 conference calls (Sep 2011 – Sept 2014) 78, 422 VM instantiations accounted for in the last year 190, 513 CPU Hours on certified resources • EGI-In. SPIRE RI-261323 Currently no storage accounting though sites have examples of heavy utilisation www. egi. eu

Accounting • • Current Usage Record Content Site User: local user/group, Digital Certificate DN VMUUID (VM instance identifier) Start (and End) Time Status Wall and CPU Duration VM source image Cloud Type 156276 VMs launched across 24 clouds EGI-In. SPIRE RI-261323 11 www. egi. eu

Accounting Usage Record Content • Site • User: local user/group, Digital Certificate DN • VMUUID (VM instance identifier) • Start (and End) Time • Status • Wall and CPU Duration • VM source image • Cloud Type 217352 VMs launched across 25 clouds EGI-In. SPIRE RI-261323 12 www. egi. eu

EGI Fed. Cloud Communities 9/2014 • Ecology – Bio. Ve. L: Biodiversity Virtual e-Laboratory • Structural biology – We. NMR: a worldwide e-Infrastructure for NMR and structural biology • Linguistics – CLARIN: ‘British National Corpus’ service (BNCWeb) • Earth Observation – SSEP: European Space Agency’s Supersites Exploitation Platform for volcano and earthquakes monitoring (Collaboration with Helix Nebula) • Software Engineering – SCI-BUS: simulated environments for portal testing • Software Engineering – DIRAC: deploying ready-to-use distributed computing systems • Software Engineering – Catania Science Gateway Framework • Musicology – Peachnote: dynamic analysis of musical scores • Earth Observation – ENVRI: Common Operations of Environmental Research infrastructures (collaboration with EISCAT 3 D) • Geology – VERCE: Virtual Earthquake and seismology Research • Ecology – Life. Watch: E-Science European Infrastructure for Biodiversity and Ecosystem Research • High Energy Physics – CERN ATLAS: ATLAS processing cluster via Helix. Nebula More info: https: //wiki. egi. eu/wiki/Fedcloud-tf: Users EGI-In. SPIRE RI-261323 13 www. egi. eu

New EGI Fed. Cloud Communities since launch • Education – Cranfield University distributed systems course • Cultural Heritage – DCH-RP management of preservation services in the cloud • Hydrological Modelling – Running Hydrological models to support real time analysis • Bioinformatics – ELIXIR execution of the Ensamble application in the Federated Cloud environment • Systems implementations – deployment of FTK developed tools and services and data preservation • Internet of Things – Smart Grid systems investigation • Software Development – deployment of research Paa. S • RNA Sequencing – deployment of analysis engines in the cloud • Physiological Modelling – Calibration, scenario mapping and development More info: https: //wiki. egi. eu/wiki/Fedcloud-tf: Users EGI-In. SPIRE RI-261323 14 www. egi. eu

Technology Evolution • Broader support for open standards in Cloud management frameworks Value added services for User Communities – Utilisation of r. OCCI for interfaces to commercial cloud frameworks • New feature additions to foundational tools depending on requests – Accounting, monitoring, service discovery, Application DB • Innovate deployments of opensource Paa. S • Large improvements on documentation for all 3 stakeholder groups Strengthening the underpinning platform EGI-In. SPIRE RI-261323 15 www. egi. eu

Conclusions • 7 new communities have joined the fedcloud activities since launch • Resources Launch capability Q 3 Capability Q 4 2014 (planned) – – – 5, 000 cores, 225 TB storage ~6, 000 cores, ~300 TB storage 18, 000 cores, 6000 TB storage (3. 6 x) EGI-In. SPIRE RI-261323 (26 x) 16 www. egi. eu

EGI’s Cloud Vision for 2020 To support the digital European Research Area through a pan. European research infrastructure based on an open federation of reliable services that provide uniform access to computing and data resources provided by the public and private sector. EGI Federated Cloud capability vision 10 M cores Cloud compute 1 EB Cloud storage EGI-In. SPIRE RI-261323 17 www. egi. eu