Open Cloud e Xchange OCX as a part

Open Cloud e. Xchange (OCX) as a part of High Performance Cloud Services Delivery Infrastructure (CSDI) for Big Data Applications Yuri Demchenko (Uv. A) On behalf of OCX development team GN 3 plus JRA 1 Task 2 EGI-GEANT Symposium 25 -26 September 2014 Amsterdam connect • communicate • collaborate

Agenda connect • communicate • collaborate • • • GN 3+ JRA 1: Network Architectures for Horizon 2020 General use cases and scenarios for Cloud services delivery to NRENs and universities – Cloud and Intercloud Infrastructure and Services Geant Open Cloud e. Xchange (g. OCX) – OCX design principles and suggestions – Background concepts and related research g. OCX Demos – TNC 2014 demo (Uv. A, Okeanos, Cloud. SIGMA, SURFnet, SWITCH) – SC 14 demo 16 -21 Nov 2014 (g. OCX + Amazon AWS) Further development Additional information 2

JRA 1: Network Architectures for Horizon 2020 connect • communicate • collaborate • • Task 0: Activity Leadership Task 1: Future Network Architectures Task 2: Network Architectures for Cloud Services Task 3: Network Architecture for Aggregating High-Speed Mobile Networking 3

Use Cases for delivering Cloud services to campus based users • • • connect • communicate • collaborate Scientific application and scientific (Big) data – LHC/HEP, genomics, astronomy, climate, 4 K/UHD video, etc. Streaming high-speed high volume experimental data to labs in campus location – Direct links through campus network Distributed (Big) Scientific Data processing with MPP tools on distributed facilities – Data distributed between few locations next to local datacenters CSP and campus L 0 -L 2 (L 3) network peering – Dark fiber with termination as campus network or as CSP network Vo. IP – approach with mobile data access – Support mobile access network (LTE) and tunnel access to campus network 4

General use case for infrastructure provisioning: Workflow => Logical (Cloud) Infrastructure connect • communicate • collaborate Enterprise/Scientific workflow Storage Data Special Proc 1 Data Filtering Input Data Visual Present Special Proc 2 Instrum. Data Campus A Data Archive Visualisation CE User Group A CE User User VR 6 VR 2 Cloud 2 Paa. S User Group B VR 7 VR 4 VR 1 VR 5 Resource/ Service Provider Enterprise/Scientific workflow Is mapped to heterogeneous cloud infrastructure containing Iaa. S, Paa. S components Campus B VR 3 Enterprise/Project based Intercloud Infrastructure Cloud 1 Iaa. S Resource/ Service Provider CN CN CN Cloud Paa. S Provider CN CN CN Cloud Iaa. S Provider 5

Multi/Inter- cloud infrastructure provisioning: => Intercloud Federation and g. OCX functions Big Data and Data Intensive Tasks Enterprise/Scientific workflow Campus A Storage Data Visualisation User Group A Special Proc 1 Visualisation Data Archive Data Filtering Input Data CE connect • communicate • collaborate Visual Present CE Special Proc 2 Instrum. Data User User Campus B User Group B OCX as a part of GÉANT infrastructure OCX at NREN Open Cloud e. Xchange (OCX) interconnects CSP and Customer (campus) infrastructures OCX assumes collocation or collapsed backbone for connecting all OCX members Solving “last mile” problem OCX at NREN VR 6 VR 2 Cloud 2 Paa. S VR 7 VR 4 VR 1 VR 5 Resource/ Service Provider VR 3 Enterprise/Project based Intercloud Infrastructure Cloud 1 Iaa. S Resource/ Service Provider CN CN CN Cloud Paa. S Provider CN CN CN Cloud Iaa. S Provider 6

g. OCX Definition and Operational Principles connect • communicate • collaborate • Direct service/inter-member peering – – – • – L 0 -L 2 network interconnection facility – • Associated service should allow topology information exchange between providers and customers in a secure and consistent way No value added third party (intermediary/broker) services – • Re-use and leverage two known concepts: Internet e. Xchange Point and GLIF Open Lightpath Exchange (GOLE) Physical Point of Presence (Po. P) for providers and customers Scalability for growing number of members Controlled network parameters (Qo. S) – Transparency for cloud based services No involvement into peering or mutual business relations Trusted Third Party (TTP) services – – – To support dynamic federation establishment Trusted Introducer for dynamic trust establishment Open Marketplace and SLA clearinghouse for member CSP’s 7

OCX Topological model and Connectivity connect • communicate • collaborate OCX L 0 -L 2/L 3 topology • Any-to-any • Distributed or collapsed backbone • Hierarchical • Topology information exchange L 0 -L 2 (+ L 3? ) between members • SDN control over OCX switching OCX Nodes: Providers and customers OCX Facilities Qo. S parameters • Bandwidth • Speed, latency • Jitter, impairment OCX backbone links/connectivity (Lo-L 2) Cloud. Com 2013 Open Cloud e. Xchange (OCX) 8 8

OCX Trusted Third Party services connect • communicate • collaborate TTP goals and services TTP • • • OCX • Enable dynamic federations establishing Service Registry and Discovery – SLA repository and clearinghouse – Repository of CSP policies Trusted Certificates and CA’s Repository – Pre-established trust relation with OCX as TTP Open Marketplace – Similar to TACAR (TERENA Academic CA Repository) Trusted Introducer Service Trust relations established as a part of dynamic federation between OCX members Cloud. Com 2013 Open Cloud e. Xchange (OCX) 9 9

OCX location options: GEANT, NREN connect • communicate • collaborate GÉANT CSP NREN Visualisation VR 6 VR 7 DFlow IP/L 3 University VR 4 CE OCX User VR 5 VR 3 L 2 Visualisation VR 6 L 1 CE VR 7 OCX VR 4 User VR 5 L 0 Cloud. Com 2013 VR 3 Open Cloud e. Xchange (OCX) 10 10

OCX Hierarchical and Distributed Topology Model connect • communicate • collaborate GÉANT CSP NREN University Visualisation VR 6 VR 7 VR 4 CE OCX User VR 5 DFlow VR 3 IP/L 3 L 2 Visualisation VR 6 L 1 CE VR 7 OCX VR 4 User VR 5 L 0 Cloud. Com 2013 VR 3 Open Cloud e. Xchange (OCX) 11 11

Background concepts and research connect • communicate • collaborate OCX concept is conceptually justified and (to be) practically proven • • Big Data and Cloud Computing – Complete fusion + High-Performance Scalable Computing – Multi-provider services infrastructure and inter-provider communication – Moving data, tasks and workflow => Intercloud Cloud Services Delivery Infrastructure (ICSDI) Intercloud Architecture Framework (ICAF) proposed by Uv. A Intercloud Federation: part of ICAF and mainstream development in Cloud Computing industry is striving to solve the problem of services and content delivery from cloud provider to customer – Direct. Connect by AWS and Express. Connect by Microsoft Azure – Equinix Cloud Exchange for inter-provider business and infrastructure level integration – CSP’s Content Delivery Networks: Cloud. Front by AWS and CDN by Microsoft Azure – Federated IDM and Access Control – Marketplace and VMs/Services/Apps directory 12

Gartner Hypercycle for Cloud Computing and Big Data connect • communicate • collaborate Big Data Cloud Computing 13

g. OCX Development – Milestones and Next Steps connect • communicate • collaborate • Design and implementation – – • – API definition and topology visualization (+ Opt/Net BV) – SDN based control and management over OCX Update based on the public draft discussion – • Started November 2013 Followed by implementation March – May 2014 TNC 2014 Demo 19 -22 May 2014, Dublin OCX Design meeting 3 -4 July 2014, Amsterdam – Whitepaper GN 3 plus-JRA 1 -task 2 (published) Security, topology, protocols, use cases Planned demos – – TNC 2014, Dublin (May 2014) SC 14, USA (November 2014) GN 3 plus Symposium (February 2015) TNC 2015, Porto (June 2015) Cloud. Com 2013 Dissemination and Standardization • Standardization contribution: use case, awareness – OGF, IETF, IEEE, NIST IEEE Inter. Cloud Testbed (ICTB) Initiative – To consider Conference and publications – Cloud. Com 2013 and Net. Cloud 2013 Workshop, Bristol – TNC 2014, Dublin – TNC 2015, Porto (TBC) – Intercloud 2015 Workshop @IC 2 E 2015 – • Open Cloud e. Xchange (OCX) 14 14

g. OCX Demo @TNC 2014 connect • communicate • collaborate Demo Scenario: HD video editing and streaming • • The University of Amsterdam (Uv. A) has some 4 K movies that need efficient transcoding. Using their local OCX (Nether. Light) the Uv. A can easily get access to necessary compute resources at different Cloud Service Providers via high performance dedicated network links. In the demo we use Okeanos (connected via GRNET OCX) and Cloud Sigma (connected via SWITCH OCX). The Uv. A created scheduling software that is able to spawn virtual machines at Okeanos or Cloud Sigma. The machines are spawned inside the L 2 -domain of the Uv. A, which gives the Uv. A full control. 15

Planned SC 14 Demo – 16 -21 November 2014, Tempe, Arizona USA • • • connect • communicate • collaborate Scenario: large volume 4 K video processing – Parallel processing, upload and download, visualization/streaming Participants – Uv. A – SURFnet, GRNET, SWITCH, HEAnet (TBC) – Okeanos, Cloud. Sigma, AWS (confirmed), Microsoft Azure (TBC) New tools to be involved – Vampire (Uv. A): VM placement and tasks allocation based on Bag of Tasks (Bo. T) – Application Layer – Topology Visualisation: External contribution by Opt/Net BV New OCX development involved – OCX API: CSP-OCX-CSP, Apps-OCX, Signaling, Topology and path exchange Testing real-life scenario: VM images compatibility, Live migration, inter-cloud/inter-processes signaling 16

Future Developments connect • communicate • collaborate • • Combining SDN and OCX – OCX functions control, integration with network environment and Application layer control OCX API Definition – CSP-OCX-CSP – Application Layer to OCX – Signaling, Topology and path exchange TTP to support dynamic federations creation Application layer integration: Bo. T Vampire (Uv. A) and marketplace/brokering 17

Proposal: g. OCX to SDN-g. OCX connect • communicate • collaborate Service: • Connectivity p. CSPs through OCX into the NREN (a. CSP/a. ICTs) and vice versa Value proposal from SDN: • Configurability (policy based) and programmability of the control/data plane • Automation • Compatible APIs with other SDN components 18

Thank you Questions? More information: • g. OCX Whitepaper http: //www. geant. net/Resources/White_Papers/Documents/MS 101_M J 1 -2 -1_Network-Architectures-for-Cloud-Services. pdf • TNC 2014 g. OCX presentation and paper https: //tnc 2014. terena. org/core/presentation/21 • TNC 2014 Demo video (4: 50) http: //www. youtube. com/watch? v=q 7 IAAFUc. TY 0&feature=youtu. be connect • communicate • collaborate