Introduction to ESnet Network Operating System ENOS SENSE

Introduction to ESnet Network Operating System (ENOS) SENSE PI Meeting FNAL, February 16 th, 2016 Eric Pouyoul (lomax@es. net)

Is ESnet really developing Yet Another Network Operating System (YANOS) ? Well, yes, sorry, but we had to ! Focus on: • Security • Resource sharing (multiapplications) • design patterns • Integrates ESnet services • Low learning curve and ease of use 2 1/3/2022 • ODL, ONOS, other were not available at the time and not with consistent feature set • ESnet traffic is not typical. • Not a controller (does not implement one) • Will leverage any WAN, production quality controller Co-design with hardware and network

ENOS Basic Modules User Management Local storage SSHD support Interactive shell Java and Python support Application Scheduler (to be implemented) Application Life Cycle (partially implemented) Centralized Logging Messaging / Events (in progress) Web Service Provider (to be implemented) Topology Service (partially implemented) Distributed database (not implemented) 3 1/3/2022

Optional ENOS Modules ESnet production network topology provider ESnet Testbed network topology provider ESnet perf. SONAR Testers topology provider ESnet Testbed Multipoint VPN Service (partial, in progress) perf. SONAR Lookup Service client ESnet SNMP collector (partial) Open. Daylight resource provider (partial, in progress) Inter-ENOS communication (in progress) Host local network configuration service (in progress) OVS switches resource provider (in progress) Linux Virtualization resource provider (partial, in progress) 4 1/3/2022

ENOS Modular Architecture REST API, Portal Layer 2 services Multi. Point L 2 VPN Layer 3 SDX Netshell security, core services, API, SSHD Network Services Drivers Topology, NSI, OSCARS, … 5 1/3/2022 Stats, Tests perf. SONAR, SNMP, Sampling Database Immediately or eventually consistent Controller Drivers ODL, ONOS, … VM Driver Linux containers (libvirt)

ENOS Building Blocks ENOS Services ENOS Modules Netshell Bundles, Python Karaf OSGi Java Virtual Machine 6 1/3/2022

ENOS and Software Technologies • Java Virtual Machine (JVM) for a production-ready environment • ENOS security leverages Java Security Manager • Performance • Python for rapid software development • Useful for experiments, quick prototyping • “The natural language of ENOS” • ENOS integrates Jython, a Python 2. 7 interpreter running in the JVM. • OSGi, Karaf for module, application management. • Open. Daylight for controlling SDN devices 7 1/3/2022

ENOS Applications An ENOS application runs within the ENOS application framework. It leverages the ENOS services and follows ENOS design patterns. A typical ENOS application architecture will implement some if not all of the following: 8 1/3/2022

ENOS Resource Drivers An ENOS Resource Driver is an ENOS application that interfaces to a non-ENOS service. An ENOS driver must convert the thirdparty, external service API to ENOS data types and API. ENOS Application Generic API Resource Driver Adaptation ENOS 9 1/3/2022 Third Party Software or Service

ENOS Resource Provider A Resource Provider is an ENOS application that let other ENOS applications access resources. It is responsible for maintaining the state of the resources as well as controlling its access (generally through ACLs). ENOS Application Resources Generic API Resource Provider Resources ACL ENOS 10 1/3/2022 An ENOS application can be a resource driver and provider at the same time.

ENOS Security Primitives ENOS security leverages the Java Virtual Machine’s security manager in order to provide basic mechanisms that are used by the applications, services and drivers to implement the desired protection to resources: 1. A system call defining and enforcing privileged and unprivileged execution. 2. All ENOS threads are associated with an ENOS user. Only privileged users can change the ownership of a thread. 3. Support for resource access control. 11 1/3/2022

Resource Access Control in ENOS 12 1/3/2022

Example: ENOS Multipoint VPN Service ENOS Virtual Machine Management DHCP Layer 2 Broadcast MAC Learning Layer 2 Provisioning Open Daylight Driver 13 1/3/2022 Virtualization / Isolation Path Computation OSCARS Driver ENOS Topology Service

SDN ESnet SDN Testbed ESnet PE Router Point of Presence (2+)x 10 GE (SDN POP) Planned SDN Testbed node locations (n)x 10 GE Planned SDN Testbed connectivity overlay (using OSCARS circuits) Testbed Host STAR DENV AMST AOFA LBL WASH ATLA CERN

ESnet SDN Testbed Hardware: Corsa Technology DP 6440 4 x 100 G CFP 2 Ports 24 x 10 G SFP+ Ports DP 6430 2 x 100 G CFP 2 Ports 24 x 10 G SFP+ Ports 100 G 10 G DP 6420 48 x 10 G SFP+ Ports 4 x 40 G QSFP+ Ports Open v. Switch + DP 6410 24 x 10 G SFP+ Ports Open. Flow 1. 3 / 1. 4 / 1. 5+ • FPGA-based SDN switch, with Open. Flow controller interface • Highly scalable both in number of flows and speed • Doing common actions (defined by pipeline) very fast • No broadcast, subset of Open. Flow matches and actions.

ESnet SDN POP Physical View ESnet SDN POP Logical View ESnet PE Router (2+)x 10 GE (n)x 10 GE Testbed Host Services VM

ESnet SDN POP Logical View ESnet PE Router (2+)x 10 GE (n)x 10 GE Services VM • Software switch and Services VM paired with every hardware switch. • Most flows only pass through hardware switch (green) • Flows requiring special handling go through software switch (blue), possibly to Services VM (red). • Hardware switch provides performance, stability • Software switch and service VM provide flexibility, without compromising reliability

Design Patterns at work: Traffic Mirroring. OVS LBL Private OSCARS layer 2 circuit VM 18 1/3/2022 ANL Flow STAR SDN POP Flow LBL SDN POP OVS Shared OSCARS layer 2 circuits Private OSCARS layer 2 circuit CERN SDN POP OVS CERN

Thank You / Questions ? Eric Pouyoul: lomax@es. net / epouyoul@lbl. gov 19 1/3/2022

Additional Slides 20 1/3/2022

>>> from net. es. netshell. api import Topology. Provider >>> graph = topo. get. Graph(Topology. Provider. Weight. Type. Traffic. Engineering) >>> lbl = topo. get. Node('lbl-mr 2@es. net') >>> amst= topo. get. Node('amst-cr 5@es. net') >>> from org. jgrapht. alg import Dijkstra. Shortest. Path >>> path = Dijkstra. Shortest. Path. find. Path. Between(graph, lbl, amst) >>> for link in path: . . . node = topo. get. Node. By. Link(link. get. Id() . . . print "Node= " + node. get. Id() + "tlink. Id= " + link. get. Id() Node= urn: ogf: network: es. net: sunn-cr 5 link. Id= urn: ogf: network: es. net: sunn-cr 5: to_lbl-mr 2_ip-a: 0 Node= urn: ogf: network: es. net: sacr-cr 5 link. Id= urn: ogf: network: es. net: sacr-cr 5: to_sunn-cr 5_ip-a: 0 Node= urn: ogf: network: es. net: denv-cr 5 link. Id= urn: ogf: network: es. net: denv-cr 5: to_sacr-cr 5_ip-a: 0 Node= urn: ogf: network: es. net: kans-cr 5 link. Id= urn: ogf: network: es. net: kans-cr 5: to_denv-cr 5_ip-a: 0 Node= urn: ogf: network: es. net: chic-cr 5 link. Id= urn: ogf: network: es. net: chic-cr 5: to_kans-cr 5_ip-a: 0 Node= urn: ogf: network: es. net: wash-cr 5 link. Id= urn: ogf: network: es. net: wash-cr 5: to_chic-cr 5_ip-a: 0 Node= urn: ogf: network: es. net: aofa-cr 5 link. Id= urn: ogf: network: es. net: aofa-cr 5: to_wash-cr 5_ip-a: 0 Node= urn: ogf: network: es. net: lond-cr 5 link. Id= urn: ogf: network: es. net: lond-cr 5: to_aofa-cr 5_ip-a: 0 Node= urn: ogf: network: es. net: amst-cr 5 link. Id= urn: ogf: network: es. net: amst-cr 5: to_lond-cr 5_ip-a: 0 21 1/3/2022

Lessons Learned and Summary • Java + Python + SSHD is very powerful and yet simple combination. • Aggregating, grooming and normalizing data such as topology simplifies greatly network aware applications. • OSGI/Karaf allows us to painlessly change ODL version, ONOS, other OSGi based controller/applications. • Security is not trivial, especially when using third party software: we needed to turn off security in order to use ODL. • Will be open sourced very soon. A network operating system does really look like a computer operating system. Perhaps Linux itself could be the execution environment for network policies. 22 1/3/2022