Testing and Modeling Ethernet Switches and Networks for

Testing and Modeling Ethernet Switches and Networks for Use in ATLAS High-level Triggers R. W. Dobinson, S. Haas, K. Korcyl, M. J. Le. Vine, J. Lokier, B. Martin, C. Meirosu, F. Saka, K. Vella Copyright© 2000 OPNET Technologies, Inc.

Overview • Ethernet as a candidate technology for ATLAS LVL 2 • ATLAS - the networking requirement • Parameterized model of an Ethernet switch • Large network modeling results using the parameterized switch model • GBE measurements • Fast Ethernet tester/ROB emulator DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 2

ATLAS LVL 2 system network 1564 buffers (distributed 2 Mbyte image) Readout buffers (ROB) Fast Ethernet Concentrating switches Gigabit Ethernet CENTRAL Gigabit switch Trigger rate 100 k. Hz Concentrating switches Fast Ethernet Processing nodes ~500 500 MIPS processors analyze data from 5% of buffers DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 3

ATLAS LVL 2 network traffic • Network nodes • 1564 ROBs (Readout Buffers ) • 500 -600 Processor CPUs • Message scenario • LVL 1 accepts @ 75 k. Hz [scalable to 100 k. Hz] • Supervisor CPU selects processor CPU for each event accepted by LVL 1 • Processor CPU sends ROB_REQUEST to selected (~5%) of ROBs • ROBs reply with ROB_REPLY containing designated ROI data • Traffic @ ROB: ~11000 request-response/sec • Worst case data rate: 4 MB/s • Traffic @ processor node: ~14000 request-response/sec • Average data rate: 6. 6 MB/s DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 4

Modeling • Parameterized model of a switch Motivation • Limited number of physical parameters • Fast execution • Direct determination by measurement § commodity Ethernet switches § parameterized model of a switch § comparison of parameterized model and results from measurements § • Parameterized model in large Ethernet network § § ATLAS HEP experiment at LHC in CERN second level trigger (LVL 2) for ATLAS experiment DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 5

Commodity Ethernet switches Most of the Ethernet switches on the market today have a hierarchical architecture • ports • modules • backplane chassis backplane Store-and-forward mode of operation • frame is fully stored in module with • an input port for inter-module transfer the frame is subsequently stored in a module with an output port module ports DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 6

Parameterized model for switch: inter-module communication Input buffer P 1 MAC P 3 P 7 P 4 Output buffer P 2 MAC Backplane Buffer manager P 6 Input module Buffer manager Output module Parameters: P 1 - Input Buffer Length [ #frames] P 2 - Output Buffer Length [ # frames] P 3 - Max To. Backplane Throughput [MB/s] P 4 - Max From. Backplane Throughput [MB/s] P 6 - Max Backplane Throughput [MB/s] P 7 - Inter-module Transfer Bandwidth [ MB/s] P 9 - Inter-module Fixed Overhead [µs] (not shown) DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 7

Switch parameters from latency and rate measurements • Measure latency as function of packet size (ping-pong) • Measure packet rate as function of packet size • No switch, intra-module, inter-module DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 8

Modeling calculations • Switch parameters determined from simple ping-pong and streaming measurements described • Switch model implemented as C++ code • Event-driven simulations • Object-oriented switch model embedded in OPNET (commercial package) • Calculations also carried out using the same model within Ptolemy DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 9

parameterized model vs measurement latency vs throughput DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 10

parameterized model vs measurement lost frame rate vs input data rate DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 11

ATLAS LVL 2 network performance studies Central switch: generic - fully non-blocking DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 12

Modeling results • The parameterized model uses a limited number of parameters, which can be measured on COTS switches • The parameterized model has been successfully verified on a number of switches • The parameterized switch model was used in modeling the large Ethernet network for the ATLAS second level trigger system • The model of the network can be used to verify applicability of different switches currently on the market and their impact on the network performance DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 13

CPU utilization measurements • Ping-pong configuration at different packet rates • Measure CPU utilization for various packet sizes, rates ATLAS LVL 2 REQUIREMENT DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 14

Limitations of PC-based measurements • PC with standard software unable to drive switches at line speed n n n 100% for FE only for packet size > 500 Byte [half duplex] 35% for GBE for any packet size - limited by host transfer Parameterization of a single switch needs to be tested over the entire operational envelope • Number of nodes limited to 40 in present tests n n Scaling up to ~2000 nodes is not credible Model needs to be verified on a larger test bed Conclusion: • Dedicated traffic generators needed for FE, GBE in order to fully characterize switches to be used in ATLAS HLT • Need larger test bed for greater confidence in model applied to full ATLAS HLT system DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 15

Rates: GB Ethernet NIC development • Commercial NIC - Alteon ACENIC • Inadequate performance using manufacturer’s firmware • Modifications - • Custom software developed • All traffic local to NIC (host memory not involved) • Full duplex line speed achieved for any size packet PCI BUS TIGON CHIP PCI interface PCI BUS MAC DMA 1 DMA 2 Phy CPU 1 Mem CPU 2 Mem GE Ext Mem 0. 5/1 Mbyte DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 16

8 Gigabit switch tester in a single chassis DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 17

Latency vs throughput for 8 -port GBE switch DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 18

Fast Ethernet tester • Custom board, built at CERN • Programmable network traffic to characterize network switches • IP, Ethernet, including Qo. S • Why build it instead of buying? • Economics • Reprogrammable for ATLAS purposes DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 19

FE tester: capabilities & status • 32 ports full-duplex fast Ethernet • Parallel port connection to host • MAC function: FPGA • All FPGAs programmed in Handel C • Handel C is a high-level language with ‘C’ syntax • Enhancements for FPGAs (e. g. , arbitrary width bit fields) • Simulation facility built into implementation • Output packets generated, time stamped • Incoming packets time stamped, CRC checked • Queue dwell times histogrammed • All of this full duplex @ full line speed DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 20

Fast Ethernet tester board DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 21

FE tester as ROB emulator • Accepts MESH ROB_DATA_REQUESTs • Generates MESH ROB_DATA_REPLYs • Programmable latency in ROB • Response size depends on ROI size and detector type • Response contents not meaningful • Measure latencies and queue depths DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 22

Large scale test bed using ROB emulator • Use 8 of these boards [11 K CHF each] to provide 256 emulated ROB ports • Add: - • 64 PCs - supervisors + farm nodes • Switch fabric • We have a ~15% scale test bed of Atlas LVL 2 • Previous test bed: order of magnitude smaller DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 23

Future activity • Measurements - • Improved switch characterization • Modeling - • Parameterized model of GE switch (central switch) • Detailed modeling of ATLAS LVL 2 • Modeling of unified network to handle LVL 2 traffic + Event Filter (LVL 3 trigger) DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 24

Summary • TCP/IP is very expensive in terms of CPU utilization at rates required • MESH will likely be used in ATLAS LVL 2 trigger • GBE needs custom firmware in NIC to get required performance • Modeling in good agreement with single switch measurements • Large scale test bed minimizes uncertainty associated with behavior extrapolated from a few nodes • Hardware ROB emulation provides opportunity to see network behavior under realistic conditions DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 25