ABw E Available Bandwidth Estimator Jiri Navratil R

ABw. E: Available Bandwidth Estimator Jiri Navratil R. Les. Cottrell Stanford Linear Accelerator Center (SLAC), 2575 Sand Hill Road, Menlo Park, California 94025 jiri@slac. stanford. edu, cottrell@slac. stanford. edu

ABw. E: Available Bandwidth Estimator • Introduction (motivation, needs, . . ) • Basic principles • Path characteristics and the examples of packet pair dispersion delays • Bandwidth estimation • ABw. E versus Iperf • Conclusions

Introduction 1 • The HEP community is increasingly dependent on networking as internal cooperation grows (needs transfer huge amount of data between experimental sites as SLAC, CERN, etc. and home institutes spread over the world) • Our main task is to provide the physicists reliable access to the network (and the integral part of this activity is NETWORK MONITORING) • We have several monitoring system in operation (active: as ping or iperf, and passive: reading SNMP counts or using netflows data

Introduction 2 • Network administrators and the users need to know RTT, losses, routing path, and estimations of available bandwidth to our partners • Currently we have such information in limited sampling periods. The big question is. Do we have valid information if we do measurements once per 90 minutes and can we do measurements with tools as Iperf or to transfer the test files more frequently? Probably no. • We need a tool that could be used in continuous mode 24 hours a day 7 days a week which can quickly and non intrusively detect changes on multiple path

Specification • Tool based on dispersion techniques which doesn’t pollute Internet (and overload an entry point to the Internet) with huge amount of testing packets • Get the result from one path during in a few seconds and produce results that could be easily preprocessed by graphical tools or enter to other systems (prediction, warning etc. ) • Easily configurable and manageable from one site • We evaluated several tools using dispersion techniques but none of them in their current implementation met our demands. (Some of them were slow, some of them failed for high capacity paths and some of them were just technically too complicated).

Basic principles of ABw. E • ABw. E is based on the simplest way of probing (using only Packet pairs) • Evaluation is based on detailed technical analysis of how the packets pass via queuing devices • Complete path is cascade of queuing devices with different capacities • The separation of probing packets will happen even if there is no cross traffic • The final dispersion PP 1 and PP 2 is the results of superposition of many factors

How we measure the dispersion time • Using Netdyn package (package from University of Maryland 1991) • 20 packets pairs probes for each path • Probes are repeated with the period 20 msec and once a minute per each path. Set of 20 probes is called bunch. The bunch is evaluated as one statistical set of measurements. Linux Send PP 1 Send PP 2 dt_send (7 -25 ms) t 2 Path-1 NIC Lpp/C 2 3 . . 20 20 ms t 1 Path-2 time 1 0 1 20 ms bunch

ABw. E Basic principles (The simple linear situations) time PP 2 PP 1 rtt Td-send S Probe-Sender Probe Receiver R hop T-stamp Cross-traffic input CT packets Td-receive PP 1 time PP 2 PP 1 T-stamp PPD = Td-receive - Td-send , Td-receive T-stamp ( Td-receive >= Td-send ) REAL ratio between generally used long and short packets Dynamic Dispersion Delay

CESNET SLAC NERSC CERN APAN CALTECH

Detail Timing for PP on the way via experimental path (stretching, compressing and contracting) PP 2 PP 1 time move direction Td-send Probe-Sender H 1 1000 Mbps S H 2 622 Mbps H 3 622 Mbps 155 Mbps hop Cross-traffic H 4 1000 Mbps hop Cross-traffic Input-H 1 PP 2 PP 1 OUT-H 1 time Input-H 2 Output-H 2 (stretching) Input-H 3 Td 23 Free spaces for Cross-traffic Output-H 3 (contracting) Input-H 4 Td 34 PP 2 Td-receive Td 23 = LPP/C 23 Free spaces for cross-traffic Static Dispersion Delay Td-receive = Td 34 = Td 23 PP 1 R

(Td)

NTT - Normalized Transfer Time

What type of traffic we can expect on the path

ABw. E “Narrow Band” hop characteristics PP 2 PP 1 H 2 622 Mbps hop Td 12 Ex. 1: “Stretching + absorbing Td effect ” a) Different input T d 12 same output Td 23 PP 2 155 Mbps H 3 hop Cross-traffic (622 Mbps lines) PP 1 Input-H 2 Output H 2 (stretching PP) Input-H 2 b) Td not changed Input-H 2 Td 23 Absorbing CT Td 23 Td 12 Output H 2 (stretching PP)

The principle of gradually narrowing bandwidth Impact No impact load 1000 622 load 622 1000 155 Light beam Light source 622 622 Remarks: -Fully valid and easily applicable for continuous streams or for data with strong source - not easily applicable for immediate situation on the path with Poisson’s traffic or heavy bursty traffic with lights periods

ABw. E “Narrow Band” hop characteristics time PP 2 PP 1 H 2 622 Mbps Td 12 Input-H 2 PP 2 155 Mbps H 3 hop Cross-traffic (622 Mbps lines) PP 1 Cross-traffic Output H 2 (stretching PP) Td 23 Ex. 2: “Multiplication Input-H 3 (no cross-traffic from H 2) effect” Td 23 Output H 3 CT packets from H 2 Input – H 3 time PP 2 Td 3 ~ n*Td 23 Output H 3 PP 1

Multiplication factor in Td

Example of very low NB-Narrow Band

Example of superposition on Low NB 16 + - 100 + - 622 Mbits Tdmin 551, 0 Tdmin + - n*64 Mbits 726, 6

Traceroute Graph of Monitoring paths According to the previous paragraphs an each path has: Static Dispersion Delay and given by the hops I/O capacities in the tree Dynamical Dispersion Delay caused by CT

Can we convert Td to bandwidth (Capacity) ? What does it represent ? We know that If r grows than If linear than If non-linear We Td ~ r (utilization factor) Td grows C ~ K * 1/Td have problem But we know that we deal with Bottleneck band hop which can: - eliminate previous Td - replace them by own Td ~ Lpacket/C - Queuing start to play important role Use non linear solution

The principle of gradually narrowing bandwidth Impact No impact load 1000 622 load 622 1000 155 Light beam Light source 622 622

The principle of gradually narrowing bandwidth Impact No impact load 1000 622 load 622 1000 155 Light beam Light source 622 622 We assume that: Most of the time only one queue dominates in the instant of our measurements !

The principle of gradually narrowing bandwidth Impact No impact load 1000 622 622 1000 155 Light beam Light source 622 622 We assume that: Most of the time only one queue dominates in the instant of our measurements !

All “probing packets PP” share same queue with outside Cross. Traffic Queue in Node-x (It means that Td caused by Queuing is not dependent only on the pkt_lengths of PP) Open question: What to use for estimation of CT (the average pkt_length or pkt_length close to MTU) ?

From the Queuing theory for M/M/1: Tsojourn= (1+E(N)) Tservice and in this we replace Tsojourn ~ Td , Tservice = Lp/C and also use LPP and LCT instead of Lp Tdi = LPP/Ci + Td = Tdinit E(N)* LCT/Ci + Tdvar (1) Tdjinit. Td = min=i (min (Tdij)( |(Td 1<i<=20 ) ) | 1<i<=20 ) jinit i ij QDFi = (Tdij – Tdjinit)/NTTclass this allows us to replace E(N) in formula (1) by QDF: Tdij = LPP /Ci + QDFj * LCT/Ci From this we can calculate Ci for each singleton in one bunch j. Ci = (LPP + QDFj * LCT)/ Tdij (2)

Graphical interpretation of the formula Ci = LPP /Tdi + QDFi * LCT/Tdi Cmax= Lpp/Tdmin C [Mbits/s] (when no CT, QDF=0) Ci Time [s]

EWMA: Filtration characteristics (avgi = (1 – a) *yi +a* avgi-1 )

Monitoring sites (average values)

Iperf versus ABw. E (few unclear points) • How to configure Iperf to achieve maximum performance in changing environment ( difference ~ 10 - 100 %) • Limitation on the Entry-points to the Internet (SLAC 622 Mbits, customer load (10% - 40% ) • Machine performance (400 -550 Mbits) • Iperf aggressiveness (it suppress bandwidth of other running applications) and reports all what Iperf transferred • Synchronization problem to avoid dependency

ABw. E compare with Iperf

ABw. E compare with Iperf

ABw. E compare with Iperf

ABw. E compare with Iperf

ABw. E compare with Iperf

Conclusions • We have demonstrated several network analysis and a new method for monitoring ABw and bottleneck capacity in the range several Mbits to 1000 Mbits • ABw. E is a non intrusive method which can be run in a continuous mode 24 hours a day 7 days a week • It can detect changes in the path capacity based on heavy traffic and also discover dramatic changes in routing. The usefulness of ABw. E has been proven several times since last summer • Unfortunately, ABw. E still doesn’t exists as a publicly used tool