FAST TCP in Linux Cheng Jin David Wei

FAST TCP in Linux Cheng Jin David Wei http: //netlab. caltech. edu/FAST/WANin. Lab/nsfvisit

Outline o o Overview of FAST TCP. Implementation Details. SC 2002 Experiment Results. FAST Evaluation and WAN-in-Lab. netlab. caltech. edu

FAST vs. Linux TCP Flows Linux TCP Bmps Peta Throughput (Mbps) Transfer (GB) 1 1. 86 185 78 1 2. 67 266 111 FAST 19. 11. 2002 1 9. 28 925 387 Linux TCP 2 3. 18 317 133 2 9. 35 931 390 2 18. 03 1, 797 753 txqlen=100 Linux TCP txqlen=10000 FAST 19. 11. 2002 Distance = 10, 037 km; Delay = 180 ms; MTU = 1500 B; Duration: 3600 s Linux TCP Experiments: Jan 28 -29, 2003 netlab. caltech. edu

Aggregate Throughput 92% FAST o Standard MTU o Utilization averaged over 1 hr 2 G 48% Average utilization 95% 1 G 27% 16% 19% txq=10000 Linux TCP netlab. caltech. edu FAST Linux TCP FAST

Summary of Changes o o RTT estimation: fine-grain timer. Fast convergence to equilibrium. Delay monitoring in equilibrium. Pacing: reducing burstiness. netlab. caltech. edu

FAST TCP Flow Chart Fast Convergence Slow Start Normal Recovery Equilibrium Time-out Loss Recovery netlab. caltech. edu

RTT Estimation o o Measure queueing delay. Kernel timestamp with s resolution. Use SACK to increase the number of RTT samples during recovery. Exponential averaging of RTT samples to increase robustness. netlab. caltech. edu

Fast Convergence o o Rapidly increase or decrease cwnd toward equilibrium. Monitor the per-ack queueing delay to avoid overshoot. netlab. caltech. edu

Equilibrium o o o Vegas-like cwnd adjustment in large time-scale -- per RTT. Small step-size to maintain stability in equilibrium. Per-ack delay monitoring to enable timely detection of changes in equilibrium. netlab. caltech. edu

Pacing o What do we pace? n o Increment to cwnd. Time-Driven vs. event-driven. n n Trade-off between complexity and performance. Timer resolution is important. netlab. caltech. edu

Time-Based Pacing data ack data cwnd increments are scheduled at fixed intervals. netlab. caltech. edu

Event-Based Pacing Detect sufficiently large gap between consecutive bursts and delay cwnd increment until the end of each such burst. netlab. caltech. edu

SCinet Caltech-SLAC experiments SC 2002 Baltimore, Nov 2002 unn yva le Highlights FAST TCP • Standard MTU • Peak window = 14, 255 pkts • Throughput averaged over > 1 hr • 925 Mbps single flow/GE card 10 2 1 I 2 LSR 7 1 #flows 9. 28 petabit-meter/sec 1. 89 times LSR • e al 34. 0 petabit-meter/sec 6. 32 times LSR v ny n Su e- • or tim l Ba Internet: distributed feedback system R (s) Theory Experiment f AQM TCP Rb’(s) netlab. caltech. edu/FAST 21 TB in 6 hours with 10 flows Implementation • Sender-side modification • Delay based 2 x 8. 6 Gbps with 10 flows p Geneva 7000 km FAST Ge nev a-S 9 Sunnyvale 3000 km Baltimore Chicago 1000 km C. Jin, D. Wei, S. Low FAST Team and Partners

Network (Sylvain Ravot, caltech/CERN) netlab. caltech. edu

ale FAST BMPS ny v 10 Ge ne va -S un 9 FAST 7 2 le a yv nn 1 Internet 2 Land Speed Record netlab. caltech. edu 1 2 e or u -S tim l Ba FAST o Standard MTU o Throughput averaged over > 1 hr #flows

Aggregate Throughput 88% FAST o Standard MTU o Utilization averaged over > 1 hr 90% Average utilization 92% 95% 1 hr 1 flow netlab. caltech. edu 1 hr 2 flows 6 hr 7 flows 1. 1 hr 9 flows 6 hr 10 flows

Caltech-SLAC Entry Power glitch Reboot Rapid recovery after possible hardware glitch 100 -200 Mbps ACK traffic netlab. caltech. edu

SCinet Caltech-SLAC experiments Acknowledgments SC 2002 Baltimore, Nov 2002 netlab. caltech. edu/FAST • Prototype C. Jin, D. Wei • Theory D. Choe (Postech/Caltech), J. Doyle, S. Low, F. Paganini (UCLA), J. Wang, Z. Wang (UCLA) • Experiment/facilities • Caltech: J. Bunn, C. Chapman, C. Hu (Williams/Caltech), H. Newman, J. Pool, S. Ravot (Caltech/CERN), S. Singh • CERN: O. Martin, P. Moroni • Cisco: B. Aiken, V. Doraiswami, R. Sepulveda, M. Turzanski, D. Walsten, S. Yip • Data. TAG: E. Martelli, J. P. Martin-Flatin • Internet 2: G. Almes, S. Corbato • Level(3): P. Fernes, R. Struble • SCinet: G. Goddard, J. Patton • SLAC: G. Buhrmaster, R. Les Cottrell, C. Logg, I. Mei, W. Matthews, R. Mount, J. Navratil, J. Williams • Star. Light: T. de. Fanti, L. Winkler • Tera. Grid: L. Winkler • Major sponsors ARO, CACR, Cisco, Data. TAG, Do. E, Lee Center, NSF

Evaluating FAST o o o End-to-End monitoring doesn’t tell the whole story. Existing network emulation (dummynet) is not always enough. Better optimization if we can look inside and understand the real network. netlab. caltech. edu

Dummynet and Real Testbed netlab. caltech. edu

Dummynet Issues o o Not running on a real-time OS -imprecise timing. Lack of priority scheduling of dummynet events. Bandwidth fluctuates significantly with workload. Much work needed to customize dummynet for protocol testing. netlab. caltech. edu

10 Gb. E Experiment o o o Long-distance testing of Intel 10 Gb. E cards. Sylvain Ravot (Caltech) achieved 2. 3 Gbps usingle stream with jumbo frame and stock Linux TCP. Tested HSTCP, Scalable TCP, FAST, and stock TCP under Linux. 1500 B MTU: 1. 3 Gbps SNV -> CHI; netlab. caltech. edu 9000 B MTU: 2. 3 Gbps SNV -> GVA

TCP Loss Mystery o o Frequent packet loss with 1500 -byte MTU. None with larger MTUs. Packet loss even when cwnd is capped at 300 - 500 packets. Routers have large queue size of 4000 packets. Packets captured at both sender and receiver using tcpdump. netlab. caltech. edu

How Did the Loss Happen? loss detected netlab. caltech. edu

How Can WAN-in-Lab Help? o o o We will know exactly where packets are lost. We will also know the sequence of events (packet arrivals) that lead to loss. We can either fix the problem in the network if any, or improve the protocol. netlab. caltech. edu

Conclusion o o o FAST improves the end-to-end performance of TCP. Many issues are still to be understood and resolved. WAN-in-Lab can help make FAST a better protocol. netlab. caltech. edu