BBR An Implementation of Bottleneck Bandwidth and Roundtrip

BBR' - An Implementation of Bottleneck Bandwidth and Round-trip Time Congestion Control for ns-3 Mark Claypool, Feng Li and Jae Chung In Proceedings of the Workshop on ns-3 (WNS 3) Mangalore, India, June 2018 http: //www. cs. wpi. edu/~claypool/papers/bbr-prime/ 1

Introduction • TCP dominant protocol on Internet • Yesterday • Designed in an era of limited resources – low network bandwidths, small queues • Packet loss a sign of congestion • Today • Router queues can be large (buffer bloat) • Packet loss can be from wireless errors https: //image. slidesharecdn. com/bufferbloatpresentation-140602070926 -phpapp 02/95/effekterna-avbufferbloat-och-latency-p-internet-4 -638. jpg? cb=1401693007 https: //image. slidesharecdn. com/7 -140402115120 -phpapp 02/95/7 -wang-jingyuan-tcpfit-4 -638. jpg? cb=1396439521 Revisit model of congested link 2

Congestion and Bottlenecks 3

Congestion and Bottlenecks 4

Congestion and Bottlenecks 5

Congestion and Bottlenecks 6

Congestion and Bottlenecks Bandwidth-Delay Product 7

Estimating Optimal Point 8

Estimating Optimal Point To see both max BW and min RTT Must probe on both sides of BDP! 9

Estimating Optimal Point N. Cardwell, Y. Cheng, C. S. Gunn, S. H. Yeganeh, and V. Jacobson. BBR: Congestion -Based Congestion Control, Communications of the ACM, 60(2), February 2017. • BBR – Bottleneck Bandwidth and Round-trip time • Model network • Each ACK, update max BW and min RTT • Control sending based on model • Probe for max BW and min RTT • Pace near estimate BW • Keep inflight near BDP • Goal: high throughput with small bottleneck queue 10

Outline • Introduction • BBR’ • Validation • Evaluation • Conclusions (done) (next) 11

BBR – Walk towards max BW, min RTT 12

BBR – STARTUP: exponential search 13

BBR – DRAIN: drain queue built up in STARTUP 14

BBR – PROBE_BW: explore max BW, drain, cruise 15

BBR – PROBE_RTT: check if min RTT changed 16

BBR – State Machine Warm-up • 2 Phases of probing • Increase inflight to get max BW • Lower inflight to get min RTT Apply during Warm-up and Steady-state • Warm-up • Startup – ramp up quickly (ala Slow Start) • Drain – drain built up queue • Steady-state • Probe. BW – cycle pacing rate to probe BW • Probe. RTT – if needed, coordinated dip to probe RTT Steady-state 17

BBR – Median Latency versus Buffer Size BBR vs. CUBIC. Synthetic bulk download. 8 flows, bottleneck 128 Kb/s, RTT 40 ms 18

BBR – Mean Goodput versus Loss Rate BBR vs. CUBIC. Synthetic bulk download. 1 flow, bottleneck 100 Mb/s, RTT 100 ms 19

BBR Status of BBR v 1. 0 • Deployed widely at Google • Open source for Linux TCP and QUIC • Documented in IETF Internet Drafts Actively working on BBR v 2. 0 • ACK aggregation (e. g. , Wi. Fi) • Packet loss signals • Dynamics of sharing with lossbased congestion control • Free. BSD TCP @ Net. Flix https: //datatracker. ietf. org/meeting/101/materials/slides-101 -iccrg-an-update-on-bbr-work-at-google-00 20

ns – The Network Simulator https: //www. nsnam. org/ • Discrete event simulators • ns-1 (LBNL) • ns-2 (DARPA) • ns-3 (NSF and INRIA) 1995 – 2001 2000 – 2010 2008 – present • Educational and Research use • Yearly workshop (WNS 3) https: //tinyurl. com/y 7 gof 6 p 6 • Share ideas, code, research Does not support BBR! Implementation of BBR for ns-3 21

Outline • Introduction • BBR’ • Validation • Evaluation • Conclusions (done) (next) 22

BBR’ Implementation Developed based on specifications: N. Cardwell, Y. Cheng, S. H. Yeganeh, and V. Jacobson. BBR Congestion Control, IETF Draft, July 2017. Y. Cheng, N. Cardwell, S. H. Yeganeh, and V. Jacobson. Delivery Rate Estimation, IETF Draft, July 2017. • Protocol states • Solution: state machine • BDP computation • Solution: compute each ACK • Delivery rate estimation • Solution: hook to record when send • Packet pacing • Solution: “hijack” at TCP layer 23

BBR’ Implementation 1. 2. 3. 4. 5. 6. 7. 8. 9. BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send? Callback to APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send BBR’ APP Tcp. Bbr. State. Machine Bulk. Send. Application TCP Tcp. Socket. Base Tcp. Socket. State 24

BBR’ Implementation 1. 2. 3. 4. 5. 6. 7. 8. 9. BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send? Callback to APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send BBR’ APP Tcp. Bbr. State. Machine 1 Pkts. Acked() Bulk. Send. Application TCP Tcp. Socket. Base Tcp. Socket. State 25

BBR’ Implementation 1. 2. 3. 4. 5. 6. 7. 8. 9. BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send? Callback to APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send BBR’ APP Tcp. Bbr. State. Machine 1 Pkts. Acked() 2 Set. Pacing. Rate() Bulk. Send. Application TCP Tcp. Socket. Base Tcp. Socket. State 26

BBR’ Implementation 1. 2. 3. 4. 5. 6. 7. 8. 9. BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send? Callback to APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send BBR’ tcb → m_cwnd 3 Tcp. Bbr. State. Machine 1 Pkts. Acked() 2 Set. Pacing. Rate() APP Bulk. Send. Application TCP Tcp. Socket. Base Tcp. Socket. State 27

BBR’ Implementation 1. 2. 3. 4. 5. 6. 7. 8. 9. BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send? Callback to APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send BBR’ tcb → m_cwnd 3 Tcp. Bbr. State. Machine 1 Pkts. Acked() 2 Set. Pacing. Rate() APP Bulk. Send. Application TCP Tcp. Socket. Base Tcp. Socket. State Data. Send() 4 28

BBR’ Implementation 1. 2. 3. 4. 5. 6. 7. 8. 9. BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send? Callback to APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send BBR’ tcb → m_cwnd 3 Tcp. Bbr. State. Machine 1 Pkts. Acked() 2 Set. Pacing. Rate() APP Bulk. Send. Application TCP Tcp. Socket. Base Tcp. Socket. State 5 Send() Data. Send() 4 29

BBR’ Implementation 1. 2. 3. 4. 5. 6. 7. 8. 9. BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send? Callback to APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send BBR’ tcb → m_cwnd 3 APP Tcp. Bbr. State. Machine 1 Pkts. Acked() 2 Set. Pacing. Rate() Bulk. Send. Application TCP Tcp. Socket. Base Tcp. Socket. State Send. Data. Packet() 5 Send() Data. Send() 4 6 30

BBR’ Implementation 1. 2. 3. 4. 5. 6. 7. 8. 9. BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send? Callback to APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send BBR’ tcb → m_cwnd 3 APP Tcp. Bbr. State. Machine 1 Pkts. Acked() 2 Set. Pacing. Rate() Bulk. Send. Application TCP Tcp. Socket. Base Tcp. Socket. State 5 Send() Data. Send() 4 (Timer) Send. Data. Packet() 6 7 Pace. Packets() 31

BBR’ Implementation 1. 2. 3. 4. 5. 6. 7. 8. 9. BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send? Callback to APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send BBR’ Tcp. Bbr. State. Machine tcb → m_cwnd 3 APP Bulk. Send. Application 8 Send() 1 Pkts. Acked() 2 Set. Pacing. Rate() TCP Tcp. Socket. Base Tcp. Socket. State 5 Send() Data. Send() 4 (Timer) Send. Data. Packet() 6 7 Pace. Packets() 32

BBR’ Implementation 1. 2. 3. 4. 5. 6. 7. 8. 9. BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send? Callback to APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send BBR’ Tcp. Bbr. State. Machine tcb → m_cwnd 3 APP Bulk. Send. Application 8 Send() 1 Pkts. Acked() 2 Set. Pacing. Rate() TCP Tcp. Socket. Base Tcp. Socket. State 5 Send() Data. Send() 4 (Timer) Send. Data. Packet() 6 7 Pace. Packets() Send. Data. Packet. Real() 9 33

BBR’ Implementation 1. 2. 3. 4. 5. 6. 7. 8. 9. BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send? Callback to APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send BBR’ Tcp. Bbr. State. Machine tcb → m_cwnd 3 APP Bulk. Send. Application 8 Send() 1 Pkts. Acked() 2 Set. Pacing. Rate() TCP Tcp. Socket. Base Tcp. Socket. State 5 Send() Data. Send() 4 (Timer) Send. Data. Packet() 6 7 Pace. Packets() Send. Data. Packet. Real() 9 34

Outline • Introduction • BBR’ • Validation • Evaluation • Conclusions (done) (next) 35

Verification: BBR’ – Basic Bottleneck Single flow. Bottleneck: 10 Mb/s, 22 ms RTT BBR’ works as expected (based on specification) 36

Verification: BBR’ – Basic Bottleneck (Estimated versus Actual) BBR’ accurately infers RTT and BW 37

Validation: BBR’ versus BBR – Steady State Single flow. Bottleneck: 10 Mb/s, 40 ms RTT BBR’ looks similar to BBR 38

Validation: BBR’ versus BBR – Bwidth Increase Single flow. Bottleneck: 40 ms RTT. Time 20: 10 Mb/s 20 Mb/s BBR’ looks similar to BBR 39

Validation: BBR’ versus BBR – Bwidth Decrease Single flow. Bottleneck: 40 ms RTT. Time 40: 20 Mb/s 10 Mb/s BBR’ looks similar to BBR 40

Outline • Introduction • BBR’ • Validation • Evaluation • Conclusions (done) (next) 41

Evaluation: BBR’ vs. CUBIC – Wired BBR’ has similar Throughput, but lower RTT 42

Evaluation: BBR’ vs. CUBIC – Wireless 5 km Results for 4 G LTE similar to wired 43

Evaluation: BBR’ vs. CUBIC – Wireless 0 – 20 km BBR’ has similar Throughput, but lower RTT with distance 44

Conclusions • Evolution of networks needs evolution of TCP • BBR promising, but needs more vetting • ns-3 network simulator great for vetting, but no BBR • BBR’ – Implementation of BBR for ns-3 • Use existing CC code base • Integrate with other NS layers • Validation – looks like BBR • Evaluation compared to CUBIC • Similar throughputs • Lower RTTS Acknowledgements Apoova Lad and Chengle Zhang for early work 45

Future Work • Incorporate more BBR features • Application limited flows • Adverse Internet conditions – packet reordering, ACK loss • Adjustments of BBR parameters • Pacing • Gain rates • Evaluation over more network conditions • Competing protocol stacks • Wireless (Wi. Fi and LTE) • Queue sizes (small, super large) 46

BBR' - An Implementation of Bottleneck Bandwidth and Round-trip Time Congestion Control for ns-3 Mark Claypool, Feng Li and Jae Chung In Proceedings of the Workshop on ns-3 (WNS 3) Mangalore, India, June 2018 http: //www. cs. wpi. edu/~claypool/papers/bbr-prime/ 47