Congestion Control for Highspeed Extremely Shallowbuffered Datacenter Networks

Congestion Control for High-speed Extremely Shallow-buffered Datacenter Networks Wei Bai, Kai Chen, Shuihai Hu, Kun Tan, Yongqiang Xiong APNet 2017, Hong Kong 1

Data Center Applications • Network requirements – Desire low latency for small messages – Desire high throughput for large flows The challenge is to achieve both goals simultaneously. 2

Tension Between Requirements • High throughput: large switch buffer occupancies • Low latency: small switch buffer occupancies – Reduce queueing delay – Reduce packet losses with large headroom Occupancy Size Headroom Time 3

Current Practice • Dynamic buffer allocation at switch – Reduce packet losses Shared Buffer Pool 1 2 3 4 5 6 7 8 4

Current Practice • Dynamic buffer allocation at switch – Reduce packet losses • ECN-based transports 5

Current Practice • Dynamic buffer allocation at switch – Reduce packet losses • ECN-based transports Occupancy Size K Time 6

Current Practice • 7

Current Practice • Basic Buffer Requirement 8

Current Practice • Is current practice good enough? No with recent trends! Basic Buffer Requirement 9

Recent Trends in DCNs • The link speed scales up – TCP’s basic buffer requirement also scales up. • The switch buffer does not increase expectedly – Reasons: cost, price, etc. 1 Gbps 10 Gbps 40 Gbps 100 Gbps 80 KB 192 KB 384 KB 512 KB Buffer / port of Broadcom chips 10

Observation • More and more shallow switch buffer Buffer / port / Gbps (KB） – Buffer port per Gbps keeps decreasing 80 60 40 20 0 1 10 40 Link Speed 100 11

Observation • More and more shallow switch buffer Buffer / port / Gbps (KB） – Buffer port per Gbps keeps decreasing 80 Extremely Shallow-buffered DCNs 60 40 20 0 1 10 40 Link Speed 100 12

Problems of Existing Solutions (1) • Buffer Occupancy K Time 13

Problems of Existing Solutions (1) • Egress Ports 1 2 3 4 5 6 7 8 14

Problems of Existing Solutions (1) • Active Ports 1 15

Problems of Existing Solutions (1) • Active Ports 1 2 3 4 16

Problems of Existing Solutions (1) • Packet Losses Active Ports 1 2 3 4 5 17

Problems of Existing Solutions (1) • 18

Problems of Existing Solutions (2) • Conservative ECN configuration – Leave headroom for low packet loss rate Buffer Occupancy Avg. Buffer / Port K Time 19

Problems of Existing Solutions (2) • Conservative ECN configuration – Leave headroom for low packet loss rate Active Ports 1 2 3 4 5 6 7 8 20

Problems of Existing Solutions (2) • Conservative ECN configuration – Leave headroom for low packet loss rate Active Ports 1 21

Problems of Existing Solutions (2) • Conservative ECN configuration – Leave headroom for low packet loss rate – Significant throughput degradation with few active ports 22

Summary of Problems • 23

Design Goals • High throughput • Low packet loss rate • When many ports are active? – Packet loss rate prioritized over throughput • Readily-deployable – Legacy network stacks & commodity switch ASIC 24

Our Solution • High throughput • Low packet loss rate • When many ports are active? – Packet loss rate prioritized over throughput • Readily-deployable – Legacy network stacks & commodity switch ASIC Buffer-aware Congestion Control 25

BCC Mechanisms • End-host – Legacy ECN-based transports • Switch – Per port standard ECN configuration – Shared buffer ECN/RED OR 26

How BCC Works Shared Buffer ECN/RED Egress Ports 1 2 3 4 5 6 7 8 27

Few Active Ports • Per port standard ECN configuration ensures high throughput & low packet loss rate Active Ports 1 2 3 4 5 6 7 8 28

Many Active Ports • Shared buffer ECN/RED achieves low packet loss rate at the cost of a small throughput loss Active Ports 1 2 3 4 5 6 7 8 29

BCC in 1 Slide • Buffer Aware 30

BCC in 1 Slide • 31

More Details in the Paper • Probabilistic marking – Reduce throughput loss due to flow synchronization • Parameter selection for BCC • Impact of many factors – Multiple MMUs – Static reserved buffers – Different per-queue shared buffer configuration 32

Testbed Validation • Arista 7060 CX-32 S switch(config)# qos random-detect ecn global-buffer minimum-threshold 500 kbytes maximum-threshold 500 kbytes 33

Large Scale Simulations • Settings: – 128 -host 100 Gbps spine-leaf fabric – Realistic web search traffic • Schemes compared – Standard per port ECN/RED (K = 720 KB) – Conservative per port ECN/RED (K = 200 KB) • Metrics: – Flow completion time (FCT) & packet loss rate 34

Packet Loss Rate BCC keeps low packet loss rate 35

99 th percentile FCT for Flows <100 KB RTO: 5 ms BCC keeps low packet loss rate 36

Average FCT for Flows > 10 MB BCC only trades a little throughput 37

BCC Recap • Abundant buffer – Deliver high throughput & low packet loss rate • Scarce buffer – Trade a little throughput for low packet loss rate • Readily-deployable – One more ECN configuration is enough 38

Thanks! 39