Deconstructing Datacenter Packet Transport Mohammad Alizadeh Shuang Yang

Deconstructing Datacenter Packet Transport Mohammad Alizadeh, Shuang Yang, Sachin Katti, Nick Mc. Keown, Balaji Prabhakar, Scott Shenker Stanford University Hot. Nets 2012 U. C. Berkeley/ICSI 1

Transport in Datacenters • Latency is King Who does she know? What has Large-scale she done? – Web app response time depends on completion of 100 s of small RPCs Web Application App Alice Logic Logic Logic App Tier Fabric • But, traffic also diverse – Mice AND Elephants – Often, elephants are the root cause of latency Hot. Nets 2012 Data Tier Eric Minnie Pics Apps Videos 2

Transport in Datacenters • Two fundamental requirements – High fabric utilization • Good for all traffic, esp. the large flows – Low fabric latency (propagation + switching) • Critical for latency-sensitive traffic • Active area of research – DCTCP[SIGCOMM’ 10], D 3[SIGCOMM’ 11] HULL[NSDI’ 11], D 2 TCP[SIGCOMM’ 12] PDQ[SIGCOMM’ 12], De. Tail[SIGCOMM’ 12] Hot. Nets 2012 vastly improve performance, but fairly complex 3

p. Fabric in 1 Slide Packets carry a single priority # • e. g. , prio = remaining flow size p. Fabric Switches • Very small buffers (e. g. , 10 -20 KB) • Send highest priority / drop lowest priority pkts p. Fabric Hosts • Send/retransmit aggressively • Minimal rate control: just prevent congestion collapse Hot. Nets 2012 4

DC Fabric: Just a Giant Switch! H 1 H 2 Hot. Nets 2012 H 3 H 4 H 5 H 6 H 7 H 8 H 9 5

DC Fabric: Just a Giant Switch! H 1 H 2 Hot. Nets 2012 H 3 H 4 H 5 H 6 H 7 H 8 H 9 6

H 2 H 3 H 4 H 5 H 3 H 6 H 2 H 8 H 9 H 7 H 1 TX H 1 DC Fabric: Just a Giant Switch! Hot. Nets 2012 RX 7

H 2 H 3 H 4 H 5 H 3 H 6 H 2 H 8 H 9 H 7 H 1 TX H 1 DC Fabric: Just a Giant Switch! Hot. Nets 2012 RX 8

DC transport = Flow scheduling on giant switch Objective? Ø Minimize avg FCT H 1 H 2 H 3 H 4 H 5 H 6 H 7 ingress & egress capacity constraints H 9 Hot. Nets 2012 H 8 TX RX 9

“Ideal” Flow Scheduling Problem is NP-hard [Bar-Noy et al. ] – Simple greedy algorithm: 2 -approximation Hot. Nets 2012 1 1 2 2 3 3 10

p. Fabric Design Hot. Nets 2012 11

p. Fabric Switch Ø Priority Scheduling send higher priority packets first 5 9 4 3 7 prio = remaining flow size Hot. Nets 2012 Ø Priority Dropping drop low priority packets first Switch Port 1 small “bag” of packets per-port 12

Near-Zero Buffers • Buffers are very small (~1 BDP) – e. g. , C=10 Gbps, RTT=15µs → BDP = 18. 75 KB – Today’s switch buffers are 10 -30 x larger Priority Scheduling/Dropping Complexity • Worst-case: Minimum size packets (64 B) – 51. 2 ns to find min/max of ~300 numbers – Binary tree implementation takes 9 clock cycles – Current ASICs: clock = 1 -2 ns Hot. Nets 2012 13

p. Fabric Rate Control • Priority scheduling & dropping in fabric also simplifies rate control – Queue backlog doesn’t matter One task: Prevent congestion collapse when elephants collide H 1 Hot. Nets 2012 50% Loss H 2 H 3 H 4 H 5 H 6 H 7 H 8 H 9 14

p. Fabric Rate Control • Minimal version of TCP 1. Start at line-rate • Initial window larger than BDP 2. No retransmission timeout estimation • Fix RTO near round-trip time 3. No fast retransmission on 3 -dupacks • Allow packet reordering Hot. Nets 2012 15

Why does this work? Key observation: Need the highest priority packet destined for a port available at the port at any given time. • Priority scheduling Ø High priority packets traverse fabric as quickly as possible • What about dropped packets? Ø Lowest priority → not needed till all other packets depart Ø Buffer larger than BDP → more than RTT to retransmit Hot. Nets 2012 16

Evaluation • 54 port fat-tree: 10 Gbps links, RTT = ~12µs • Realistic traffic workloads – Web search, Data mining <100 KB 55% of flows 3% of bytes Hot. Nets 2012 * From Alizadeh et al. [SIGCOMM 2010] >10 MB 5% of flows 35% of bytes 17

Evaluation: Mice FCT (<100 KB) Average 99 th Percentile Near-ideal: almost no jitter Hot. Nets 2012 18

Evaluation: Elephant FCT (>10 MB) Congestion collapse at high load w/o rate control Hot. Nets 2012 19

Summary p. Fabric’s entire design: Near-ideal flow scheduling across DC fabric • Switches – Locally schedule & drop based on priority • Hosts – Aggressively send & retransmit – Minimal rate control to avoid congestion collapse Hot. Nets 2012 20

Thank You! Hot. Nets 2012 21

Hot. Nets 2012 22