Taiji Managing Global User Traffic for LargeScale Internet

Taiji Managing Global User Traffic for Large-Scale Internet Services at the Edge David Chou, Tianyin Xu, Kaushik Veeraraghavan, Andrew Newell, Sonia Margulis, Lin Xiao, Pol Mauri Ruiz, Justin Meza, Kiryong Ha, Shruti Padmanabha, Kevin Cole, Dmitri Perelman

Facebook infrastructure is geo-distributed Data Center 1

User requests are routed via edge nodes Data Center Edge Node Problem: how to route users from edge to datacenters? 2

Routing users for efficient utilization Data Center Edge Node 3

Routing users for a fast experience Data Center Edge Node 4

Routing connected users for cache efficiency Data Center Edge Node 5

Sites are growing, site is changing, failure is common Data Center Edge Node 6

User demand is dynamic Edge Node Traffic 3. 5 M 3 M 2. 5 M RPS 10 x Site traffic changes greatly based on geography and time of day 2 M 1. 5 M 1 M 0. 5 M 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Time 7

Statically mapping edge-to-datacenter traffic 50% Data Center Edge Node 8

Utilization (%) Static mapping is inefficient and dangerous 2015 Dangerous Danger Zone Static routing is inefficient, error-prone, and time-consuming Inefficient Time 9

Taiji: User Traffic Management at the Edge • Continuous operation, reactive system • Balances datacenter utilization • Minimizes network latency • Leverages user connections for cache efficiency • Running in production for 4+ years 10

Agenda • Taiji Overview • Traffic Allocation • User Routing • Evaluation 11

Taiji Objectives • How much traffic to allocate from each edge to datacenter? • Balanced utilization • Minimize network latency • How to route users? • Cache efficiency Taiji Traffic Allocator Traffic Allocation Conn. Aware Routing User-to-Datacenter Mapping 12

Taiji Allocator decides traffic allocation Traffic Users Allocator Traffic Allocation Edge Nodes 50% Data Centers Policy 50% 40% 60% Conn. Aware Routing User-to-Datacenter Mapping 13

Connection-aware routing maps users to datacenters Users Traffic Allocator Traffic Allocation Edge Nodes 50% Data Centers Policy 50% 40% 60% Conn. Aware Routing User-to-Datacenter Mapping 14

Part 1 of Taiji: Traffic Allocator Users Edge Nodes • Goal: compute the traffic allocation accounting for utilization and network latency. Data Centers 15

Balancing utilization with latencyawareness Target Avg Utilization = Users Edge Node Traffic (RPS) Data Center Capacity (RPS) Total demand Total capacity • (4 + 4) / (6 + 4 + 6) • = 8 / 16 • =1/2 (higher network latency) 16

Balancing utilization with latencyawareness Target Avg Utilization = Users Total demand Total capacity Edge Node Traffic (RPS) 3 Data Center Capacity (RPS) (higher network latency) 1 1 3 • (4 + 4) / (6 + 4 + 6) • = 8 / 16 • =1/2 • Routing policy trades off network latency vs balance 17

Services want different routing Better policies Network Latency More Balanced DC Utilization 18

Constraint solving for traffic allocation • Constraint solver to solve traffic allocation problem • Easily reconfigurable for service owners to experiment • Unify the different closed-form solvers 19

Part 2 of Taiji: Connection-aware Routing • Goal: route connected users to the same datacenter Traffic Allocator Policy Traffic Allocation Conn. Aware Routing User-to-Datacenter Mapping 20

Intuition: Route buckets of connected users User Buckets • Initial approach: 21 k fixed-size user buckets • Small buckets break up social groups 21

Intuition: Route buckets of connected users User Buckets • Initial approach: 21 k fixed-size user buckets • Small buckets break up social groups • Large buckets sacrifice routing granularity 22

Connection-aware Routing • Goal: maximize routing connected users to the same datacenter • Approach: • Fine-grained buckets for routing accuracy • Route groups of related buckets for higher locality Traffic Allocator Policy Traffic Allocation Conn. Aware Routing User-to-Datacenter Mapping 23

Hierarchical structure encodes bucket relationships All Users • Leaf nodes are buckets • Inner nodes represents subtree of user buckets Inner Nodes …… … … Buckets • Key Characteristics: • User buckets contain 0. 01% of all users 24

How to assign user buckets? All Users • Stable Segment Assignment • Assigns large subtrees of buckets and splits few for fine-grained routing Inner Nodes …… … … Buckets 25

Recap • Problem: How to route users from edge to datacenters? • Traffic Allocator • Balances utilization • Minimizes network latency Taiji Traffic Allocator Traffic Allocation Conn. Aware Routing User-to-Datacenter Mapping • Connection-aware 26

Evaluation • Traffic Allocator • How well can Taiji balance utilization? • How optimal is network latency? • Connection-aware Routing • Does it improve cache Taiji Traffic Allocator Traffic Allocation Conn. Aware Routing User-to-Datacenter Mapping 27

Humans managing edge-to-datacenter routing Utilization (%) 2015 Time 28

Taiji balances utilization effectively 2019 Utilization (%) • Balances within 3%, 80% of the time • Balances during dynamic events Drain Undrain Time • Continuous deployment • Datacenter outages 29

Network RTT (ms) 0 43543. 044444 43543. 0027777778 Best of both worlds Time 43543. 961111 43543. 9194444444 43543. 877778 43543. 8361111111 43543. 794444 43543. 7527777778 43543. 711111 43543. 6694444444 43543. 6277777778 43543. 5861111111 43543. 544444 43543. 5027777778 43543. 461111 43543. 4194444444 43543. 377778 43543. 3361111111 43543. 294444 43543. 2527777778 43543. 211111 43543. 1694444444 43543. 1277777778 43543. 0861111111 Taiji optimizes network latency Average Network Latency 180 160 140 120 100 Taiji Lower Bound 2015 Static Random 80 60 40 20 Lower bound is theoretical and assumes infinite capacity 30

Taiji reduces demand on backend storage DB Reads (QPS) Conn. Aware Routing Rollout Days • 17% reduction in reads to backend storage 31

Limitations • Taiji only considers edge-to-datacenter latency • Service owners specify routing policy to optimize end -to-end latency • Taiji only considers frontend utilization • Continuous loadtesting to ensure backend sized properly 32

Conclusion • Running in production for 4+ years, reacts to daily site dynamics • Flexible configuration to trade off balanced utilization with network latency optimization • Routing connected users reduced reads on backend storage by 17% 33