Gaia GeoDistributed Machine Learning Approaching LAN Speeds Kevin

Gaia: Geo-Distributed Machine Learning Approaching LAN Speeds Kevin Hsieh Aaron Harlap, Nandita Vijaykumar, Dimitris Konomis, Gregory R. Ganger, Phillip B. Gibbons, Onur Mutlu† †

Machine Learning and Big Data • Machine learning is widely used to derive useful information from large-scale data Pictures Videos User Activities …… Image Classification Video Analytics Preference Prediction 2

Big Data is Geo-Distributed • A large amount of data is generated rapidly, all over the world 3

Centralizing Data is Infeasible [1, 2, 3] • Moving data over wide-area networks (WANs) can be extremely slow • It is also subject to data sovereignty laws 1. Vulimiri et al. , NSDI’ 15 2. Pu et al. , SIGCOMM’ 15 3. Viswanathan et al. , OSDI’ 16 4

Geo-distributed ML is Challenging • No ML system is designed to run across data centers (up to 53 X slowdown in our study) 5

Our Goal • Develop a geo-distributed ML system • Minimize communication over wide-area networks • Retain the accuracy and correctness of ML algorithms • Without requiring changes to the algorithms Key Result: 1. 8 -53. 5 X speedup over state-of-the-art ML systems on WANs 6

Outline • Problem & Goal • Background & Motivation • Gaia System Overview • Approximate Synchronous Parallel • System Implementation • Evaluation • Conclusion 7

Background: Parameter Server Architecture • The parameter server architecture has been widely adopted in many ML systems Training Data 1 Worker Machine 1 Update … … Data N Worker Machine N Update ML Model Parameter Read … Synchronization is critical to the accuracy Server and correctness of ML algorithms 8

Deploy Parameter Servers on WANs • Deploying parameter servers across data centers requires a lot of communication over WANs Data Center 2 Data Center 1 Worker Machine 1 … Parameter Server … Worker Machine N ML Model Parameter Server 9

WAN: Low Bandwidth and High Cost 11 Amazon EC 2 Regions eg io ns R 11 A m az on EC São Paulo Mumbai Sydney Singapore Seoul Tokyo Frankfurt Ireland Oregon California Virginia 2 1000 900 800 700 600 500 400 300 200 100 0 Vi C rgin al ia ifo r O nia re g Ire on Fr lan an d kf u To rt ky o S Si e ng ou ap l Sy ore d M ney Sa um o ba Pa i ul o Network Bandwidth (Mb/s) • WAN bandwidth is 15 X smaller than LAN bandwidth on average, and up to 60 X smaller • In Amazon EC 2, the monetary cost of WAN communication is up to 38 X the cost of renting machines 10

ML System Performance on WANs Normalized Execution Time until Convergence Matrix Factorization 25 20 15 10 5 0 Iter. Store Bӧsen 23. 8 X 24. 2 X Running ML systems on WANs can 5. 9 X 3. 5 X 4. 4 X 3. 7 Xdown seriously slow ML applications LAN EC 2 -ALL V/C WAN S/S WAN 1) Cui et al. , “Exploiting Iterative-ness for Parallel ML Computations”, 11 EC 2 Regions Singapore / São Paulo So. CC’ 14 2) Wei et al. , “Managed Communication and Consistency for Fast Data. Virginia / California Parallel Iterative Analytics”, So. CC’ 15 11

Outline • Problem & Goal • Background & Motivation • Gaia System Overview • Approximate Synchronous Parallel • System Implementation • Evaluation • Conclusion 12

Gaia System Overview • Key idea: Decouple the synchronization model within the data center from the synchronization model between data centers Data Center 2 Data Center 1 Worker Parameter Machine Server Worker Machine Local Sync Remote Sync Parameter Server Communicate over WANs only significant updates Approximately Correct Model Copy … … Approximately Correct 13 Model Copy

Insignificant Updates % Key Finding: Study of Update Significance Matrix Factorization Topic Modeling Image Classification 95. 6% 95. 2% 97. 0% 100% 80% 60% 40% 20% 0% 10% 5% 1% 1% 0% Threshold of Significant Updates 0% The vast majority of updates are insignificant 0% 14

Outline • Problem & Goal • Background & Motivation • Gaia System Overview • Approximate Synchronous Parallel • System Implementation • Evaluation • Conclusion 15

Approximate Synchronous Parallel The significance filter • Filter updates based on their significance ASP selective barrier • Ensure significant updates are read in time Mirror clock • Safe guard for pathological cases 16

The Significance Filter Worker Update (Δ 21) on X Machine Parameter X Value Parameter Server Aggregated Update Δ 01+Δ 2 Other Parameters Significance Function >? Significance Threshold 1% 17

Approximate Synchronous Parallel The significance filter • Filter updates based on their significance ASP selective barrier • Ensure significant updates are read in time Mirror clock • Safeguard for pathological cases 18

ASP Selective Barrier Data Center 1 Significant Update Parameter Server Data Center 2 Parameter Server Only workers that depend on Arrive too late! these parameters are blocked Data Center 1 Selective Significant Barrier Update Parameter Server Data Center 2 Parameter Server 19

Outline • Problem & Goal • Background & Motivation • Gaia System Overview • Approximate Synchronous Parallel • System Implementation • Evaluation • Conclusion 20

Put it All Together: The Gaia System Data Center Boundary Gaia Parameter Control. Gaia Parameter Server messages (barriers, etc. ) Server Worker are always prioritized Update Aggregated Local Machine Parameter Store Server Worker Machine Update Control Queue Mirror Server … Significance Selective Mirror No change is required for ML Filter Worker Barrier Client Data Machine algorithms and Queue. ML programs 21

Problem: Broadcast Significant Updates Communication overhead is proportional to the number of data centers 22

Mitigation: Overlay Networks and Hubs Data Center Group Hub Hub Data Center Group Save communication on WANs by aggregating the updates at hubs 23

Outline • Problem & Goal • Background & Motivation • Gaia System Overview • Approximate Synchronous Parallel • System Implementation • Evaluation • Conclusion 24

Methodology • Applications • Matrix Factorization with the Netflix dataset • Topic Modeling with the Nytimes dataset • Image Classification with the ILSVRC 12 dataset • Hardware platform • 22 machines with emulated EC 2 WAN bandwidth • We validated the performance with a real EC 2 deployment • Baseline • Iter. Store (Cui et al. , So. CC’ 14) and Gee. PS (Cui et al. , Euro. Sys’ 16) on WAN • Performance metrics • Execution time until algorithm convergence • Monetary cost of algorithm convergence 25

Performance – 11 EC 2 Data Centers Normalized Exec. Time Baseline Gaia LAN 1 0. 8 0. 6 0. 4 3. 8 X 3. 7 X 0. 2 3. 7 X 4. 8 X 6. 0 X 8. 5 X 0 Matrix Factorization Topic Modeling Image Classification Gaia achieves 3. 7 -6. 0 X speedup over Baseline Gaia is at most 1. 40 X of LAN speeds 26

Performance and WAN Bandwidth Normalized Exec. Time Baseline Gaia LAN Baseline 1 1 0. 8 0. 6 0. 4 0. 2 3. 7 X 3. 5 X 0. 4 3. 7 X 3. 9 X 7. 4 X 0 Matrix Topic Modeling Image Factorization Classification 0. 2 23. 8 X 25. 4 X Gaia LAN 17. 3 X 14. 1 X 53. 7 X 53. 5 X 0 Matrix Topic Modeling Image Factorization Classification V/C WAN 3. 7 -53. 5 X speedup over. S/S WAN Gaia achieves Baseline (Singapore/São Paulo) (Virginia/California) Gaia is at most 1. 23 X of LAN speeds 27

Results – EC 2 Monetary Cost Matrix Factorization ai a G in e 59. 0 X Ba Ba EC 2 -ALL V/C WAN S/S WAN se l in e se l ai a G se l in e ai a 10. 7 X Gaia is 2. 6 -59. 0 X cheaper than Baseline EC 2 -ALL V/C WAN S/S WAN Ba G in e se l ai a Ba G se l in e 0 8. 5 X ai a 5. 7 X G 0 Ba 2. 6 X 28. 5 X 0. 5 in e 6. 0 X Ba 4. 2 X 0. 5 se l 1 ai a 1 G 1. 5 in e 1. 5 se l 2 ai a 2 4 3. 5 3 2. 5 2 1. 5 18. 7 X 1 0. 5 0 G 2. 5 Ba Normliaed Cost 2. 5 Machine Cost (Compute) Machine Cost (Network) Communication Cost EC 2 -ALL V/C WAN S/S WAN Topic Modeling Image Classification 28

More in the Paper • Convergence proof of Approximate Synchronous Parallel (ASP) • ASP vs. fully asynchronous • Gaia vs. centralizing data approach 29

Key Takeaways • The Problem: How to perform ML on geo-distributed data? • Centralizing data is infeasible. Geo-distributed ML is very slow • Our Gaia Approach • Decouple the synchronization model within the data center from that across data centers • Eliminate insignificant updates across data centers • A new synchronization model: Approximate Synchronous Parallel • Retain the correctness and accuracy of ML algorithms • Key Results: • 1. 8 -53. 5 X speedup over state-of-the-art ML systems on WANs • at most 1. 40 X of LAN speeds • without requiring changes to algorithms 30

Gaia: Geo-Distributed Machine Learning Approaching LAN Speeds Kevin Hsieh Aaron Harlap, Nandita Vijaykumar, Dimitris Konomis, Gregory R. Ganger, Phillip B. Gibbons, Onur Mutlu† †

Executive Summary • The Problem: How to perform ML on geo-distributed data? • Centralizing data is infeasible. Geo-distributed ML is very slow • Our Goal • Minimize communication over WANs • Retain the correctness and accuracy of ML algorithms • Without requiring changes to ML algorithms • Our Gaia Approach • Decouple the synchronization model within the data center from that across data centers: Eliminate insignificant updates on WANs • A new synchronization model: Approximate Synchronous Parallel • Key Results: • 1. 8 -53. 5 X speedup over state-of-the-art ML systems on WANs • within 1. 40 X of LAN speeds 32

Approximate Synchronous Parallel The significance filter • Filter updates based their significance ASP selective barrier • Ensure significant updates are read in time Mirror clock • Safeguard for pathological cases 33

Mirror Clock Data Center 1 Data Center 2 No guarantee under Barrier extreme network conditions Parameter Server Data Center 1 Clock N Data Center 2 Clock N + DS Guarantees all significant updates Clock N d are seen after DS clocks Parameter Server 34

Effect of Synchronization Mechanisms 1 E+09 9 E+08 8 E+08 7 E+08 6 E+08 5 E+08 4 E+08 3 E+08 2 E+08 1 E+08 0 E+00 Gaia_Async Convergence value 0 50 100 150 200 250 300 350 Time (Seconds) Matrix Factorization Gaia Objective value Gaia -9. 0 E+08 Gaia_Async Convergence value -1. 0 E+09 -1. 1 E+09 -1. 2 E+09 -1. 3 E+09 -1. 4 E+09 -1. 5 E+09 0 250 500 750 1000 Time (Seconds) Topic Modeling 35

Methodology Details • 36

ML System Performance Comparison • Iter. Store [Cui et al. So. CC’ 15] shows 10 X performance improvement over Power. Graph [Gonzalez et al. , OSDI’ 12] for Matrix Factorization • Power. Graph matches the performance of Graph. X [Gonzalez et al. , OSDI’ 14], a Spark-based system 37

Matrix Factorization (1/3) • Matrix factorization (also known as collaborative filtering) is a technique commonly used in recommender systems 38

Matrix Factorization (2/3) Movie Rank (User Preference Parameters) (θ) Rank (Movie Parameters) (x) User 39

Matrix Factorization (3/3) • Objective function (L 2 regularization) • Solve with stochastic gradient decent (SGD) 40

Background – BSP • BSP (Bulk Synchronous Parallel) • All machines need to receive all updates before proceeding to the next iteration ates d p u ll a s d a e r d n a Worker 1 Always sends Worker 2 Accurate but slow Worker 3 Clock 0 1 2 3 41

Background – SSP • SSP (Stale Synchronous Parallel) • Allows the fastest worker ahead of the slowest worker by a bounded number of iterations ent, m e ir u q e r h t id w d rk ban o w t Worker 1 e n e h t s e t ia v e All ates Worker 2 but still sends all upd Worker 3 Clock 0 1 2 3 Staleness = 1 42

Compare Against Centralizing Approach Gaia Speedup over Gaia to Centralize Cost Ratio Matrix Factorization EC 2 -ALL V/C WAN S/S WAN 1. 11 1. 22 2. 13 3. 54 1. 00 1. 17 Topic Modeling EC 2 -ALL V/C WAN S/S WAN 0. 80 1. 02 1. 25 6. 14 1. 26 1. 92 Image Classification EC 2 -ALL V/C WAN S/S WAN 0. 76 1. 12 1. 86 3. 33 1. 07 1. 08 43

SSP Performance – 11 Data Centers Nromalized Execution Time Matrix Factorization 1 0. 9 0. 8 0. 7 0. 6 0. 5 0. 4 0. 3 0. 2 0. 1 0 Amazon-EC 2 Emulation-Full-Speed 2. 0 X 1. 8 X 3. 8 X Baseline Gaia BSP 2. 0 X 1. 8 X 1. 5 X 1. 3 X 3. 7 X LAN 3. 0 X Baseline Gaia 1. 5 X 1. 3 X 2. 7 X LAN SSP 44

SSP Performance – 11 Data Centers Nromalized Execution Time Topic Modeling 1 0. 9 0. 8 0. 7 0. 6 0. 5 0. 4 0. 3 0. 2 0. 1 0 Emulation-EC 2 Emulation-Full-Speed 2. 0 X 3. 7 X Baseline 1. 5 X 2. 5 X Gaia BSP 1. 7 X 2. 0 X 4. 8 X LAN 3. 5 X Baseline Gaia SSP LAN 45

SSP Performance – V/C WAN Matrix Factorization 1 0. 9 0. 8 0. 7 0. 6 0. 5 0. 4 0. 3 0. 2 0. 1 0 Baseline Gaia LAN 3. 7 X 3. 5 X 2. 6 X 2. 3 X BSP SSP Topic Modeling 1 0. 9 0. 8 0. 7 0. 6 0. 5 0. 4 0. 3 0. 2 0. 1 0 Baseline Gaia LAN 3. 7 X 3. 9 X 3. 1 X 3. 2 X BSP SSP 46

SSP Performance – S/S WAN Matrix Factorization 1 0. 9 0. 8 0. 7 0. 6 0. 5 0. 4 0. 3 0. 2 0. 1 0 Baseline Gaia LAN 25 X 24 X 16 X 14 X BSP SSP Topic Modeling 1 0. 9 0. 8 0. 7 0. 6 0. 5 0. 4 0. 3 0. 2 0. 1 0 Baseline Gaia LAN 14 X 17 X 21 X BSP SSP 47