Cluster Computing with Dryad LINQ Mihai Budiu Microsoft

Cluster Computing with Dryad. LINQ Mihai Budiu Microsoft Research, Silicon Valley Cloud computing: Infrastructure, Services, and Applications UC Berkeley, March 4 2009

Goal 2

Design Space Grid Internet Dataparallel Dryad Search Private data center Shared memory Transaction Latency HPC Throughput 3

Data-Parallel Computation Application SQL Language Execution Storage Parallel Databases Sawzall ≈SQL Sawzall Pig, Hive Map. Reduce Hadoop GFS Big. Table HDFS S 3 LINQ, SQL Dryad. LINQ Scope Dryad Cosmos Azure SQL Server 4

Software Stack Applications legacy code PSQL C# Scope Machine Data Learning Graphs mining SSIS . Net Distributed Data Structures Distributed Shell Dryad. LINQ C++ SQL server Dryad Distributed FS (Cosmos) Azure XStore Cluster Services Windows Server SQL Server Azure XCompute Windows Server queueing Log parsing NTFS Windows HPC Windows Server 5

Outline • • Introduction Dryad. LINQ Conclusions 6

Dryad • • Continuously deployed since 2006 Running on >> 104 machines Sifting through > 10 Pb data daily Runs on clusters > 3000 machines Handles jobs with > 105 processes each Platform for rich software ecosystem Used by >> 100 developers • Written at Microsoft Research, Silicon Valley 7

Dryad = Execution Layer Job (application) Dryad Cluster ≈ Pipeline Shell Machine 8

2 -D Piping • Unix Pipes: 1 -D grep | sed | sort | awk | perl • Dryad: 2 -D grep 1000 | sed 500 | sort 1000 | awk 500 | perl 50 9

Virtualized 2 -D Pipelines 10

Virtualized 2 -D Pipelines 11

Virtualized 2 -D Pipelines 12

Virtualized 2 -D Pipelines 13

Virtualized 2 -D Pipelines • 2 D DAG • multi-machine • virtualized 14

Dryad Job Structure Channels Input files Stage sort grep sed awk perl sort awk sed Vertices (processes) Output files sort 15

Channels Finite streams of items X Items M • distributed filesystem files (persistent) • SMB/NTFS files (temporary) • TCP pipes (inter-machine) • memory FIFOs (intra-machine) 16

Dryad System Architecture data plane job schedule Files, TCP, FIFO, Network NS Job manager control plane V V V PD PD PD cluster 17

Fault Tolerance

Policy Managers R R Stage R Connection R-X X X Manager R manager X Stage X R-X Manager Job Manager 19

Dynamic Graph Rewriting X[0] X[1] X[3] Completed vertices X[2] Slow vertex X’[2] Duplicate vertex Duplication Policy = f(running times, data volumes)

Cluster network topology top-level switch top-of-rack switch rack

Dynamic Aggregation S S S rack # dynamic S S #3 S #2 S T static #1 S S #2 S #1 S # 1 A # 2 A T # 3 A 22

Policy vs. Mechanism • Application-level • Most complex in C++ code • Invoked with upcalls • Need good default implementations • Dryad. LINQ provides a comprehensive set • Built-in • • Scheduling Graph rewriting Fault tolerance Statistics and reporting 23

Outline • • Introduction Dryad. LINQ Conclusions 24

LINQ => Dryad. LINQ Dryad 25

LINQ =. Net+ Queries Collection<T> collection; bool Is. Legal(Key); string Hash(Key); var results = from c in collection where Is. Legal(c. key) select new { Hash(c. key), c. value}; 26

Collections and Iterators class Collection<T> : IEnumerable<T>; public interface IEnumerable<T> { IEnumerator<T> Get. Enumerator(); } public interface IEnumerator <T> { T Current { get; } bool Move. Next(); void Reset(); } 27

Dryad. LINQ Data Model. Net objects Partition Collection 28

Dryad. LINQ = LINQ + Dryad Vertex code Collection<T> collection; bool Is. Legal(Key k); string Hash(Key); var results = from c in collection where Is. Legal(c. key) select new { Hash(c. key), c. value}; Query plan (Dryad job) Data collection C# C# results 29

Demo 30

Example: Histogram public static IQueryable<Pair> Histogram( IQueryable<Line. Record> input, int k) { var words = input. Select. Many(x => x. line. Split(' ')); var groups = words. Group. By(x => x); var counts = groups. Select(x => new Pair(x. Key, x. Count())); var ordered = counts. Order. By. Descending(x => x. count); var top = ordered. Take(k); return top; } “A line of words of wisdom” [“A”, “line”, “of”, “words”, “of”, “wisdom”] [[“A”], [“line”], [“of”, “of”], [“words”], [“wisdom”]] [ {“A”, 1}, {“line”, 1}, {“of”, 2}, {“words”, 1}, {“wisdom”, 1}] [{“of”, 2}, {“A”, 1}, {“line”, 1}] 31

Histogram Plan Select. Many Sort Group. By+Select Hash. Distribute Merge. Sort Group. By Select Sort Take Merge. Sort Take 32

Map-Reduce in Dryad. LINQ public static IQueryable<S> Map. Reduce<T, M, K, S>( this IQueryable<T> input, Expression<Func<T, IEnumerable<M>>> mapper, Expression<Func<M, K>> key. Selector, Expression<Func<IGrouping<K, M>, S>> reducer) { var map = input. Select. Many(mapper); var group = map. Group. By(key. Selector); var result = group. Select(reducer); return result; } 33

M M M map Q Q Q Q sort G 1 G 1 groupby R R R R reduce D D D D distribute M G R X MS MS MS mergesort G 2 G 2 G 2 groupby R R R reduce X X X static S S dynamic S A T S A S dynamic MS MS mergesort G 2 groupby R R reduce X X consumer partial aggregation M reduce M map Map-Reduce Plan 34

Distributed Sorting Plan DS DS H O DS H D static DS D H D dynamic DS D D dynamic M M M S S S 35

Expectation Maximization • 160 lines • 3 iterations shown 36

Probabilistic Index Maps Images features 37

Language Summary Where Select Group. By Order. By Aggregate Join Apply Materialize 38

LINQ System Architecture Local machine . Net program (C#, VB, F#, etc) Query LINQ Provider Objects Execution engine • LINQ-to-obj • PLINQ • LINQ-to-SQL • LINQ-to-WS • Dryad. LINQ • Flickr • Oracle • LINQ-to-XML • Your own 39

The Dryad. LINQ Provider . Net Client machine Dryad. LINQ To. Collection Query Expr Data center Distributed Invoke query plan Query Vertex Concode text Dryad JM foreach Output. Net Objects Dryad. Table (11) Results Input Tables Dryad Execution Output Tables 40

Combining Query Providers Local machine . Net program (C#, VB, F#, etc) Query Objects Execution engines LINQ Provider PLINQ Provider SQL Server LINQ Provider Dryad. LINQ Provider LINQ-to-obj 41

Using PLINQ Query Dryad. LINQ Local query PLINQ 42

Using LINQ to SQL Server Query Dryad. LINQ Query LINQ to SQL Query 43

Using LINQ-to-objects Local machine LINQ to obj debug Query production Dryad. LINQ Cluster 44

Outline • • Introduction Dryad. LINQ Conclusions 45

Lessons Learned (1) • What worked well? – Complete separation of storage / execution / language – Using LINQ +. Net (language integration) – Strong typing for data – Allowing flexible and powerful policies – Centralized job manager: no replication, no consensus, no checkpointing – Porting (HPC, Cosmos, Azure, SQL Server) – Technology transfer (done at the right time) 46

Lessons Learned (2) • What worked less well – Error handling and propagation – Distributed (randomized) resource allocation – TCP pipe channels – Hierarchical dataflow graphs (each vertex = small graph) – Forking the source tree 47

Lessons Learned (3) • Tricks of the trade – Asynchronous operations hide latency – Management through distributed state machines – Logging state transitions for debugging – Complete separation of data and control – Leases clean-up after themselves – Understand scaling factors O(machines) < O(vertices) < O(edges) – Don’t fix a broken API, re-design it – Compression trades-off bandwidth for CPU – Managed code increases productivity by 10 x 10 48

Ongoing Dryad/Dryad. LINQ Research • • Performance modeling Scheduling and resource allocation Profiling and performance debugging Incremental computation Hardware acceleration High-level programming abstractions Many domain-specific applications 49

Sample applications written using Dryad. LINQ Class Distributed linear algebra Numerical Accelerated Page-Rank computation Web graph Privacy-preserving query language Data mining Expectation maximization for a mixture of Gaussians Clustering K-means Clustering Linear regression Statistics Probabilistic Index Maps Image processing Principal component analysis Data mining Probabilistic Latent Semantic Indexing Data mining Performance analysis and visualization Debugging Road network shortest-path preprocessing Graph Botnet detection Data mining Epitome computation Image processing Neural network training Statistics Parallel machine learning framework infer. net Machine learning Distributed query caching Optimization Image indexing Image processing Web indexing structure Web graph 50

Conclusions Dryad Visual Studio = LINQ 51 51

“What’s the point if I can’t have it? ” • Glad you asked • We’re offering Dryad+Dryad. LINQ to academic partners • Dryad is in binary form, Dryad. LINQ in source • Requires signing a 3 -page licensing agreement 52

Backup Slides 53

Dryad. LINQ • Declarative programming • Integration with Visual Studio • Integration with. Net • Type safety • Automatic serialization • Job graph optimizations § static § dynamic • Conciseness 54

What does Dryad. LINQ do? public struct Data { … public static int Compare(Data left, Data right); } Data g = new Data(); var result = table. Where(s => Data. Compare(s, g) < 0); Data serialization public static void Read(this Dryad. Binary. Reader reader, out Data obj); public static int Write(this Dryad. Binary. Writer writer, Data obj); Data factory public class Dryad. Factory. Type__0 : Linq. To. Dryad. Factory<Data> Channel writer Channel reader LINQ code Context serialization Dryad. Vertex. Env denv = new Dryad. Vertex. Env(args); var dwriter__2 = denv. Make. Writer(Factory. Type__0); var dreader__3 = denv. Make. Reader(Factory. Type__0); var source__4 = Dryad. Linq. Vertex. Where(dreader__3, s => (Data. Compare(s, ((Data)Dryad. Linq. Object. Store. Get(0))) < ((System. Int 32)(0))), false); dwriter__2. Write. Item. Sequence(source__4); 55

Range-Distribution Manager S S S [0 -100) Hist [0 -30), [30 -100) static T D D T [0 -? ) [0 -30) dynamic D T [? -100) [30 -100) 56

Staging 1. Build 2. Send. exe JM code 7. Serialize vertices vertex code 5. Generate graph 6. Initialize vertices 3. Start JM Cluster services 8. Monitor Vertex execution 4. Query cluster resources

Bibliography Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, and Dennis Fetterly European Conference on Computer Systems (Euro. Sys), Lisbon, Portugal, March 21 -23, 2007 Dryad. LINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language Yuan Yu, Michael Isard, Dennis Fetterly, Mihai Budiu, Úlfar Erlingsson, Pradeep Kumar Gunda, and Jon Currey Symposium on Operating System Design and Implementation (OSDI), San Diego, CA, December 8 -10, 2008 SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets Ronnie Chaiken, Bob Jenkins, Per-Åke Larson, Bill Ramsey, Darren Shakib, Simon Weaver, and Jingren Zhou Very Large Databases Conference (VLDB), Auckland, New Zealand, August 23 -28 2008 Hunting for problems with Artemis Gabriela F. Creţu-Ciocârlie, Mihai Budiu, and Moises Goldszmidt USENIX Workshop on the Analysis of System Logs (WASL), San Diego, CA, December 7, 2008 58

Data Partitioning DATA RAM DATA 59

Linear Algebra & Machine Learning in Dryad. LINQ Data analysis Machine learning Large Vector Dryad. LINQ Dryad 60

Operations on Large Vectors: Map 1 T f U f preserves partitioning T f U 61

Map 2 (Pairwise) T U f V 62

Map 3 (Vector-Scalar) T f U V T U f V 63

Reduce (Fold) f U U f f f U U U f U 64

Linear Algebra T T , U , V = 65

Linear Regression • Data • Find • S. t. 66

Analytic Solution X[0] X[1] X[2] Y[0] Y[1] Y[2] Map X×XT Y×XT Reduce Σ Σ [ ]-1 * A 67

Linear Regression Code Vectors x = input(0), y = input(1); Matrices xx = x. Map(x, (a, b) => a. Outer. Prod(b)); One. Matrix xxs = xx. Sum(); Matrices yx = y. Map(x, (a, b) => a. Outer. Prod(b)); One. Matrix yxs = yx. Sum(); One. Matrix xxinv = xxs. Map(a => a. Inverse()); One. Matrix A = yxs. Map(xxinv, (a, b) => a. Mult(b)); 68