Dryad and Drya LINQ Dryad and Dryad LINQ

Dryad and Drya. LINQ

Dryad and Dryad. LINQ Dryad provides automatic distributed execution Dryad. LINQ provides automatic query plan generation

Dryad • General-purpose execution environment for distributed, data-parallel applications • Focus on simplicity, reliability, scalability, efficiency and not latency, unreliable networks • Automatic management of scheduling, distribution, fault tolerance • Exploits Data Parallelism

Dryad • Computations expressed as a Directed Acyclic Graph – Jobs executed on vertices – Edges are communication channels – Each vertex has several input and output edges – Data transport mechanisms: Files, TCP pipes, shared memory FIFOs

Job = Directed Acyclic Graph Outputs Processing vertices Channels (file, pipe, shared memory) Inputs

Dryad vs. Map. Reduce, Parallel DB • More control to developer than Map. Reduce • Map. Reduce aims at simplicity at the expense of generality and performance • Computation Graph is implicit in Parallel DB

Dryad System Architecture • Job manager – coordinates jobs, constructs graph • Name server – exposes computers with network topology • Daemons run on each computer in the cluster

Communication

Job (Graph) Construction • Using graph operators implemented in C++ to describe the graph (from simpler sub graphs).

Job Execution • Job manager not currently fault tolerant • Vertices may be scheduled multiple times due to failures – Each execution versioned – Execution record kept- including versions of incoming vertices – Outputs are uniquely named (versioned) – Final outputs selected if job completes – Non-file communication (TCP pipe, Shared Memory FIFO) may cascade failures • Vertices specify hard constraints or preferences for set of computers required • Scheduling is greedy assuming only one job

Policy Managers R R Stage R Connection R-X X X Manager R manager X Stage X R-X Manager Job Manager 11

Cluster network topology top-level switch top-of-rack switch rack

Run-time Graph Refinement

Dynamic Aggregation S S S rack # dynamic S S #3 S #2 S T static #1 S S #2 S #1 S # 1 A # 2 A T # 3 A 14

Fault Tolerance

Sky. Server DB Query • • 3 -way join to find gravitational lens effect Table U: (obj. Id, color) 11. 8 GB Table N: (obj. Id, neighbor. Id) 41. 8 GB Find neighboring stars with similar colors: – Join U+N to find T = U. color, N. neighbor. Id where U. obj. Id = N. obj. Id – Join U+T to find U. obj. Id where U. obj. Id = T. neighbor. ID and U. color ≈ T. color

Sky. Server DB query [distinct] (u. color, n. neighborobjid) • Took SQL planby [merge outputs] [re-partition n. neighborobjid] • Manually in Dryad [order by coded n. neighborobjid] select u. color, n. neighborobjid • Manually partitioned data fromu. objid u join n H n Y Y U from u join <temp> where u. objid = n. objid u: u. objid, color = <temp>. neighborobjid n: andobjid, neighborobjid [partition objid] |u. color by - <temp>. color| < d U U S 4 n S M 4 n M D n D X n X N U N

Optimization H Y U n Y S S S Y S U M S 4 n S M 4 n M D D n D X X n X M M M U U N U N

Optimization H Y U n Y S S S Y S U M S 4 n S M 4 n M D D n D X X n X M M M U U N U N

16. 0 Dryad In-Memory 14. 0 Dryad Two-pass Speed-up 12. 0 SQLServer 2005 10. 0 8. 0 6. 0 4. 0 2. 0 0 2 4 6 Number of Computers 8 10

High level Programming Languages • Nebula – limited to existing binaries • SSIS – SQLServer workflow engine, distributed • Dryad. LINQ – Supports both imperative and declarative operations on datasets

Dryad/Dryad. LINQ • Decoupling of Dryad and Dryad. LINQ – Dryad: execution engine (given DAG, do scheduling and fault tolerance) – Dryad. LINQ: programming model (given query, generate DAG)

Dryad. LINQ • Exploits LINQ (Relational queries integrated in C#) to provide a hybrid of imperative and declarative programming • LINQ has a design choice that is easy to express computations also giving runtime leeway implementing them. • Sequential program composed of LINQ expressions • Performs side-effect free transformations on datasets • Written and Debugged using. NET development tools • More general than distributed SQL • Programs can be automatically optimized and efficiently executed on large cluster

Dryad. LINQ • Serialization for dryad are provided by High level software layers like Dray. LINQ • Dray. LINQ preserves the LINQ programming model and defines new operators and datatypes for data parallel programming

Dryad. LINQ Architecture

Dryad. LINQ Data Model. Net objects Partitioned Table Data Model is distributed implementation of LINQ Collections Each Dataset is distributed (disjoint) across the cluster Partitioned table exposes metadata information – type, partition, compression scheme, serialization, etc.

Dray. LINQ Constructs • Expressions must be side-effect free • Allows programmer to specify annotations (hints) to guide optimization • Operators – Hash Partition – Range Partition – Apply: Allows arbitrary streaming computations – Fork: Takes single input and generates multiple output datasets

System Implementation • Execution Plan Graph: Starts by converting raw LINQ expressions into EPG • Dryad. LINQ Optimizations – Static Optimizations – Dynamic Optimizations • Code Generation: Uses dynamic code generation to automatically synthesize LINQ code to be run at the Drayad vertex

Conclusions • Goal: Use a compute cluster as if it is a single computer – Dryad/Dryad. LINQ represent a significant step • Requires close collaborations across many fields of computing, including – Distributed systems – Distributed and parallel databases – Programming language design and analysis

References • Dryad: Distributed Data-parallel Programs from Sequential Building Blocks (Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, and Dennis Fetterly March 2007) • Dryad. LINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High. Level Language (Yuan Yu, Michael Isard, Dennis Fetterly, Mihai Budiu, Úlfar Erlingsson, Pradeep Kumar Gunda, and Jon Currey December 2008)

Thank you