Spark Fast Interactive LanguageIntegrated Cluster Computing Matei Zaharia

Spark Fast, Interactive, Language-Integrated Cluster Computing Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy Mc. Cauley, Michael Franklin, Scott Shenker, Ion Stoica www. spark-project. org UC BERKELEY

Project Goals Extend the Map. Reduce model to better support two common classes of analytics apps: » Iterative algorithms (machine learning, graphs) » Interactive data mining Enhance programmability: » Integrate into Scala programming language » Allow interactive use from Scala interpreter

Project Goals Extend the Map. Reduce model to better support two common classes of analytics apps: » Iterative algorithms (machine learning, graphs) » Interactive data mining Enhance programmability: Explain why the original Map. Reduce model does not » Integrate into Scala programming language efficiently support these use cases? » Allow interactive use from Scala interpreter

Motivation Most current cluster programming models are based on acyclic data flow from stable storage to stable storage Map Input Reduce Output Map Reduce

Motivation Most current cluster programming models are based on acyclic data flow from stable storage to stable storage Map Reduce Benefits of data flow: runtime can decide Map Output Input to run where tasks and can automatically recover from failures Reduce Map

Motivation Acyclic data flow is inefficient for applications that repeatedly reuse a working set of data: » Iterative algorithms (machine learning, graphs) » Interactive data mining tools (R, Excel, Python) With current frameworks, apps reload data from stable storage on each query

Solution: Resilient Distributed Datasets (RDDs) Allow apps to keep working sets in memory for efficient reuse Retain the attractive properties of Map. Reduce » Fault tolerance, data locality, scalability Support a wide range of applications

Programming Model Resilient distributed datasets (RDDs) » Immutable, partitioned collections of objects » Created through parallel transformations (map, filter, group. By, join, …) on data in stable storage » Can be cached for efficient reuse Actions on RDDs » Count, reduce, collect, save, …

Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark. text. File(“hdfs: //. . . ”) Base. Transformed RDD results errors = lines. filter(_. starts. With(“ERROR”)) messages = errors. map(_. split(‘t’)(2)) cached. Msgs = messages. cache() Driver tasks Cache 1 Worker Block 1 Action cached. Msgs. filter(_. contains(“foo”)). count Cache 2 cached. Msgs. filter(_. contains(“bar”)). count Worker . . . Cache 3 Result: scaled tosearch 1 TB data in 5 -7 sec full-text of Wikipedia data) in <1(vs sec 170 (vssec 20 for secon-disk for on-disk data) Worker Block 3 Block 2

RDD Fault Tolerance RDDs maintain lineage information that can be used to reconstruct lost partitions Ex: messages = text. File(. . . ). filter(_. starts. With(“ERROR”)). map(_. split(‘t’)(2)) HDFS File Filtered RDD filter (func = _. contains(. . . )) Mapped RDD map (func = _. split(. . . ))

Example: Logistic Regression Goal: find best line separating two sets of points random initial line + + + – – – – target

Example: Logistic Regression val data = spark. text. File(. . . ). map(read. Point). cache() var w = Vector. random(D) for (i <- 1 to ITERATIONS) { val gradient = data. map(p => (1 / (1 + exp(-p. y*(w dot p. x))) - 1) * p. y * p. x ). reduce(_ + _) w -= gradient } println("Final w: " + w)

Running Time (s) Logistic Regression Performance 4500 4000 3500 3000 2500 2000 1500 1000 500 0 127 s / iteration Hadoop Spark 1 5 10 20 Number of Iterations 30 first iteration 174 s further iterations 6 s

Spark Operations Transformations (define a new RDD) Actions (return a result to driver program) map filter sample group. By. Key reduce. By. Key sort. By. Key flat. Map union join cogroup cross map. Values collect reduce count save lookup. Key

YARN HDFS