CSDB an alternative approach with Spark Luca Menichetti

CSDB, an alternative approach with Spark Luca Menichetti, IT-ST-AD AWG meeting, 21/09/2016 1

From SQL to Spark • • SQL: high level human language I prefer to run Spark. SQL when possible: • • • Faster (to write) More readable and easy to understand I don’t use Spark. SQL when • • I need a function that is not available there or cannot be registered as UDF the operation that I need to run cannot be (or is not) implemented (e. g. lag) an equivalent operation can be written with considerably less effort (the query is too complex) plain Spark will be faster (with another approach) 2

The problem, for each SID 3

Time interval • • For each time interval, generate two tuples Each tuple contains • • +1 or -1 if it’s start or end respectively 1 or 0 that represents the number of active SID after that timestamp 4

For each SID • Generate tuples and sort them by TS • • List( (TS, FLAG, MAX) ) From left to right, two tuples at a time, A and B • • Sum flag A + flag B New MAX is the biggest between MAX of A, B and the sum of the flags 5

Reduce 6

Notebook https: //www. zeppelinhub. com/viewer/notebooks/bm 90 ZTov. L 2 xt. ZW 5 p. Y 2 hl. L 1 pl c. HBlb. Glu. LURya. XZlci 0 w. Nj. Av. Nj. E 0 Nz. Q 5 Zj. Ji. Y 2 Fm. NGFl. MGE 5 Ym. Uy. ZDQz. YTM 3 NGYw. ZTAvbm 90 ZS 5 qc 29 u 7

Spark 1. 6 4 files: Spark runs only 4 executors Executor spec: 1 Core 1 GB Repeating twice the same operation: the second time a stage is skipped. The Job Id 1 is the real value to consider. 8

Spark 1. 6, 8 executors . repartition(8) 9

Spark 1. 6, 16 executors . repartition(16) 10

Spark 2. 0 4 files: Spark runs only 3 executors, not 4 like in Spark 1. 6 Executor spec: 1 Core 1 GB 11

Spark 2. 0, 8 executors . repartition(8) 12

Spark 2. 0, 16 executors . repartition(16) 13