Map Reduce Simplied Data Processing on Large Clusters

Map. Reduce: Simplied Data Processing on Large Clusters Yunxing Dai, Huan Feng 9/25/2020 EECS 584, Fall 2011 1

Real world problem n Count the number of occurences of each word in a huge collections of word lists. – sample input: seven book of Harry Potter

Real world problem n Count the number of occurences of each word in a huge collections of word lists. Word Occurences The 15414 Good 5435 Never 6546 Tie 694 . . . .

Possible solution n Hash table – each entry is key value pair, (word, occurrence) – scan all the file, put each word into the hash table

Real world problem--follow up n What if you are given a huge set of files and have access to a large set of machines? n Problem with hash table: – low concurrency – hard to scale • one node fail, restart all work n Map. Reduce solution

Map primitive Idea from functional language n Given a function, apply the function to all element INDIVIDUALLY in the list, combine the result into a new list n e. g. increment each elems from a list by 1 n 9/25/2020 EECS 584, Fall 2011 6

Reduce primitive n Idea from functional language n Apply a function to all elems from a list, combine them into a single resule n e. g. calculate the sum of a list 9/25/2020 EECS 584, Fall 2011 7

Map reduce solution--Single node n Map each single word into a (key, value) pair. – "Good" -> ("Good", 1) n Put together all the pairs that have the same key. Input these pairs to a reduce program. Add the value together – [("Good", 1), ("Good", 1)] -> 3

Map reduce solution Files in_01 in_02 in_03 in_04 in_05. . .

Map reduce solution Files in_01 Map pairs (The, 1) in_02 (Good, 1) in_03 (The, 1) in_04 (Bad, 1) in_05 (Not, 1) . . . (The, 1) . . .

Map reduce solution Files in_01 Map pairs (The, 1) list of pairs Merge and Sort [(The, 1), (The, 1). . . ] in_02 (Good, 1) in_03 (The, 1) in_04 (Bad, 1) [(Therofiery, 1)] in_05 (Not, 1) [(Bad, 1), (Bad, 1). . . ] . . . (The, 1) . . . [(Good, 1), (Good, 1). . [(Is, 1), (Is, 1). . . ] . . .

Map reduce solution n What if now we are given a huge number of files. n And a large number of machines.

It can be scalable! n Map can be applied to different part of input in parallel. n If part of map tasks failed, just need to restart them instead of restarting all.

Map reduce solution-scalable version n Map : split the file into several parts, apply map function to every part of them. n Shuffle : distribute intemediate result into different buckets according to the hash value of key, assign buckets to several reducers. Each reducer sort the pairs by key. n Reduce : apply the reduce function to all the elements that have the same key and produce the result.

Map. Reduce n Gerneralized Software framework n Users are only responsible for provide two functions : map and reduce n Easy to scale to large amount of machines.

Split the input files into several pieces

Each piece is assigned to one worker(mapper)

Before sorting, the key value pairs are hashed by key into R buckets.

(The, 1), (Good, 1), (The 1), (Never, 1)

(The, 1), (Good, 1), (The 1), (Never, 1) (Is, 1), (Tie, 1), (Work, 1)

(The, 1), (Good, 1), (The 1), (Never, 1) (Is, 1), (Tie, 1), (Work, 1) Each bucket is read by one worker(reducer), then sort and produce the results.

Master program, control the process and assign work to workers

Fault tolerance n worker fail : simply assigned the another worker to do it. n master fail : restart the whole work

Implementation details n Locality – Take location information of input files into account – assigned a map task to closest machine of the input data.

Implementation details n Backup tasks – abnormal machines in a task lengthen the total time. – When a task is almost finished, duplicate remaining tasks as back-up tasks. – Whenever a primary or a backup execution is done, mark the task as finished. TASK 1 TASK 2 TASK 3 In progress Completed

Implementation details n Backup tasks – abnormal machines in a task lengthen the total time. – When a task is almost finished, duplicate remaining tasks as back-up tasks. – Whenever a primary or a backup execution is done, mark the task as finished. TASK 1 TASK 2 TASK 3 In progress Completed

Implementation details n Backup tasks – abnormal machines in a task lengthen the total time. – When a task is almost finished, duplicate remaining tasks as back-up tasks. – Whenever a primary or a backup execution is done, mark the task as finished. TASK 1 Back Up TASK 1 TASK 2 TASK 3 In progress Completed

Implementation details n Backup tasks – abnormal machines in a task lengthen the total time. – When a task is almost finished, duplicate remaining tasks as back-up tasks. – Whenever a primary or a backup execution is done, mark the task as finished. TASK 1 TASK 2 TASK 3 In progress Completed

Useful Extensions n Partitioning Functions – hash-based or range-based – self-defined partition function n Combiner Function (similar to Reduce Function) – <any, 1> <any , 1> – resolve significant repetition in intermediate outputs n Skipping Bad Records – errors or bugs – acceptable to ignore a few records n Local Execution – help facilitate debugging, profiling and testing 9/25/2020 EECS 584, Fall 2011 34

Performance & Evaluation n Cluster Configuration – 1800 nodes – 2× 2 GHz, 4 GB memory, 2× 160 GB IDE, Gb Ethernet link – 2 -level tree-shaped switched network n Grep – in 1010 100 -byte records – M = 15000 R = 1 – take ~150 seconds 9/25/2020 EECS 584, Fall 2011 35

Performance & Evaluation (Sort) n Sort – 1010 100 -byte records – M = 15000, R = 4000 – Normal, No-Backup, 200 tasks killed n A few things to note – Input rate is higher than the shuffle & output rate – no backup, execution flow is similar except the long tail – kill tasks, the tasks restarted & the rate drop to zero 9/25/2020 EECS 584, Fall 2011 36

9/25/2020 EECS 584, Fall 2011 37

Application of Map. Reduce n Broadly applicable – – n large-scale machine learning problems clustering problems for Google News extraction of data & properties graph computations Large-Scale Indexing – The indexing code is simpler, smaller (~3800 to ~700) – Indexing process is much easier to operate & easy spead up

Map. Reduce & Parallel DBMS n Map. Reduce is not novel at all – a entriely new paradigm? n Map. Reduce is a step backwards – no schema – no high-level access language n Map. Reduce Is a poor implementation – no Index – overlook skew – lots of P 2 P Network traffic in the shuffle phase n Missing features – indexes, updates, transactions n Not compatible to DBMS Tools

Map. Reduce & Parallel DBMS

Map. Reduce & Parallel DBMS n Parallel database – – n has significant performance advantage take a lot of time to tune and setup are not general enough (UDFs, UDTs) SQL is not that easy & straightforward Map. Reduce – easy to setup & easy to program – scalable & fault-tolerant – bruteforce solution

What is Map. Reduce n A parallel programming model /data processing paradigm rather than a complete DBMS – Does not target everything DBMS targets – It's simple but it works n Works for those who – have a lot of data (of some specific type) – UDTs and UDFs are complex to tune – would rather program in sequencial language rather than SQL – no need to index data because data change all the time – do not need to pay

Questions?