CS 492 Special Topics in Computer Science Distributed

CS 492 Special Topics in Computer Science: Distributed Algorithms and Systems Lecture #5 Map. Reduce

Map. Reduce • Simplified data processing on large clusters – A programming model and an associated implementation for processing and generating large data sets • Inspired by “map” and “reduce” primitives present in Lisp and many other functional languages • Simple and powerful interface – that enables automatic paralization and distribution of large-scale computations

Map and Reduce • Map – map <k 1, v 1> list (k 2, v 2) – example • k 1 = document, v 1 = contents • k 2 = words, v 2 = 1 • Reduce – reduce (k 2, list(v 2)) list (v 2) • k 2 = words, v 2 = 1 • list (v 2) = total count of v 2 (or k 2)

Map. Reduce Execution Overview

Examples • • Distributed grep Count of URL access frequency Reverse web-link graph Term-vector per host

Assumptions • Hardware – Components are reliable – Components are homogeneous • Software – It’s correct • Network – Latency is zero – Bandwidth is infinite – It’s secure • Overall system – Configuration is stable – There is one administrator

Terminology • Hadoop – An open source implementation of the Map. Reduce functionality • Dryad – Distributed data-parallel programs from sequential building blocks (Euro. Sys 2007)

Reading for next class 8