Creating MapReduce Programs Using Hadoop Presentation Overview Recall

Creating Map-Reduce Programs Using Hadoop

Presentation Overview Recall Hadoop Overview of the map-reduce paradigm Elaboration on the Word. Count example components of Hadoop that make Word. Count possible Major new example: N-Gram Generator step-by-step assembly of this map-reduce job Design questions to ask when creating your own Hadoop jobs

Recall why Hadoop rocks Hadoop is: Free and open source high quality, like all Apache Foundation projects crossplatform (pure Java) fault-tolerant highly scalable has bindings for non-Java programming languages applicable to many computational problems

Map-Reduce System Overview Job. Tracker – Makes scheduling decisions Task. Tracker – Manages tasks for a given node Task process Runs an individual map or reduce fragment for a given job Forks from the Task. Tracker

Map-Reduce System Overview Processes communicate by custom RPC implementation Easy to change/extend Defined as Java interfaces Server objects implement the interface Client proxy objects automatically created All messages originate at the client: (e. g. , Task to Task. Tracker) Prevents cycles and therefore deadlocks

Process Flow Diagram

Application Overview Launching Program Creates a Job. Conf to define a job. Submits Job. Conf to Job. Tracker and waits for completion. Mapper Is given a stream of key 1, value 1 pairs Generates a stream of key 2, value 2 pairs Reducer Is given a key 2 and a stream of value 2’s Generates a stream of key 3, value 3 pairs

Job Launch Process: Client program creates a Job. Conf Identify classes implementing Mapper and Reducer interfaces Job. Conf. set. Mapper. Class(); Job. Conf. set. Reducer. Class() Specify input and output formats Job. Conf. set. Input. Format(Text. Input. Format. class); Job. Conf. set. Output. Format(Text. Output. Format. class); Other options too: Job. Conf. set. Num. Reduce. Tasks() Job. Conf. set. Output. Format()

An onslaught of terminology We'll explain these terms, each of which plays a role in any non-trivial map/reduce job: Input. Format, Output. Format, File. Input. Format, . . . Job. Client and Job. Conf Job. Tracker and Task. Tracker Task. Runner, Map. Runner, … Input. Split, Record. Reader, Line. Record. Reader, . . . Writable, Writable. Comparable, Writable. Int, . . .

Input. Format and Output. Format The application also chooses input and output formats, which define how the persistent data is read and written. These are interfaces and can be defined by the application. Input. Format Splits the input to determine the input to each map task. Defines a Record. Reader that reads key, value pairs that are passed to the map task Output. Format Given the key, value pairs and a filename, writes the

Example public static void main(String[] args) throws Exception { Job. Conf conf = new Job. Conf(Word. Count. class); conf. set. Output. Key. Class(Text. class); conf. set. Output. Value. Class(Int. Writable. class); conf. set. Mapper. Class(Map. class); conf. set. Combiner. Class(Reduce. class); conf. set. Reducer. Class(Reduce. class);

Job Launch Process: Job. Client Pass Job. Conf to Job. Client. run. Job() or Job. Client. submit. Job() run. Job() blocks – wait until job finishes submit. Job() does not Poll for status to make running decisions Avoid polling with Job. Conf. set. Job. End. Notification. URI() Job. Client: Determines proper division of input into Input. Splits Sends job data to master Job. Tracker server

Job Launch Process: Job. Tracker: Inserts jar and Job. Conf (serialized to XML) in shared location Posts a Job. In. Progress to its run queue

Job Launch Process: Task. Trackers running on slave nodes periodically query Job. Tracker for work Retrieve job-specific jar and config Launch task in separate instance of Java main() is provided by Hadoop

Job Launch Process: Task. Tracker. Child. main(): Sets up the child Task. In. Progress attempt Reads XML configuration Connects back to necessary Map. Reduce components via RPC Uses Task. Runner to launch user process

Job Launch Process: Task. Runner, Map. Runner work in a daisy-chain to launch your Mapper Task knows ahead of time which Input. Splits it should be mapping Calls Mapper once for each record retrieved from the Input. Split Running the Reducer is much the same

Creating the Mapper You provide the instance of Mapper Should extend Map. Reduce. Base Implement interface Mapper<K 1, V 1, K 2, V 2> One instance of your Mapper is initialized by the Map. Task. Runner for a Task. In. Progress Exists in separate process from all other instances of Mapper – no data sharing!

Mapper Override function – map() void map(Writable. Comparable key, Writable value, Output. Collector output, Reporter reporter) Emit (k 2, v 2) with output. collect(k 2, v 2)

Example public static class Map extends Map. Reduce. Base implements Mapper<Long. Writable, Text, Int. Writable> { private final static Int. Writable one = new Int. Writable(1); private Text word = new Text(); public void map(Long. Writable key, Text value, Output. Collector<Text, Int. Writable> output, Reporter reporter) throws IOException {

What is Writable? Hadoop defines its own “box” classes for strings (Text), integers (Int. Writable), etc. All values are instances of Writable All keys are instances of Writable. Comparable

Reading data Data sets are specified by Input. Formats Defines input data (e. g. , a directory) Identifies partitions of the data that form an Input. Split Factory for Record. Reader objects to extract (k, v) records from the input source

File. Input. Format and friends Text. Input. Format – Treats each ‘n’-terminated line of a file as a value Key. Value. Text. Input. Format – Maps ‘n’- terminated text lines of “k SEP v” Sequence. File. Input. Format – Binary file of (k, v) pairs with some add’l metadata Sequence. File. As. Text. Input. Format – Same, but maps (k. to. String(), v. to. String())

Filtering File Inputs File. Input. Format will read all files out of a specified directory and send them to the mapper Delegates filtering this file list to a method subclasses may override e. g. , Create your own “xyz. File. Input. Format” to read *. xyz from directory list

Record Readers Without a Record. Reader, Hadoop would be forced to divide input on byte boundaries. Each Input. Format provides its own Record. Reader implementation Provides capability multiplexing Line. Record. Reader – Reads a line from a text file Key. Value. Record. Reader – Used by Key. Value. Text. Input. Format

Input Split Size File. Input. Format will divide large files into chunks Exact size controlled by mapred. min. split. size Record. Readers receive file, offset, and length of chunk Custom Input. Format implementations may override split size – e. g. , “Never. Chunk. File”

Sending Data To Reducers Map function receives Output. Collector object Output. Collector. collect() takes (k, v) elements Any (Writable. Comparable, Writable) can be used

Writable. Comparator Compares Writable. Comparable data Will call Writable. Comparable. compare() Can provide fast path for serialized data Explicitly stated in Job. Conf setup Job. Conf. set. Output. Value. Grouping. Comparator()

Sending Data To The Client Reporter object sent to Mapper allows simple asynchronous feedback incr. Counter(Enum key, long amount) set. Status(String msg) Allows self-identification of input Input. Split get. Input. Split()

Partitioner int get. Partition(key, val, num. Partitions) Outputs the partition number for a given key One partition == values sent to one Reduce task Hash. Partitioner used by default Uses key. hash. Code() to return partition num Job. Conf sets Partitioner implementation

Reducer reduce( Writable. Comparable key, Iterator values, Output. Collector output, Reporter reporter) Keys & values sent to one partition all go to the same reduce task Calls are sorted by key – “earlier” keys are reduced and output before “later” keys

Example public static class Reduce extends Map. Reduce. Base implements Reducer<Text, Int. Writable, Text, Int. Writable> { public void reduce(Text key, Iterator<Int. Writable> values, Output. Collector<Text, Int. Writable> output, Reporter reporter) throws IOException { int sum = 0; while (values. has. Next()) { sum += values. next(). get();

Output. Format Analogous to Input. Format Text. Output. Format – Writes “key valn” strings to output file Sequence. File. Output. Format – Uses a binary format to pack (k, v) pairs Null. Output. Format – Discards output

Presentation Overview Recall Hadoop Overview of the map-reduce paradigm Elaboration on the Word. Count example components of Hadoop that make Word. Count possible Major new example: N-Gram Generator step-by-step assembly of this map-reduce job Design questions to ask when creating your own Hadoop jobs

Major example: N-Gram Generation N-Gram is a common natural language processing technique (used by Google, etc) N-Gram is a subsequence of N items in a given sequence. (i. e. subsequence of words in a given text) Example 3 -grams (from Google) with corresponding occurrences ceramics collectables collectibles (55) ceramics collected by (52) ceramics collectibles cooking (45)

Understanding the process Someone wise said, “A week of writing code saves an hour of research. ” Before embarking on developing a Hadoop job, walk through the process step by step manually and understand the flow and manipulation of data. Once you can comfortably (and deterministically!) do it mentally, begin writing code.

Requirements Input: a beginning word/phrase n-gram size (bigram, trigram, n-gram) the minimum number of occurrences (frequency) whether letter case matters Output: all possible n-grams that occur sufficiently frequently.

High-level view of data flow Given: one or more files containing regular text. Look for the desired startword. If seen, take the next N-1 words and add the group to the database. Similarly to word count, find the number of occurrences of each N-gram. Remove those N-grams that do not occur frequently enough for our liking.

Follow along The N-grams implementation exists and is ready for your perusal. Grab it: if you use Git revision control: git clone git: //git. qnan. org/pmw/hadoop-ngram to get the files with your browser, go to: http: //www. qnan. org/~pmw/software/hadoop-ngram We used Project Gutenberg ebooks as input.

Follow along Start Hadoop bin/start-all. sh Grab the NGram code and build it: Type “ant” and all will be built Look at the README to see how to run it. Load some text files into your HDFS good source: http: //www. gutenberg. org Run it yourself (or see me do it) before we proceed.

Can we just use Word. Count? We have the Word. Count example that does a similar thing. But there are differences: We don't want to count the number of times our startword appears; we want to capture the subsequent words too. A more subtle problem is that wordcount maps one line at a time. That's a problem if we want 3 -grams with startword of “pillows” in the book containing this: The guests stayed in the guest bedroom; the pillows were delightfully soft and had a faint scent of mint.

Steps we must perform Read our text in paragraphs rather than in discrete lines: Record. Reader Input. Format Develop the mapper and reducer classes: first mapper: find startword, get the next N-1 words, and return <N-gram, 1> first reducer: sum the number of occurrences of each Ngram second mapper: no action second reducer: discard N-grams that are too rare Driver program

A new Record. Reader Ours must implement Record. Reader<K, V> Contain certain functions: create. Key(), create. Value(), get. Pos(), get. Progress(), next() Hadoop offers a Line. Record. Reader but no support for Paragraphs We'll need a Paragraph. Record. Reader Use Delegation Pattern instead of extending Line. Record. Reader. We couldn't extend it because it has private elements. Create new next() function

public synchronized boolean next(Long. Writable key, Text value) throws IOException { Text linevalue = new Text(); boolean appended, gotsomething; boolean retval; byte space[] = {' '}; value. clear(); gotsomething = false; do { appended = false; retval = lrr. next(key, linevalue); if (retval) { if (linevalue. to. String(). length() > 0) { byte[] rawline = linevalue. get. Bytes(); int rawlinelen = linevalue. get. Length(); value. append(rawline, 0, rawlinelen); value. append(space, 0, 1); appended = true; } gotsomething = true; } } while (appended); //System. out. println("Paragraph. Record. Reader: : next() returns "+gotsomething+" after setting value to: ["+value. to. String()+"]"); return gotsomething; }

A new Input. Format Given to the Job. Tracker during execution get. Record. Reader method This is the why we need Input. Format Must return our Paragraph. Record. Reader

public class Paragraph. Input. Format extends File. Input. Format<Long. Writable, Text> implements Job. Configurable { private Compression. Codec. Factory compression. Codecs = null; public void configure(Job. Conf conf) { compression. Codecs = new Compression. Codec. Factory(conf); } protected boolean is. Splitable(File. System fs, Path file) { return compression. Codecs. get. Codec(file) == null; } public Record. Reader<Long. Writable, Text> get. Record. Reader(Input. Split generic. Split, Job. Conf job, Reporter reporter) throws IOException { reporter. set. Status(generic. Split. to. String()); return new Paragraph. Record. Reader(job, (File. Split) generic. Split); } }

First stage: “Find” Mapper Define the startword at startup Each time map is called we parse an entire paragraph and output matching N-Grams Tell Reporter how far done we are to track progress Output <N-Gram, 1> like Word. Count output. collect(ngram, new Int. Writable(1)); This last part is important. . . next slide explains.

Importance of “output. collect()” Remember Hadoop's data type model: map: (K 1, V 1) → list(K 2, V 2) This means that for every single (K 1, V 1) tuple, the map stage can output zero, one, two, or any other number of tuples, and they don't have to match the input at all. Example: output. collect(ngram, new Int. Writable(1)); output. collect(“good-ol'-”+ngram, new Int. Writable(0));

Find Mapper Our mapper must have a configure() class public void configure(Job. Conf conf) { desired. Phrase = conf. get("mapper. desired-phrase"); Nvalue = conf. get. Int("mapper. N-value", 3); case. Sensitive = conf. get. Boolean("mapper. case-sensitive", false); } We can pass primitives through Job. Conf

“Find” Reducer Like Word. Count example Sum all the numbers matching our N-Gram Output <N-Gram, #of occurences>

Second stage: “Prune” Mapper Parse line from previous output and divide into Key/Value pairs “Prune” Reducer This way we can sort our elements by frequency If this N-Gram occurs fewer times than our minimum, trim it out

Piping data between M/R jobs How does the “Find” map/reduce job pass its results to the “Reduce” map/reduce job? I create a temporary file within HDFS. This temporary file is used as the output of Find and the input of Reduce. At the end, I delete the temporary file.

Counters The N-Gram generator has one programmerdefined counter: the number of partial/incomplete N-grams. These occur when a paragraph ends before we can read N-1 subsequent words. We can add as many counters as we want.

Job. Conf We need to set everything up 2 Jobs executing in series Find and Prune Job. Conf ngram_find_conf = new Job. Conf(get. Conf(), NGram. class), ngram_prune_conf = new Job. Conf(get. Conf(), NGram. class); User inputs parameters Starting N-Gram word/phrase N-Gram size Minimum frequency for pruning

Find Job. Conf Now we can plug everything in: ngram_find_conf. set. Job. Name("ngram-find"); ngram_find_conf. set. Input. Format(Paragraph. Input. Format. class); ngram_find_conf. set. Output. Key. Class(Text. class); ngram_find_conf. set. Output. Value. Class(Int. Writable. class); ngram_find_conf. set. Mapper. Class(Find. Job_Map. Class. class); ngram_find_conf. set. Reducer. Class(Find. Job_Reduce. Class. class); Also pass input parameters ngram_find_conf. set("mapper. desired-phrase", args. get(2), true)); ngram_find_conf. set. Int("mapper. N-value", new Integer(other_args. get(3)). int. Value()); ngram_find_conf. set. Boolean("mapper. case-sensitive", case. Sensitive); And point to our input and output files File. Input. Format. set. Input. Paths(ngram_find_conf, other_args. get(0)); File. Output. Format. set. Output. Path(ngram_find_conf, temp. Dir);

Prune Job. Conf Perform set up as before ngram_prune_conf. set. Job. Name("ngram-prune"); ngram_prune_conf. set. Int("reducer. min-freq", min_freq); ngram_prune_conf. set. Output. Key. Class(Text. class); ngram_prune_conf. set. Output. Value. Class(Int. Writable. class); ngram_prune_conf. set. Mapper. Class(Prune. Job_Map. Class. class); ngram_prune_conf. set. Reducer. Class(Prune. Job_Reduce. Class. class); We need to point our inputs to the outputs of the previous job File. Input. Format. set. Input. Paths(ngram_prune_conf, temp. Dir); File. Output. Format. set. Output. Path(ngram_prune_conf, new Path(other_args. get(1)));

Execute Jobs Run as blocking process with run. Job Batch processing is done in series Job. Client. run. Job(ngram_find_conf); Job. Client. run. Job(ngram_prune_conf);

Design questions to ask From where will my input come? Input. File. Format How is my input structured? Record. Reader (There already several common IFFs and RRs. Don't reinvent the wheel. ) Mapper and Reducer classes Do Key (Writable. Comparator) and Value (Writable) classes exist?

Design questions to ask Do I need to count anything while job is in progress? Where is my output going? Executor class What information do my map/reduce classes need? Must I block, waiting for job completion? Set File. Format?