Map Reduce Map Reduce Map Reduce Log Processing

  • Slides: 44
Download presentation

Map / Reduce

Map / Reduce

Map / Reduce Το Map / Reduce χρησιμοποιείται κυρίως για: ◦ Log Processing. ◦

Map / Reduce Το Map / Reduce χρησιμοποιείται κυρίως για: ◦ Log Processing. ◦ Web Index Building. ◦ Data mining και Machine Learning.

Map / Reduce

Map / Reduce

Configuration (hadoopsite. xml) <configuration> <property> <name>fs. default. name</name> <value>localhost: 9000</value> </property> <name>mapred. job. tracker</name>

Configuration (hadoopsite. xml) <configuration> <property> <name>fs. default. name</name> <value>localhost: 9000</value> </property> <name>mapred. job. tracker</name> <value>localhost: 9001</value> </property> <name>dfs. replication</name> <value>1</value> </property> </configuration>

Configuration {path_prefix}/bin/hadoop namenode – format ◦ Format καινούριου DFS στον namenode {path_prefix}/bin/start-all. sh ◦

Configuration {path_prefix}/bin/hadoop namenode – format ◦ Format καινούριου DFS στον namenode {path_prefix}/bin/start-all. sh ◦ Έναρξη του hadoop daemon … {path_prefix}/bin/stop-all. sh ◦ Τερματισμού του daemon

Random. Text. Writer Εκτέλεση: {path_prefix}/bin/hadoop jar hadoop${version}-examples. jar randomtextwriter <out-dir> [conf_file] Submit του grep

Random. Text. Writer Εκτέλεση: {path_prefix}/bin/hadoop jar hadoop${version}-examples. jar randomtextwriter <out-dir> [conf_file] Submit του grep job στον Job. Tracker Επικοινωνία του Job. Tracker με τον Name. Node για εξεύρεση των κατάλληλων Task. Trackers Ο Job. Tracker κάνει submit την δουλειά στους διάφορους Task. Trackers

Εκτέλεση Random. Text. Writer [orestis: hadoop] bin/hadoop jar hadoop-*-examples. jar randomwriter rand_output rtw. conf

Εκτέλεση Random. Text. Writer [orestis: hadoop] bin/hadoop jar hadoop-*-examples. jar randomwriter rand_output rtw. conf Running 10 maps. Job started: Tue Apr 07 19: 00: 48 EEST 2009 09/04/07 19: 00: 49 INFO mapred. Job. Client: Running job: job_200904071847_0001 09/04/07 19: 00: 50 INFO mapred. Job. Client: map 0% reduce 0% 09/04/07 19: 02: 41 INFO mapred. Job. Client: map 20% reduce 0% 09/04/07 19: 04: 34 INFO mapred. Job. Client: map 30% reduce 0% 09/04/07 19: 04: 35 INFO mapred. Job. Client: map 40% reduce 0% 09/04/07 19: 06: 21 INFO mapred. Job. Client: map 50% reduce 0% 09/04/07 19: 06: 24 INFO mapred. Job. Client: map 60% reduce 0% 09/04/07 19: 08: 07 INFO mapred. Job. Client: map 70% reduce 0% 09/04/07 19: 08: 09 INFO mapred. Job. Client: map 80% reduce 0% 09/04/07 19: 09: 56 INFO mapred. Job. Client: map 90% reduce 0%. . Job ended: Tue Apr 07 19: 09: 57 EEST 2009 The job took 548 seconds.

Εκτέλεση Random. Text. Writer [orestis: hadoop] bin/hadoop dfs -ls Found 1 items drwxr-xr-x -

Εκτέλεση Random. Text. Writer [orestis: hadoop] bin/hadoop dfs -ls Found 1 items drwxr-xr-x - orestis supergroup 0 2009 -04 -07 19: 09 /user/orestis/rand_output [orestis: hadoop] bin/hadoop dfs -ls rand_output/ Found 11 items drwxr-xr-x - orestis supergroup 0 2009 -04 -07 19: 00 /user/orestis/rand_output/_logs -rw-r--r-- 1 orestis supergroup 1077289470 2009 -04 -07 19: 00 /user/orestis/rand_output/part 00000 -rw-r--r-- 1 orestis supergroup 1077275793 2009 -04 -07 19: 00 /user/orestis/rand_output/part 00001 -rw-r--r-- 1 orestis supergroup 1077283821 2009 -04 -07 19: 02 /user/orestis/rand_output/part 00002 -rw-r--r-- 1 orestis supergroup 1077298379 2009 -04 -07 19: 02 /user/orestis/rand_output/part 00003 -rw-r--r-- 1 orestis supergroup 1077292822 2009 -04 -07 19: 04 /user/orestis/rand_output/part 00004 -rw-r--r-- 1 orestis supergroup 1077286019 2009 -04 -07 19: 04 /user/orestis/rand_output/part 00005 -rw-r--r-- 1 orestis supergroup 1077287527 2009 -04 -07 19: 06 /user/orestis/rand_output/part 00006 -rw-r--r-- 1 orestis supergroup 1077287446 2009 -04 -07 19: 06 /user/orestis/rand_output/part 00007

Εκτέλεση Grep [orestis: hadoop] bin/hadoop jar hadoop-*-examples. jar grep rand_output grep_output 'here' 09/04/07 19:

Εκτέλεση Grep [orestis: hadoop] bin/hadoop jar hadoop-*-examples. jar grep rand_output grep_output 'here' 09/04/07 19: 12: 36 INFO mapred. File. Input. Format: Total input paths to process : 10 09/04/07 19: 12: 37 INFO mapred. Job. Client: Running job: job_200904071847_0002 09/04/07 19: 12: 38 INFO mapred. Job. Client: map 0% reduce 0% 09/04/07 19: 12: 52 INFO mapred. Job. Client: map 1% reduce 0%. . . 09/04/07 19: 31: 46 INFO mapred. Job. Client: HDFS bytes read=107 09/04/07 19: 31: 46 INFO mapred. Job. Client: Local bytes read=29 [orestis: hadoop] bin/hadoop dfs -ls Found 2 items drwxr-xr-x - orestis supergroup 0 2009 -04 -07 19: 31 /user/orestis/grep_output drwxr-xr-x - orestis supergroup 0 2009 -04 -07 19: 09 /user/orestis/rand_output [orestis: hadoop] bin/hadoop dfs -ls grep_output Found 2 items drwxr-xr-x - orestis supergroup -rw-r--r-- 1 orestis supergroup 0 2009 -04 -07 19: 31 /user/orestis/grep_output/_logs 7 2009 -04 -07 19: 31 /user/orestis/grep_output/part-00000 [orestis: hadoop] bin/hadoop dfs -cat grep_output/part-00000 4 here

Βιβλιογραφία http: //wiki. apache. org/hadoop/ http: //en. wikipedia. org/wiki/Hadoop http: //en. wikipedia. org/wiki/Map. Reduc

Βιβλιογραφία http: //wiki. apache. org/hadoop/ http: //en. wikipedia. org/wiki/Hadoop http: //en. wikipedia. org/wiki/Map. Reduc e

Log Parser Log Parser Log Parser Log ParserLog Parser Log Parser Log Parser Log Parser
Map Reduce Map Reduce Outline Map Reduce ArchitectureMap Reduce Map Reduce Outline Map Reduce Architecture
HadoopMap Reduce Map Reduce Map Reduce Part 1HadoopMap Reduce Map Reduce Map Reduce Part 1
HadoopMap Reduce Map Reduce Map Reduce Part 1HadoopMap Reduce Map Reduce Map Reduce Part 1
Map reduce distributed processing technologies Hadoop Map ReduceMap reduce distributed processing technologies Hadoop Map Reduce
2 Log 2x32 2 Log 222 Log 22 Log 2x32 2 Log 222 Log 2
Ex log ab log ab log a bEx log ab log ab log a b
Ex log ab log ab log a bEx log ab log ab log a b
CHAPTER 6 Map Reduce Outline Map Reduce MapCHAPTER 6 Map Reduce Outline Map Reduce Map
Hadoop Map Reduce Hadoop Map Reduce Hadoop MapHadoop Map Reduce Hadoop Map Reduce Hadoop Map
982021 Map Reduce MAP REDUCE 1 982021 Map982021 Map Reduce MAP REDUCE 1 982021 Map
DATA PROCESSING EDITING CODING PROCESSING Data Processing ProcessingDATA PROCESSING EDITING CODING PROCESSING Data Processing Processing
Map Reduce Types Formats and Features Map ReduceMap Reduce Types Formats and Features Map Reduce
Map Reduce Google and Map Reduce n nMap Reduce Google and Map Reduce n n
Map Reduce Hadoop Implementation Outline Map Reduce overviewMap Reduce Hadoop Implementation Outline Map Reduce overview
Map Reduce Parallel Computing Map Reduce Examples ParallelMap Reduce Parallel Computing Map Reduce Examples Parallel
Distributed Computations Map Reduce Dryad Map Reduce slidesDistributed Computations Map Reduce Dryad Map Reduce slides
Map Reduce Programming Map Reduce Recap Programmers mustMap Reduce Programming Map Reduce Recap Programmers must
Map Reduce Google and Map Reduce n nMap Reduce Google and Map Reduce n n
Map Reduce Outline r Map Reduce overview rMap Reduce Outline r Map Reduce overview r
Map Reduce Parallel Computing Map Reduce Examples ParallelMap Reduce Parallel Computing Map Reduce Examples Parallel
Map Reduce What is Map Reduce 1 AMap Reduce What is Map Reduce 1 A
Map Reduce Based on Map Reduce Simplified DataMap Reduce Based on Map Reduce Simplified Data
Cloud Map Reduce A Map Reduce Implementation onCloud Map Reduce A Map Reduce Implementation on