Hadoop Yihong Li Motivation Definition Advantages and disadvantages

Hadoop - Yihong Li

§ Motivation § Definition § Advantages and disadvantages Outline § HDFS § Map. Reduce § Conclusion

Motivation

Motivation § In the early 2000’s, Google developed the “Google File System” to support large distributed data-intensive applications § Shortly after, they developed “Map. Reduce” to allow developers to easily carry out large scale parallel computations - Examples: processing crawled documents, web request logs, etc. to produce inverted indices, statistics, etc. § Hadoop is an open source implementation of Google’s proprietary Map. Reduce framework; HDFS is the file system component of Hadoop

§ Hadoop is a distributed system infrastructure Hadoop definition § Maximize the use of clusters for high-speed computing and storage § The core design of Hadoop is: HDFS and Map. Reduce

§ Advantages: - High reliability with bit-wise storage and processing of data - Highly scalable - Efficient - High fault tolerance Hadoop § Disadvantages: - Not available for low latency data access - Cannot efficiently store large numbers of small files - Do not support multi-user writes and modify files arbitrarily

Hadoop Components HDFS: § Expandable file storage and delivery system § Divided into two parts: Namenode and Datenode

§ Highly fault tolerant § Support for large data sets HDFS Characteristics § Support streaming read data § Simple consistency § Mobile computing instead of mobile data § Portability between heterogeneous hardware and software platforms

Name. Node & Data. Node § The Namenode is the primary node in HDFS and is used to maintain and manage the blocks that exist on the Datanode. § User data does not reside on the Namenode and the data only resides on the Datanode.

Namenode features: § Maintain and manage the master daemon of the Datanode § Record metadata for all files stored in the cluster, such as block location, file size, permissions, hierarchy, and more § Each change that occurred on the file system metadata is logged. For example, if a file is deleted in HDFS, the Namenode will immediately log the operation in the Edit. Log § Receive heartbeat information and block reports from all Datanodes in the cluster on a regular basis to ensure that the Datanode is active § Keep records of all blocks in HDFS and the nodes where these blocks are located § Responsible for managing the copying of all blocks § In the event of a Datanode failure, the Namenode will select a new Datanode for the replica, balancing disk usage and managing traffic to the Datanode

Datanode features: § The actual data is stored on the Datanode § Execute the underlying read and write request of the file system client § Send heartbeat reports to Namenodes on a regular basis. Overall health of HDFS. The default frequency is 3 seconds.

Hadoop Components Map. Reduce § Hadoop Map. Reduce: A distributed offline parallel framework § Divided into two stages of map and reduce, the map stage processes the input data in parallel, and the reduce stage summarizes the map results.

Map. Reduce Execution § As can be seen from the diagram on the left, after the client submits the calculation task, the file is first read. By default, a block will have a Maptask for processing. After the processing is completed, the Shuffle phase passes through a series of partitions. , sort, protocol, grouping, enter the reduce end, the reduce end is processed, the output processing result.

§ The main function of hadoop is to handle big data. Hadoop's Conclusion distributed architecture, the big data processing engine is as close as possible to the storage. Hadoop's Map. Reduce function breaks down a single task and sends the fragmented task to multiple nodes, then loads the data into a single data set. In the warehouse.

§ Borthakur, Dhruba. "The hadoop distributed file system: Architecture and design. " (2008). § Shvachko, Konstantin, et al. "The hadoop distributed file system. " MSST. Vol. 10. 2010. References § Lam, Chuck. Hadoop in action. Manning Publications Co. , 2010. § White, Tom. Hadoop: The definitive guide. " O'Reilly Media, Inc. ", 2012. § Bhandarkar, Milind. "Map. Reduce programming with apache Hadoop. " 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS). IEEE, 2010.