Introduction to Hadoop and Spark Antonino Virgillito THE

Introduction to Hadoop and Spark Antonino Virgillito THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION Eurostat

Large-scale Computation • Traditional solutions for computing large quantities of data relied mainly on processor • Complex processing made on data moved in memory • Scale only by adding power (more memory, faster processor) • Works for relatively small-medium amounts of data but cannot keep up with larger datasets • How to cope with today’s indefinitely growing production of data? • Terabytes per day 2 Eurostat

Distributed Computing • Multiple machines connected among each other and cooperating for a common job • «Cluster» • Challenges • Complexity of coordination – all processes and data have to be maintained syncronized about the global system state • Failures • Data distribution 3 Eurostat

Hadoop • Open source platform for distributed processing of large datasets • Based on a project developed at Google • Functions: • Distribution of data and processing across machines • Management of the cluster • Simplified programming model • Easy to write distributed algorithms Eurostat

Hadoop scalability Hadoop can reach massive scalability by exploiting a simple distribution architecture and coordination model Huge clusters can be made up using (cheap) commodity hardware Cluster can easily scale up with little or no modifications to the programs Eurostat

Hadoop Concepts • Applications are written in common high-level languages • Inter-node communication is limited to the minimum • Data is distributed in advance • Bring the computation close to the data • Data is replicated for availability and reliability • Scalability and fault-tolerance 6 Eurostat

Scalability and Fault-tolerance • Scalability principle • Capacity can be increased by adding nodes to the cluster • Increasing load does not cause failures but in the worst case only a graceful degradation of performance • Fault-tolerance • Failure of nodes are considered inevitable and are coped with in the architecture of the platform • System continues to function when failure of a node occurs – tasks are re-scheduled • Data replication guarantees no data is lost • Dynamic reconfiguration of the cluster when nodes join and leave Eurostat 7

Benefits of Hadoop • Previously impossible or impractical analysis made possible • Lower cost of hardware • Less time • Ask Bigger Questions 8 Eurostat

Hadoop Components Hive Pig Sqoop Flume Mahout Oozie HBase Core Components 9 Eurostat

Hadoop Core Components • HDFS: Hadoop Distributed File System • Abstraction of a file system over a cluster • Stores large amount of data by transparently spreading it on different machines • Map. Reduce • Simple programming model that enables parallel execution of data processing programs • Executes the work on the data near the data • In a nutshell: HDFS places the data on the cluster and Map. Reduce does the processing work Eurostat

Structure of an Hadoop Cluster • Hadoop Cluster: • Group of machines working together to store and process data • Any number of “worker” nodes • Run both HDFS and Map. Reduce components • Two “Master” nodes • Name Node: manages HDFS • Job Tracker: manages Map. Reduce 11 Eurostat

Hadoop Principle Hadoop is basically a middleware platform that manages a cluster of machines I’m one big data set The core components is a distributed file system (HDFS) Files in HDFS are split into blocks that are scattered over the cluster Hadoop HDFS The cluster can grow indefinitely simply by adding new nodes Eurostat

The Map. Reduce Paradigm Parallel processing paradigm Programmer is unaware of parallelism Programs are structured into a two-phase execution Map Reduce x 4 x 5 x 3 Data elements are classified into categories Eurostat An algorithm is applied to all the elements of the same category

Map. Reduce Concepts • Automatic parallelization and distribution • Fault-tolerance • A clean abstraction for programmers • Map. Reduce programs are usually written in Java • Can be written in any language using Hadoop Streaming • All of Hadoop is written in Java • Map. Reduce abstracts all the ‘housekeeping’ away from the developer • Developer can simply concentrate on writing the Map and Reduce functions Eurostat 14

Map. Reduce and Hadoop Map. Reduce is logically placed on top of HDFS Map. Reduce HDFS Eurostat

Map. Reduce and Hadoop MR works on (big) files loaded on HDFS Hadoop MR MR HDFS Each node in the cluster executes the MR program in parallel, applying map and reduces phases on the blocks it stores Output is written on HDFS Scalability principle: Perform the computation were the data is Eurostat

Hive • Apache Hive is a high-level abstraction on top of Map. Reduce – Uses an SQL/like language called Hive. QL – Generates Map. Reduce jobs that run on the Hadoop cluster – Originally developed by Facebook for data warehousing – Now an open/source Apache project 17 Eurostat

Overview • Hive. QL queries are transparently mapped into Map. Reduce jobs at runtime by the Hive execution engine • Also makes optimizations • Jobs are submitted to the Hadoop cluster 18 Eurostat

Hive Tables • Hive works on the abstraction of table, similar to a table in a relational database • Main difference: a Hive table is simply a directory in HDFS, containing one or more files • By default files are in text format but different formats can be specified • The structure and location of the tables are stored in a backing SQL database called the metastore • Transparent for the user • Can be any RDBMS, specified at configuration time 19 Eurostat

Hive Tables • At query time, the metastore is consulted to check if the query is consistent with the tables it invokes • The query itself operates on the actual data files stored in HDFS 20 Eurostat

Hive Tables • By default, tables are stored in a warehouse directory on HDFS • Default location: /user/hive/warehouse/<db>/<table> • Each subdirectory of the warehouse directory is considered a database • Each subdirectory of a database directory is a table • All files in a table directory are considered part of the table when querying • Must have the same structure 21 Eurostat

Pig • Tool for querying data on Hadoop clusters • Widely used in the Hadoop world • Yahoo! estimates that 50% of their Hadoop workload on their 100, 000 CPUs clusters is genarated by Pig scripts • Allows to write data manipulation scripts written in a high-level language called Pig Latin • Interpreted language: scripts are translated into Map. Reduce jobs • Mainly targeted at joins and aggregations Eurostat

Overview of Pig • Pig. Latin • Language for definition of data flow • Grunt • Interactive shell for typing and executing Pig. Latin statements • Interpreter and execution engine 23 Eurostat

RHadoop • Collection of packages that allows integration of R with HDFS and Map. Reduce • Hadoop provides the storage while R brings the processing • Just a library • Not a special run-time, Not a different language, Not a special purpose language • Incrementally port your code and use all packages • Requires R installed and configured on all nodes in the cluster Eurostat

RHadoop Packages • rhdfs • Interface for reading and writing files from/to a HDFS cluster • rmr 2 • Interface to Map. Reduce through R • rhbase • Interface to HBase Eurostat

rhdfs • As Hadoop Map. Reduce programs use HDFS for taking their input and writing their output, it is necessary to access them from R console • The R programmer can easily perform read and write operations on distributed data files. • Basically, rhdfs package calls the HDFS API in backend to operate data sources stored on HDFS. Eurostat

rmr 2 • rmr 2 is an R interface for providing Hadoop Map. Reduce facility inside the R environment. • So, the R programmer needs to just divide their application logic into the map and reduce phases and submit it with the rmr 2 methods. • After that, rmr 2 calls the Hadoop streaming Map. Reduce API with several job parameters as input directory, output directory, mapper, reducer, and so on, to perform the R Map. Reduce job over Hadoop cluster. Eurostat

mapreduce • The mapreduce function takes as input a set of named parameters • • • input: input path or variable input. format: specification of input format output: output path or variable map: map function reduce: reduce function • map and reduce function present the usual interface • A call to keyval(k, v) inside the map and reduce function is used to emit respectively intermediate and output key-value pairs Eurostat

Word. Count in R wordcount = function( input, output = NULL, pattern = " "){ wc. reduce = function(word, counts ) { keyval(word, sum(counts))} mapreduce( input = input , output = output, input. format = "text", map = wc. map, reduce = wc. reduce, combine = T)} wc. map = function(. , lines) { keyval( unlist( strsplit( x = lines, split = pattern)), 1)} Eurostat

Reading delimited data tsv. reader = function(con, nrecs){ lines = read. Lines(con, 1) if(length(lines) == 0) NULL else { delim = strsplit(lines, split = "t") keyval( sapply(delim, function(x) x[1]), sapply(delim, function(x) x[-1]))}} freq. counts = mapreduce( input = tsv. data, input. format = tsv. format, map = function(k, v) keyval(v[1, ], 1), reduce = function(k, vv) keyval(k, sum(vv))) Eurostat

Reading named columns tsv. reader = function(con, nrecs){ lines = read. Lines(con, 1) if(length(lines) == 0) NULL else { delim = strsplit(lines, split = "t ") keyval(sapply(delim, function(x) x[1]), data. frame( location = sapply(delim, function(x) x[2]), name = sapply(delim, function(x) x[3 ]), value = sapply(delim, function(x) x[4])))}} freq. counts = mapreduce( input = tsv. data, input. format = tsv. format, map = function(k, v) { filter = (v$name == "blarg") keyval(k[filter], log(as. numeric(v$value[filter ])))}, reduce = function(k, vv) keyval(k, mean(vv))) Eurostat

Apache Spark • A general purpose framework for big data processing • It interfaces with many distributed file systems, such as Hdfs (Hadoop Distributed File System), Amazon S 3, Apache Cassandra and many others • 100 times faster than Hadoop for in-memory computation 32 Eurostat

Multilanguage API • You can write applications in various languages • • Java Python Scala R • In the context of this course we will consider Python 33 Eurostat

Built-in Libraries 34 Eurostat

RDD - Resilient Distributed Dataset • The RDD is the core abstraction used by Spark to work on data • A RDD is a collection of elements partitioned in every cluster node. Spark operates in parallel on them • Every RDD is created from a file on Hadoop filesystem • They can be made persistent in memory 35 Eurostat

Transformations • For example, map is a transformation that takes all elements of the dataset, pass them to a function and returns another RDD with the results result. RDD = original. RDD. map(my. Function) 36 Eurostat

Actions • For example, reduce is an action. It aggregates all elements of the RDD using a function and returns the result to the driver program result = rdd. reduce(function) 37 Eurostat

Spark. SQL and Data. Frames • Spark. SQL is the spark module for structured data processing • Data. Frame API is one of the ways to interact with Spark. SQL 38 Eurostat

Data. Frames • A Data. Frame is a collection of data organized into columns • Similar to tables in relational databases • Can be created from various sources: structured data files, Hive Tables, external db, csv etc… 39 Eurostat

Example operations on Data. Frames • To show the content of the Data. Frame • df. show() • To print the Schema of the Data. Frame • df. print. Schema() • To select a column • df. select(‘column. Name’). show() • To filter by some parameter • df. filter(df[‘column. Name’] > N). show() 40 Eurostat