Next Generation of Apache Hadoop Map Reduce Arun

  • Slides: 19
Download presentation
Next Generation of Apache Hadoop Map. Reduce Arun C. Murthy - Hortonworks Founder and

Next Generation of Apache Hadoop Map. Reduce Arun C. Murthy - Hortonworks Founder and Architect @acmurthy (@hortonworks) Formerly Architect, Map. Reduce @ Yahoo! 8 years @ Yahoo! © Hortonworks Inc. 2011 June 29, 2011

Hello! I’m Arun… • Architect & Lead, Apache Hadoop Map. Reduce Development Team at

Hello! I’m Arun… • Architect & Lead, Apache Hadoop Map. Reduce Development Team at Hortonworks (formerly at Yahoo!) • Apache Hadoop Committer and Member of PMC − Full-time contributor to Apache Hadoop since early 2006

Hadoop Map. Reduce Today • Job. Tracker − Manages cluster resources and job scheduling

Hadoop Map. Reduce Today • Job. Tracker − Manages cluster resources and job scheduling • Task. Tracker − Per-node agent − Manage tasks

Current Limitations • Scalability − Maximum Cluster size – 4, 000 nodes − Maximum

Current Limitations • Scalability − Maximum Cluster size – 4, 000 nodes − Maximum concurrent tasks – 40, 000 − Coarse synchronization in Job. Tracker • Single point of failure − Failure kills all queued and running jobs − Jobs need to be re-submitted by users • Restart is very tricky due to complex state • Hard partition of resources into map and reduce slots © Hortonworks Inc. 2011 5

Current Limitations • Lacks support for alternate paradigms − Iterative applications implemented using Map.

Current Limitations • Lacks support for alternate paradigms − Iterative applications implemented using Map. Reduce are 10 x slower. − Example: K-Means, Page. Rank • Lack of wire-compatible protocols − Client and cluster must be of same version − Applications and workflows cannot migrate to different clusters © Hortonworks Inc. 2011 6

Requirements • Reliability • Availability • Scalability - Clusters of 6, 000 -10, 000

Requirements • Reliability • Availability • Scalability - Clusters of 6, 000 -10, 000 machines − Each machine with 16 cores, 48 G/96 G RAM, 24 TB/36 TB disks − 100, 000+ concurrent tasks − 10, 000 concurrent jobs • Wire Compatibility • Agility & Evolution – Ability for customers to control upgrades to the grid software stack. © Hortonworks Inc. 2011 7

Design Centre • Split up the two major functions of Job. Tracker − Cluster

Design Centre • Split up the two major functions of Job. Tracker − Cluster resource management − Application life-cycle management • Map. Reduce becomes user-land library © Hortonworks Inc. 2011 8

Architecture

Architecture

Architecture • Resource Manager − Global resource scheduler − Hierarchical queues • Node Manager

Architecture • Resource Manager − Global resource scheduler − Hierarchical queues • Node Manager − Per-machine agent − Manages the life-cycle of container − Container resource monitoring • Application Master − Per-application − Manages application scheduling and task execution − E. g. Map. Reduce Application Master © Hortonworks Inc. 2011 10

Improvements vis-à-vis current Map. Reduce • Scalability − Application life-cycle management is very expensive

Improvements vis-à-vis current Map. Reduce • Scalability − Application life-cycle management is very expensive − Partition resource management and application life-cycle management − Application management is distributed − Hardware trends - Currently run clusters of 4, 000 machines ü 6, 000 2012 machines > 12, 000 2009 machines ü <16+ cores, 48/96 G, 24 TB> v/s <8 cores, 16 G, 4 TB> © Hortonworks Inc. 2011 11

Improvments vis-à-vis current Map. Reduce • Fault Tolerance and Availability − Resource Manager ü

Improvments vis-à-vis current Map. Reduce • Fault Tolerance and Availability − Resource Manager ü No single point of failure – state saved in Zoo. Keeper ü Application Masters are restarted automatically on RM restart ü Applications continue to progress with existing resources during restart, new resources aren’t allocated − Application Master ü Optional failover via application-specific checkpoint ü Map. Reduce applications pick up where they left off via state saved in HDFS © Hortonworks Inc. 2011 12

Improvements vis-à-vis current Map. Reduce • Wire Compatibility − Protocols are wire-compatible − Old

Improvements vis-à-vis current Map. Reduce • Wire Compatibility − Protocols are wire-compatible − Old clients can talk to new servers − Rolling upgrades © Hortonworks Inc. 2011 13

Improvements vis-à-vis current Map. Reduce • Innovation and Agility − Map. Reduce now becomes

Improvements vis-à-vis current Map. Reduce • Innovation and Agility − Map. Reduce now becomes a user-land library − Multiple versions of Map. Reduce can run in the same cluster (a la Apache Pig) ü Faster deployment cycles for improvements − Customers upgrade Map. Reduce versions on their schedule − Users can customize Map. Reduce e. g. HOP without affecting everyone! © Hortonworks Inc. 2011 14

Improvements vis-à-vis current Map. Reduce • Utilization − Generic resource model ü Memory ü

Improvements vis-à-vis current Map. Reduce • Utilization − Generic resource model ü Memory ü CPU ü Disk b/w ü Network b/w − Remove fixed partition of map and reduce slots © Hortonworks Inc. 2011 15

Improvements vis-à-vis current Map. Reduce • Support for programming paradigms other than Map. Reduce

Improvements vis-à-vis current Map. Reduce • Support for programming paradigms other than Map. Reduce − MPI − Master-Worker − Machine Learning − Iterative processing − Enabled by allowing use of paradigm-specific Application Master − Run all on the same Hadoop cluster © Hortonworks Inc. 2011 16

Summary • Map. Reduce. Next takes Hadoop to the next level − Scale-out even

Summary • Map. Reduce. Next takes Hadoop to the next level − Scale-out even further − High availability − Cluster Utilization − Support for paradigms other than Map. Reduce © Hortonworks Inc. 2011 17

Status – June, 2011 • Feature complete • Rigorous testing cycle underway − Scale

Status – June, 2011 • Feature complete • Rigorous testing cycle underway − Scale testing at ~500 nodes ü Sort/Scan/Shuffle benchmarks ü Grid. Mix. V 3! − Integration testing ü Pig integration complete! • Coming in the next release of Apache Hadoop! • Beta deployments of next release of Apache Hadoop at Yahoo! in Q 4, 2011 © Hortonworks Inc. 2011 18

Questions? http: //developer. yahoo. com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/ © Hortonworks Inc. 2011 19

Questions? http: //developer. yahoo. com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/ © Hortonworks Inc. 2011 19

Thank You. © Hortonworks Inc. 2011

Thank You. © Hortonworks Inc. 2011