YARN YARN YARN YARN DockerOpen Stack on YARN

目录 YARN简单介绍 YARN基础架构 YARN 作流程 YARN调度器介绍 Docker、Open. Stack on YARN 总结

资源申请流程 Resource Manager 4. Resource_Update Resource. Tracker. Service Resource. Scheduler 5. Scheduling and allocate

Docker on YARN Node. Manager提供三种Container. Executor实现： Default. Container. Executor：以管理员身份运行Container启动脚本。 Linux. Container. Executor：以Container所属用户身份运行Container启动脚本。使用cgroups进行资源隔离。 Docker.

Open. Stack & Yarn CMDB Other Cluster Open. Stack Cluster provisioning Monitor Cluster Hadoop

Open. Stack & YARN Sky. Form Scheduler NMmgr Sky. Form Node. Mgr Collect Server

其它运行在Yarn上的框架 HBase on YARN https: //github. com/hortonworks/hoya/ Impala On YARN http: //cloudera. github. io/llama/

总结 • 设计目标：资源统一管理与调度平台 • 基本组件： Client Node. Manager Resource. Manager Application. Master • 调度器：

Slides: 21

Download presentation

目录 YARN简单介绍 YARN基础架构 YARN 作流程 YARN调度器介绍 Docker、Open. Stack on YARN 总结

Resource Manager

Node Manager

Node Manager事件

资源申请流程 Resource Manager 4. Resource_Update Resource. Tracker. Service Resource. Scheduler 5. Scheduling and allocate resource Assigned containers 7. Containers as response 2. Node. Heartbeat 3. response 1. resource request 6. App. Master Heartbeat Node. Manager Application Master 8. Run tasks in containers Task

Docker on YARN Node. Manager提供三种Container. Executor实现： Default. Container. Executor：以管理员身份运行Container启动脚本。 Linux. Container. Executor：以Container所属用户身份运行Container启动脚本。使用cgroups进行资源隔离。 Docker. Container. Executor：在Docker中运行Container。

Open. Stack & Yarn CMDB Other Cluster Open. Stack Cluster provisioning Monitor Cluster Hadoop Cluster Sky. Form Scheduler

Open. Stack & YARN Sky. Form Scheduler NMmgr Sky. Form Node. Mgr Collect Server Analysis Server App mgr KVM VM VM libvirt XEN VM VM xenapi VMWARE VM VM api Sky. Form Plugin Open. Stack Scheduler

Open. Stack & YARN

其它运行在Yarn上的框架 HBase on YARN https: //github. com/hortonworks/hoya/ Impala On YARN http: //cloudera. github. io/llama/ Spark On YARN https: //github. com/apache/incubator-spark/tree/branch 0. 8/yarn Kubernetes On YARN https: //github. com/hortonworks/kubernetesyarn#kubernetes-yarn

总结 • 设计目标：资源统一管理与调度平台 • 基本组件： Client Node. Manager Resource. Manager Application. Master • 调度器：基于队列 FIFO、Capacity、Fair • 修改Node. Manager执行器支持Docker • 支持的框架： Hbase Spark Storm … 可方便的扩展