GEC 21 The 21 st GENI Engineering Conference

GEC 21 The 21 st GENI Engineering Conference Oct 20 -23, 2014 -- Bloomington, IN, USA Hadoop-in-a-Hybrid-Cloud Luis Russi¹, Carlos R. Senna¹, Edmundo R. M. Madeira¹, Xuan Liu², Shuai Zhao², and Deep Medhi² ¹Institute of Computing, State University of Campinas – Brazil ²University of Missouri–Kansas City – USA

Agenda • Motivation and Objectives • Proposed Architecture – Web Cloud Portal – Execution Engine – Execution Service • Why using GENI • Testbed GEC 21 2014 - 2

Motivation and Objectives • Why – Hadoop installed in a private cloud may not have sufficient resources for all types of computational requirements – Need a seamless environment where Hadoop in a private cloud can access resources in other clouds • Hybird Cloud • An architecture to make the orchestration of Hadoop applications in hybrid clouds – Automatic preparation of a cross-domain cluster – Provisioning files – Making the results available to the user GEC 21 2014 - 3

Cont. . • Execution of Hadoop applications in hybrid cloud is not easy! – – – Spends time Needs technical knowledge Continuous evaluation of cloud resources On-demand preparation of public cloud resources Hybrid cloud requires an appropriate model that combines performance with minimal cost • GENI platforms allows us to test out the Hadoop in a hybrid cloud concept

The Proposed Architecture HM – Hadoop Master Node HW – Hadoop Worker Nodes GEC 21 2014 - 5

Web Cloud Portal • User interface • Management of files (application, data and submission) • Simple XML-Based submission file – Number of Virtual Machines (VM) – Image identification (Hadoop Master and Workers) – Requirements of VMs (memory, disk, flavor, etc) • Organizing the application workspace GEC 21 2014 - 6

Orchestration Engine • Prepares working place in the private cloud’s storage • Creates an Execution Service Instance (ESI) already associated with this cloud storage area • Releases the ESI to manage the application execution (asynchronously) • Copies the resulting files from the cloud storage to the user’s working place • Eliminates ESI • Notifies WCP GEC 21 2014 - 7

Execution Engine • ES Instance interacts with the private cloud monitoring system to evaluate the computational resources conditions • Checks for extra resources from the public cloud (if needed) • Automatic Hadoop Cluster preparation (Master and Workers) • Makes a copy of the resulting files from the HDFS to the cloud storage accessible by the Orchestration Engine • Eliminates all involved VMs • Notifies the Orchestration Engine about the end of processes • Monitors all stages of processing GEC 21 2014 - 8

Why using GENI? • Great environment for testing the Hybrid Cloud • High speed networks • Provisinable environments for cloud computing • Public cloud deployment • Cluster installation automation • API integration

UNICAMP-UMKC Hybrid Testbed Word Count Java software prototype GEC 21 2014 - 10

Initial Results ü Deploy exo. GENI virtual machines with Hadoop 2. 5. 1 ü Include the UMKC compute node at the UNICAMP cloud controller ü GRE Tunnel established between UMKC and UNICAMP GEC 21 2014 - 11

Future Work Ø Exo. GENI virtual machines and cloud Hadoop cluster joint Ø Execute the Wordcount Hadoop application at the cluster (http: //wiki. apache. org/hadoop/Word. Count) Ø Integrate GENI API to the private cloud framework GEC 21 2014 - 12

Hadoop-in-a-Hybrid-Cloud Thank you! Luis Russi¹, Carlos R. Senna¹, Edmundo R. M. Madeira¹, Xuan Liu², Shuai Zhao², and Deep Medhi² luisrussi@lrc. ic. unicamp. br¹, crsenna@ic. unicamp. br¹, edmundo@ic. unicamp. br¹, xuan. liu@mail. umkc. edu², shuai. zhao@mail. umkc. edu², dmedhi@umkc. edu² ¹Institute of Computing, State University of Campinas – Brazil ²University of Missouri–Kansas City – USA