Recent Advances in Grid Computing and Business Models

Recent Advances in Grid Computing and Business Models: A Gridbus Perspective Grid Business Symposium 2005, Seoul, Korea Rajkumar Buyya Grid and Distributed Systems (GRIDS) Laboratory Dept. of Computer Science and Software Engineering The University of Melbourne, Australia www. gridbus. org WW Grid

Outline n n n Introduction n Utility Networks and Grid Computing n World-wide with Australia and India Perspective n Architecture, Design and Implementation Global Grids and Challenges Grid Initiatives Introduction to Gridbus Project and Grid Economy Grid Service Broker Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids n n 2 n A Case Study in High Energy Physics Economy-based Scheduling in Data Grids Summary

4 Essential Utilities and Delivery Networks (1) Water (2) Electricity (3) Gas (4) Telephone 3

(5) IT services as the fifth utility (water, electricity, gas, telephone, IT) 4 e. Science e. Business e. Government e. Health Multilingual e. Education …

A Bird Eye View of World-Wide Grid Environment Grid Information Service Grid Resource Broker R 2 R 3 R 5 Application database R 4 RN Grid Resource Broker R 6 Grid Information Service 5 R 1 Resource Broker

Grid Challenges Security Computational Economy Uniform Access Resource Discovery 7 Resource Allocation & Scheduling Application Construction System Management Data locality Network Management

Some Grid Initiatives Worldwide n n Australia n n n Brazil n n n China 120 million – 5 yrs n n Europe n 450 million – 5 yrsn 486 million – 5 yrsn n India 1. 3 billion (Rs) § n Japan n Nimrod-G Gridbus DISCWorld Grange. Net. 27 million APACGrid ARC e. Research n n n n n UK e. Science EU Grids. . and many more. . . n n Singapore NGP n n Global Grid Forum Australian Grid Forum Conferences: n n 8 1. 3 billion – 3 yrs IBM On Demand Computing HP Adaptive Computing Sun N 1 Microsoft -. NET Oracle 10 g Satyam – Grid Practice Infosys, Wipro, TCS Storage. Tek –Grid. . Public Forums n NAGERI Globus NASA IPG Access. Grid Tera. Grid Cyberinfrasture Industry Initiatives n China. Grid – Education CNGrid - application 1 billion – 5 nyrs. Korea. . . N*Grid n n Our. Grid, Easy. Grid LNCC-Grid + many others I-Grid USA CCGrid HPDC E-Science http: //www. gridcomputing. com 2? billion

Grid Computing in Australia (Courtesy: Jihyoun Park, SNU Visitor to Melbourne) Government Academia Collaboration Indus try 10

Academic activities 1 University laboratories for Grid computing - Uni. of Melbourne(GRIDS lab): Gridbus (Grid. Sim, GMD, Grid. Bank, Alchemi, . . ), Master of Engineering in Distributed Computing - Monash Uni. : Griddls. S (Legacy SW to the computational grid), Nimrod-G - Australian national Uni. (Internet Futures Group) - Sydney Uni. (Vi. SLAB): high performance visualization &computing - Uni. of Adelaide (DHPC Group): DISCWorld - Queensland Uni. of Technology (PLAS): G 2 (. NET based) 2 Grid Infrastructure Projects APACGrid, National Neurosciece Facility, Australian Virtual Observatory, several state level facilities (VPAC, TPAC, SAPAC, QPSF, IVEC) 3 Grid Applications * Asia Pacific Bioinformatics Network/ Virtual Drug Design: Molecular Modeling for Drug Design on P 2 P Grid/ HEPGrid: High Energy Physics and the Grid Network/ Access Grid/Australian Computational Earth Systems Simulator/. * Recently 30 more applications are funded as part of ARC e-Research * Govt. has formed “National e-Research Coordination Committee”. 11

Grid Computing in India Academia Governm Collabor ent ation Industry (majority focus on Grid integration) 12

Grid Computing in India: Academic and Industrial Activities n n n 13 Academic and Government Initiatives: n TIFR, IITM, Anna University, IITD, Uo. H, etc. n C-DAC’s Garuda – Ministry of IT Software Companies in India: n Top 4 Indian IT Companies: Satyam, Infosys, TCS (Tata Consultancy Service), and Wipro. n Oracle 10 g, IBM, HP, Sun ertc. have a large Grid development centers in Bangalore, India. Satyam is leading the pack in Grid Business push: n Grid Practice Centre with top management support. n Singned Mo. U with Melbourne University and extensively using Gridbus in powering applications. n Also contributing the development of Gridbus technologies (e. g. , Alchemi) – SEI CMM Level 5 principles. n Application Verticals: Manufacturing, Security, Life Sciences, Finance

14

Australian and Indian Grid Efforts Compared Australia Korea: Is it like Australia or India? 15 India

The Gridbus Project @ Melbourne: Enable Leasing of ICT Services on Demand Distributed Data WWG Gridbus World Wide Grid! On Demand Utility Computing 16

The Gridbus Project: http: //www. gridbus. org n A multi-institutional “Open Source” R&D Project with focus on: n n n n Alchemi: Harnessing. NET/Windows-based Resources Grid Market Directory and Web Services Grid Bank: Accounting and Transaction Management Visual Tools for Creation of Distributed Applications Workflow Composition and Deployment Services Data Grid Brokering and Grid Economy Services Data Replication Strategies Grid. Sim Toolkit: Enhanced to support Data Grid, Reservation, etc. Libra: SLA-based Allocation of Cluster Resources Coupling of Clusters and Computational Economy WWG: Global Data Intensive Grid Testbed Application Enabler Projects: n n 17 Architecture, Specification, and Open Source Reference Implementation. Service-Oriented Grid, Utility Computing & Distributed Data and Computation Economy Scaling from Desktops, Cluster Federation, Enterprise Grids to Global Grids. High-Energy Physics , Astronomy, Brain Activity Analysis – Osaka U. , Natural Language Processing, Portfolio Analysis – Spain, Bio. Grid - WEHI (via APACGrid), Sensor. Grid (NICTA), Medical Imaging (HFI) Supported by:

Grid Economy: Methodology for Sustained Resourced Sharing and Managing Supply-and-Demand for Resources 18

Grid Entities and Architecture Grid consumer GSP site scheduler broker accounting 20 GSP global scheduler GSP site scheduler Resource owners Market Maker End users Private enterprises National providers

A Reference Service-Oriented Architecture for Utility Grids Data Catalogue Grid Bank Job Control Agent Grid Node N Secure Schedule Advisor Qo. S Grid Node 1 Grid Resource Broker Trading … Deployment Agent Job. Exec Misc. services Resource Allocation Storage Grid Middleware Services Accounting Resource Reservation R 1 Grid Consumer 21 Pricing Algorithms Trade Server Trade Manager Health Monitor … Grid Explorer Info ? Information Service … Programming Environments Applications Sign-on Grid Market Services R 2 … Rm Grid Service Providers

Gridbus and Complementary Technologies – realizing Utility Grid Science Commerce … MPI Excell. Grid Brokers: Grid Economy Alchemi Nordu. Grid Windows Gridscape Unicore … XGrid JVM Solaris Collaboratories Workflow Engine Nimrod-G Globus . NET Engineering Grid Storage Economy Condor Linux … IRIX Libra Tomcat Mac OSF 1 PDB Worldwide Grid Core Grid Middleware Grid Market Directory CDB 22 User-Level Middleware (Grid Tools) Gridbus Data Broker SGE AIX X-Parameter Sweep Lang. Grid Exchange & Federation Grid Bank PBS … Grid Applications Portals G R I D S I M Grid Fabric Software Grid Fabric Hardware

Alchemi: . NET-based Enterprise Grid Platform & Web Services Alchemi Manager Web Services Internet Alchemi Users Internet 23 • SETI@Home like Model • General Purpose • Dedicated/Non-dedicate workers • Role-based Security • . NET and Web Services • C# Implementation • Grid. Thread and Job Model Programming • Easy to setup and use • Widely in use! Alchemi Worker Agents

Some Users of Alchemi Tier Technologies, USA Large scale document processing using Alchemi framework Satyam Computers Applied Research Laboratory, India Micro-array data processing using Alchemi framework CSIRO, Australia Natural Resource Modeling The University of Sao Paulo, Brazil The Alchemi Executor as a Windows Service The Friedrich Miescher Institute (FMI) for Biomedical Research, Switzerland Patterns of transcription factors in mammalian genes 24 stochastix Gmb. H, Germany Asynchronous Excel Tasks using Managed. XLL and Alchemi. Net Grid Computing framework. Many users in Universities: See next for an example.

On Demand Assembly of Services: Putting Them All Together Application Code Explore data 1 Data Source Visual Application Composer Data 10 ts+ l u Res t Info Cos 2 Data Catalogue 6 Grid Resource Broker ASP Catalogue lts 9 8 Grid Service (GS) (Globus) 4 Grid Info Service 3 Job Data Replicator (GDMP) 5 Grid Market Directory 7 Bill Alchemi GS Cluster Scheduler PE 25 Grid Service Provider (GSP) (e. g. , CERN) CPU or PE GSP (e. g. , IBM) 12 Resu (Instruments/dis tributed sources) Cluster Scheduler PE GSP (e. g. , Uof. M) PE GTS GSP (e. g. , VPAC) 11 Gridbus Grid. Bank GSP (Accounting Service)

The Gridbus Grid Service Broker for Data Grid Applications Builds on the Nimrod-G Computational Grid Broker and Computational Economy [Buyya, Abramson, Giddy, Monash University, 1999 -2001] And Extends its notion for Data and Service Grids

Gridbus Broker Architecture Gridbus Client App, T, $, Opt (Bag of Tasks Applications) (Data Grid Scheduler) Gridbus Farming Engine Schedule Advisor Trading Manager Record Keeper Grid Dispatcher Grid Explorer Grid Middleware TM $ TS GE GIS, NWS Grid Info Server RM & TS $ $ U G Globus enabled node. 27 G L Data Node C Unicore enabled node. A RM: Local Resource Manager, TS: Trade Server Alchemi enabled node. Data Catalog

Gridbus Services for e. Science applications n Application Development Environment: n n n Resource Allocation and Scheduling n n Dynamic discovery of optional computational and data nodes that meet user Qo. S requirements. Hide Low-Level Grid Middleware interfaces n 29 XML-based language for composition of task farming (legacy) applications as parameter sweep applications. Task Farming APIs for new applications. Web APIs (e. g. , Portlets) for Grid portal development. Threads-based Programming Interface Workflow interface and Gridbus-enabled workflow engine. Globus, Alchemi, Unicore, Nordu. Grid, XGrid, etc.

Click Here for Demo Figure 3 : Logging into the portal. 30 Drug Design Made Easy!

Economy-based Data Grid Scheduling CLICK HERE TO SKIP IF RUNNING OUT of TIME High Energy Physics as e. Science Application Case Study

Australian Belle Data Grid Testbed 32

Case Study: Event Simulation and Analysis B 0 ->D*+D*-Ks • Simulation and Analysis Package - Belle Analysis Software Framework (BASF) • Experiment in 2 parts – Generation of Simulated Data and Analysis of the distributed data Analyzed 100 data files (30 MB each) were distributed among the five nodes 33

Resources Used and their Service Price 34 Organization Node details Role Cost (in G$/CPUsec) CS, Uni. Melb belle. cs. mu. oz. au 4 CPU, 2 GB RAM, 40 GB HD, Linux Broker host, Data host, NWS server N. A. (Not used as a compute resource) Physics, Uni. Melb fleagle. ph. unimelb. edu. au 1 CPU, 512 MB RAM, 40 GB HD, Linux Replica Catalog host, Data host, Compute resource, NWS sensor 2 CS, University of Adelaide belle. cs. adelaide. edu. au 4 CPU (only 1 available) , 2 GB RAM, 40 GB HD, Linux Data host, NWS sensor N. A. (Not used as a compute resource) ANU, Canberra belle. anu. edu. au 4 CPU, 2 GB RAM, 40 GB HD, Linux Data host, Compute 4 resource, NWS sensor Dept of Physics, USyd belle. physics. usyd. edu. au 4 CPU (only 1 available), 2 GB RAM, 40 GB HD, Linux Data host, Compute 4 resource, NWS sensor VPAC, Melbourne brecca-2. vpac. org 180 node cluster (only head node used), Linux Compute resource, NWS sensor 6

Network Cost (in Grid $/Currency!) 35

Deploying Application Scenario n n A data grid scenario with 100 jobs and each accessing remote data of ~30 MB Deadline: 3 hrs. Budget: G$ 60 K Scheduling Optimisation Scenario: n n n 36 Minimise Time Minimise Cost Results:

Grid and Gridbus Technologies for Various Grid (Market) Types free trading Public computing (Alchemi) Private enterprises (Libra, Gridbus, Globus) Sharing Model National provider (Globus, Gridbus, . . ) regulation scientific 37 Application Category commercial

Summary and Conclusion n Grids exploit synergies that result from cooperation of autonomous entities: n n 39 Resource sharing, dynamic provisioning, and aggregation at global level. Grid Economy provides incentive needed for sustained cooperation. Grid Network has potential to serve as Cyberinfrastructure for Utility Computing Grids offer enormous opportunities for realizing e. Science and e. Business at global level.

Any Questions ? Gridbus Project - http: //www. gridbus. org 40

Thanks for your attention! The Gridbus Cooperation! http: //www. gridbus. com 41

Backup Slides

What do Grids aim for and how to support them. n Grids aim at exploiting synergies that result from cooperation of autonomous distributed entities. Synergies include: n n n 45 Resource sharing “On-demand” Virtual Enterprises creation Aggregation of resources on demand. For this cooperation to be sustainable, participants needs to have (economic) incentive. Therefore, “incentive” mechanisms should be considered as one of key design parameters of Grid computing.

Grid Market (Participant) Types and Application Category free trading Public computing Private enterprises Sharing Model National provider regulation scientific 46 Application Category commercial

Appropriate Market Model for different market types high Variable price auction Willingn ess to Pay Commodity market Posted price oligopoly low weak 47 Demand elasticity strong

Deadline (D) and Budget (B) Constrained Scheduling Algorithms Algorithm Execution Compute Time (D) Cost (B) Grid Cost Opt Limited by D Minimize Yes Cost-Time Opt Minimize if possible Minimize Yes Time Opt Minimize Limited by B Yes Conservative -Time Opt Minimize Limited by B, jobs have guaranteed minimum budget Yes 49 Data Grid Yes