Grid Computing 1 Grid Computing Using distributed computers

Grid Computing 1

Grid Computing • Using distributed computers and resources collectively. • Usually associated with geographically distributed computers and resources on a high speed network. • Often about teams sharing resources. Grid Computing, B. Wilkinson, 2004 2

• For some people, grid computing is just cluster computing in the “large” Grid Computing, B. Wilkinson, 2004 3

Scalable Computing P E R F O R M A N C E 2100 2100 2100 Administrative Barriers + • Individual • Group • Department • Campus • State • National • Globe • Inter Planet • Universe Q o S Personal Device SMPs or Super. Computers Grid Computing, B. Wilkinson, 2004 Local Cluster Enterprise Cluster/Grid Global Grid Inter Planet Grid Figure due to Rajkumar Buyya, University of Melbourne, Australia, www. gridbus. org 4

But grid computing can be more than this. It offer the potential of virtual organizations – groups of people both geographically and organizationally distributed working together on problems, sharing computers AND other resources such as databases and experimental equipment. Grid Computing, B. Wilkinson, 2004 5

Distributed Collaborative Experiment Figure from M. Faramawi and B. Ramamurthy, SUNY- Buffalo Grid Computing, B. Wilkinson, 2004 6

Some “Computational” Grid Projects • Large Hadron Collider experimental facility for complex particle experiments at CERN (European Center for Nuclear Research, near Geneva Switzerland). • DOE Particle Physics Data grid • DOE Science grid • Astro. Grid Project • Comb-e-Chem project Grid Computing, B. Wilkinson, 2004 7

CERN grid Grid Computing, B. Wilkinson, 2004 8

Key aspects • Using distributed computers and resources collectively. • Often crossing organizational boundaries • Fueled by the Internet providing communication network. Grid Computing, B. Wilkinson, 2004 9

Background • Emergence and immense success of the Internet and the world-wide web, with agreed upon Internet standards for communication and access. • Continual improvement on computer and network technology and speeds. Grid Computing, B. Wilkinson, 2004 10

History of the Internet • Internet grew from academic research projects to interconnect of high performance computers • Started in late 1960’s with the US Defense Department Advanced Research project ARPANET. • During 1980’s, National Science Foundation expanded ARPANET into NSFNET. • In 1990’s, privatized and expanded into Internet. Grid Computing, B. Wilkinson, 2004 11

Need to harness computers • Original driving force behind Internet same as grid computing! – the need for high performance computing by connecting computers at distributed sites. Grid Computing, B. Wilkinson, 2004 12

Grid computing networks • Numerous very high performance computing projects developed in late 1990’s and 2000’s. • Examples: USA Tera. Grid (next slide), UK e-Science Grid, etc. Grid Computing, B. Wilkinson, 2004 13

Tera. Grid Computing, B. Wilkinson, 2004 14

Tera. Grid Computing, B. Wilkinson, 2004 15

UK e-Science Grid Computing, B. Wilkinson, 2004 16

EU grid Grid Computing, B. Wilkinson, 2004 17

Computational Grid Applications • • Biomedical research Industrial research Engineering research Studies in Physics and Chemistry Grid Computing, B. Wilkinson, 2004 18

Key aspects of these grids • State-of-the-art interconnection networks. • Sharing resources. • Community of scientists. Grid Computing, B. Wilkinson, 2004 19

Shared Resources Can be much more than just computers: • Storage • Sensors for experiments at particular sites in the grid • Application Software • Databases, . . . Grid Computing, B. Wilkinson, 2004 20

Resource sharing and collaborative computing • Grid computing is about collaborating and resource sharing as much as it is about high performance computing. • Many projects Grid Computing, B. Wilkinson, 2004 21

Some Grid Projects & Initiatives • Australia – Nimrod-G – Gridbus – Grid. Sim – Virtual Lab – DISCWorld – Grange. Net. –. . etc • Europe – UK e. Science – EU Data Grid – Cactus – Xtreme. Web –. . etc. • India – I-Grid § Japan – Ninf – Data. Farm • Korea. . . N*Grid • Singapore • USA – – – – – App. Le. S Globus Legion Sun Grid Engine NASA IPG Condor-G Jxta Net. Solve Access. Grid and many more. . . • Cycle Stealing &. com Initiatives – Distributed. net – SETI@Home, …. – Entropia, UD, SCS, …. • Public Forums – – – Global Grid Forum Australian Grid Forum IEEE TFCC CCGrid conference P 2 P conference NGP Figures due to Rajkumar Buyya, University of Melbourne, Grid Computing, B. Wilkinson, Australia, www. gridbus. org http: //www. gridcomputing. com 2004 22

Some American Grid Projects Initiative Focus and Technologies Developed Globus This project is developing basic software infrastructure for computations that integrate geographically distributed computational and information resources – www. globus. org Legion is an object-based metasystem. Legion supports transparent scheduling, data management, fault tolerance, site autonomy, and a wide range of security options – www. legion. virginia. edu Javelin: Internet-based parallel computing using Java – www. cs. ucsb. edu/research/javelin/ App. Les This is an application-specific approach to scheduling individual parallel applications on production heterogeneous systems – apples. ucsd. edu NASA IPG The Information Power Grid is a testbed that provides access to a Grid – a widely distributed network of high performance computers, stored data, instruments, and collaboration environments – www. ipg. nasa. gov Condor This project aims is to develop, deploy, and evaluate mechanisms and policies that support high throughput computing (HTC) on large collections of distributed computing resources – www. cs. wisc. edu/condor/ Harness builds on the concept of the virtual machine and explores dynamic capabilities beyond what PVM can supply. It focused on developing three key capabilities: Parallel plug-ins, Peer-topeer distributed control, and multiple virtual machines – www. epm. ornl. org/harness Net. Solve is a project that aims to bring together disparate computational resources connected by computer networks. It is a RPC based client/agent/server system that allows one to remotely access both hardware and software components – www. cs. utk. edu/netsolve/ Grid Port SDSCs Grid Port Toolkit generalises the Hot. Page infrastructure to develop a reusable portal toolkit –gridport. npaci. edu/ Figures due to Rajkumar Buyya, University of Melbourne, Gateway offers a programming paradigm implemented Grid Computing, B. Wilkinson, 2004 Australia, www. gridbus. org - www. npac. syr. edu/users/haupt/Web. Flow/demo. html Gateway over a virtual Web of accessible resources 23

Figures due to Rajkumar Buyya, University of Melbourne, Australia, www. gridbus. org Some European Grid Projects Initiative Focus and Technologies Developed UNICORE The UNiform Interface to Computer Resources aims to deliver software that allows users to submit jobs to remote high performance computing resources – www. fz-juelich. de/unicore MOL Metacomputer On. Line is a toolbox for the coordinated use of WAN/LAN connected systems. MOL aims at utilizing multiple WAN-connected high performance systems for solving large-scale problems that are intractable on a single supercomputer – www. uni-paderborn. de/pc 2/projects/mol e. Science The use of Grid for constructing Science applications– www. nesc. ac. uk Globe is a research project aiming to study and implement a powerful unifying paradigm for the construction of large-scale wide area distributed systems: distributed shared objects – www. cs. vu. nl/~steen/globe Pozan Poznan Centre works on development of tools and methods for metacomputing www. man. poznan. pl/metacomputing/ Date Grid This project aims to develop middleware and tools necessary for the data-intensive applications of high -energy physics –. www. eu-datagrid. org Meta. MPI supports the coupling of heterogeneous MPI systems, thus allowing parallel applications developed using MPI to be run on Grids without alteration – www. lfbs. rwthaachen. de/~martin/Meta. MPICH/ DAS This is a wide-area distributed cluster, used for research on parallel and distributed computing by five Dutch universities – www. cs. vu. nl/das Ja. WS is an economy-based computing model where both resource owners and programs using these resources place bids to a central marketplace that generates leases of use – roadrunner. ics. forth. gr Grid Computing, B. Wilkinson, 2004 24

Some Japanese Grid Projects Initiative Focus and Technologies Developed Ninf allows users to access computational resources including hardware, software and scientific data distributed across a wide area network with an easy-to-use interface – ninf. etl. go. jp Bricks is a performance evaluation system that allows analysis and comparison of various scheduling schemes on a typical high-performance global computing setting – matsuwww. is. titech. ac. jp/~takefusa/bricks Figures due to Rajkumar Buyya, University of Melbourne, Australia, www. gridbus. org Grid Computing, B. Wilkinson, 2004 25

Some Australian Grid Projects Initiative DISCWorld Focus and Technologies Developed An infrastructure for service-based metacomputing across LAN and WAN clusters. It allows remote users to login to this environment over the Web and request access to data and operations on the available data – dhpc. adelaide. edu. au/Projects/DISCWorld/ Nimrod-G A resource broker for parametric computing on computational grids. It supports computational economy paradigm for grid computing and deadline and budget constraints based scheduling. www. csse. monash. edu. au/~davida/nimrod/ Gridbus A toolkit for service-oriented computing. It provides services for (a) management of resources based on distributed computational economy at co-operative and competitive levels and (b) deployment of compute and data Grid applications on them. www. gridbus. org Figures due to Rajkumar Buyya, University of Melbourne, Grid Computing, B. Wilkinson, Australia, www. gridbus. org 2004 26

Evolution of grid computing • Started as a form of distributed computing. • Early distributed computing systems: – 1980’s - Remote Procedure calls (RPC) client server model with a service registry. – Later - Distributed objects systems: • CORBA (Common Request Broker Architecture • Java RMI (Remote Method Invocation) Grid Computing, B. Wilkinson, 2004 27

Grid computing • With the use of the Internet interconnection technology, implementation now based upon Internet technologies. • Now uses a form of web services. • Enables using existing protocols, security mechanisms, etc. Grid Computing, B. Wilkinson, 2004 28

Applications • Original e-Science applications – Computational intensive, not necessarily one big problem but a problem that has to be solved repeatedly. – Data intensive. – Experimental collaborative projects • e-Business applications to improve business models and practices. Grid Computing, B. Wilkinson, 2004 29

Security Grid Challenges and Technologies Computational Economy Uniform Access Data locality Resource Discovery Figures due to Rajkumar Buyya, University of Melbourne, Grid Computing, B. Wilkinson, Australia, www. gridbus. org System Management Resource Allocation & Scheduling Application Construction 2004 Network Management 30