Grid Computing Ian Foster Mathematics and Computer Science

Grid Computing Ian Foster Mathematics and Computer Science Division Argonne National Laboratory and Department of Computer Science The University of Chicago http: //www. mcs. anl. gov/~foster

Partial Acknowledgements l l 2 Open Grid Services Architecture design u Carl Kesselman, Karl Czajkowski @ USC/ISI u Steve Tuecke @ANL u Jeff Nick, Steve Graham, Jeff Frey @ IBM Grid services collaborators at ANL u Kate Keahey, Gregor von Laszewski u Thomas Sandholm, Jarek Gawor, John Bresnahan l Globus Toolkit R&D also involves many fine scientists & engineers at ANL, USC/ISI, and elsewhere (see www. globus. org) l Strong links with many EU, UK, US Grid projects l Support from foster@mcs. anl. gov DOE, NASA, NSF, IBM, Microsoft ARGONNE ö CHICAGO

3 Goals l Communicate the purpose, significance, state, adoption, & future of Grid technology l Persuade you that Grid technology represents an opportunity u u Grids aren’t (particularly) about science or servers—themes of virtualization, service discovery, service management, and Qo. S delivery are universal Rapid uptake in industry & science represents an exceptional opportunity for impact foster@mcs. anl. gov ARGONNE ö CHICAGO

4 Overview l l Origins: Resource sharing within scientific collaborations u Science drivers & science Grid projects u Globus Toolkit Evolution: Resource virtualization u Commercial drivers u OGSA: Grid meets Web services foster@mcs. anl. gov ARGONNE ö CHICAGO

5 Overview l l Origins: Resource sharing within scientific collaborations u Science drivers & science Grid projects u Globus Toolkit Evolution: Resource virtualization u Commercial drivers u OGSA: Grid meets Web services foster@mcs. anl. gov ARGONNE ö CHICAGO

6 E-Science: The Original Grid Driver l Pre-electronic science u l Theorize &/or experiment, in small teams Post-electronic science u Construct and mine very large databases u Develop computer simulations & analyses u Access specialized devices remotely u Þ Exchange information within distributed multidisciplinary teams Need to manage dynamic, distributed infrastructures, services, and applications foster@mcs. anl. gov ARGONNE ö CHICAGO

e. Science Application: Sloan Digital Sky Survey Analysis 7 Size distribution of galaxy clusters? Galaxy cluster size distribution Chimera Virtual Data System + Gri. Phy. N Virtual Data Toolkit + i. VDGL Data Grid (many CPUs) foster@mcs. anl. gov www. griphyn. org/chimera ARGONNE ö CHICAGO

NASA’s Information Power Grid: Aviation Safety 8 Wing Models • Lift Capabilities • Drag Capabilities • Responsiveness Airframe Models Stabilizer Models • Deflection capabilities • Responsiveness Crew Capabilities - accuracy - perception - stamina - re-action times - SOPs Engine Models Human Models • Braking performance • Steering capabilities • Traction • Dampening capabilities Landing Gear Models foster@mcs. anl. gov • Thrust performance • Reverse Thrust performance • Responsiveness • Fuel Consumption ARGONNE ö CHICAGO

Life Sciences: Telemicroscopy DATA ACQUISITION PROCESSING, ANALYSIS 9 ADVANCED VISUALIZATION NETWORK IMAGING INSTRUMENTS COMPUTATIONAL RESOURCES LARGE DATABASES foster@mcs. anl. gov ARGONNE ö CHICAGO

10 And Thus: The Grid “Resource sharing & coordinated problem solving in dynamic, multiinstitutional virtual organizations” foster@mcs. anl. gov ARGONNE ö CHICAGO

11 Underlying Technical Requirements l Dynamic formation and management of virtual organizations l Online negotiation of access to services: who, what, why, when, how l Configuration of applications and systems able to deliver multiple qualities of service l Autonomic management of distributed infrastructures, services, and applications foster@mcs. anl. gov ARGONNE ö CHICAGO

12 The Grid World: Current Status l Dozens of major Grid projects in scientific & technical computing/research & education l Open source Globus Toolkit™ a de facto standard for major protocols & services u l Simple protocols & APIs for authentication, discovery, access, etc. : infrastructure u Information-centric design u Large user and developer base u Multiple commercial support providers u Enabler of numerous tools and applications Global Grid Forum: community & standards foster@mcs. anl. gov ARGONNE ö CHICAGO

13 Overview l l Origins: Resource sharing within scientific collaborations u Science drivers & science Grid projects u Globus Toolkit Evolution: Resource virtualization u Commercial drivers u OGSA: Grid meets Web services foster@mcs. anl. gov ARGONNE ö CHICAGO

Resource Sharing within “VOs” is Not Unique to Science! l Fragmentation of enterprise infrastructure u u l 14 Driven by cheap servers, fast nets, ubiquitous Internet, e. Business workloads Need to configure distributed collections of services to deliver specified Qo. S Virtualization u u Emerging service infrastructure, utility computing models, economies of scale Services dynamically instantiated across device spectrum l B 2 B, B 2 C, foster@mcs. anl. gov C 2 C interactions ARGONNE ö CHICAGO

Virtualization and Distributed Service Management Less capable, integrated Less connected User service locus Device Continuum Distributed service management Dynamic, secure service discovery & composition foster@mcs. anl. gov 15 Larger, more integrated More connected Dynamically provisioned Resource & service aggregation Delivery of virtualized services with Qo. S guarantees ARGONNE ö CHICAGO

Realizing the Promise Requires Significant Innovation l Automation of infrastructure operation to achieve economies of scale l Management and component models for service discovery, composition, provisioning l New applications and tools powered by distributed services and resources l Business and service models to support specialization of function foster@mcs. anl. gov 16 ARGONNE ö CHICAGO

Grid Evolution: Open Grid Services Architecture l Refactor Globus protocol suite to enable common base and expose key capabilities l Service orientation to virtualize resources and unify resources/services/information l Embrace key Web services technologies: WSDL as IDL, leverage commercial efforts l Result: standard interfaces & behaviors for distributed system management foster@mcs. anl. gov 17 ARGONNE ö CHICAGO

18 OGSA System Structure l A standard substrate: the Grid service u u l The “Grid Service Specification” … supports standard service specifications u u l Standard interfaces and behaviors that address key distributed system issues Resource management, databases, workflow, security, diagnostics, etc. Target of current & planned GGF efforts … and arbitrary application-specific services based on these & other definitions foster@mcs. anl. gov ARGONNE ö CHICAGO

19 Transient Service Instances l “Web services” address discovery & invocation of persistent services u l In Grids, must also support transient service instances, created/destroyed dynamically u u l Interface to persistent state of entire enterprise Interfaces to the states of distributed activities E. g. workflow, video conferencing, distributed data analysis, workload management Significant implications for how services are named, discovered, managed, and used foster@mcs. anl. gov ARGONNE ö CHICAGO

20 OGSI, OGSA, and Web Services l OGSI (I = Infrastructure) u Small extensions to WSDL l u Conventions for naming service instances l u Handles and references port. Types for common behavior l l Nested service. Type & service. Data. Description Instance creation, lifetime management, introspection and monitoring, registration, notification, … OGSA (A = Architecture) built on OGSI u A collection of Grid service interfaces l Resource description & provisioning Higher-level services: messaging services, logging, etc. foster@mcs. anl. gov ARGONNE ö CHICAGO l

The Grid Service = Interfaces/Behaviors + Service Data Service data access Explicit destruction Soft-state lifetime Binding properties: - Reliable invocation - Authentication Grid. Service (required) Service data element … other interfaces … (optional) Service data element 21 Standard: - Notification - Authorization - Service creation - Service registry - Manageability - Concurrency Service data element Implementation + applicationspecific interfaces Hosting environment/runtime (“C”, J 2 EE, . NET, …) foster@mcs. anl. gov ARGONNE ö CHICAGO

22 Service Data l A Grid service instance maintains a set of service data elements u u u l Described in WSDL extension XML element encapsulated in standard container: name, type, lifetime, etc. Includes basic introspection information, interface-specific data, and application state Pull and push models for information query u Grid. Service: : Find. Service. Data operation l u Pull: queries this information via extensible query language Notification. Source: : Subscribe. Service. Data l Push: Subscribe to notification of changes to information foster@mcs. anl. gov ARGONNE ö CHICAGO

23 Notification Interfaces l Notification. Source for client subscription u u Subscription expression describes which service data element changes are of interest Creates a subscription manager service l Manages the lifetime and properties of subscription l Notification. Sink for asynchronous delivery of notification messages l Simple, flexible base with wide variety of uses u Dynamic discovery/registry services, monitoring, application error notification, etc. u Intermediaries: filter, aggregate, archive, et. c u Can integrate commercial messaging services foster@mcs. anl. gov ARGONNE ö CHICAGO

Grid Service Example: Database Service l l A DBaccess Grid service will support at Grid least two port. Types u Grid. Service u DBaccess Service Each has service data u u l 24 DBaccess Name, lifetime, etc. DB info Grid. Service: basic introspection information, lifetime, … DBaccess: database type, query languages supported, current load, …, … Maybe other port. Types as well u E. g. , Notification. Source foster@mcs. anl. gov ARGONNE ö CHICAGO

25 Lifetime Management l GS instances created by factory or manually; destroyed explicitly or via soft state u l l Negotiation of initial lifetime with a factory (=service supporting Factory interface) Grid. Service interface supports u Destroy operation for explicit destruction u Set. Termination. Time operation for keepalive Soft state lifetime management avoids u Explicit client teardown of complex state u Resource “leaks” in hosting environments foster@mcs. anl. gov ARGONNE ö CHICAGO

26 Factory l Factory interface’s Create. Service operation creates a new Grid service instance u Reliable creation (once-and-only-once) l Create. Service operation can be extended to accept service-specific creation parameters l Returns a Grid Service Handle (GSH) l u A globally unique URL, resolves to GSR u Uniquely identifies the instance for all time u Based on name of a handle resolver Or Grid Service Reference (GSR) foster@mcs. anl. gov ARGONNE ö CHICAGO

Example: Transient Database Services “What services can you create? ” “Create a database service” Grid Service “What database services exist? ” 28 DBaccess Factory Grid Service DBaccess Instance name, etc. Name, lifetime, etc. Factory info DB info Grid Service Registry Grid Service DBaccess Instance name, etc. Name, lifetime, etc. Registry info DB info foster@mcs. anl. gov ARGONNE ö CHICAGO

29 OGSA Design & Implementation l OGSI (I=Infrastructure) WG in GGF defining core Grid service specification u l Globus Toolkit => GT 3 (alpha end 2002) u GT 3 Core: Grid service specification u GT 3 Base: Globus Toolkit behaviors u l (At least) three implementation efforts CIM resource model, GRAM-2 SLA negotiation, database services, … Other GGF WGs address OGSA security, OGSA-compliant database services, etc. foster@mcs. anl. gov ARGONNE ö CHICAGO

30 Recap: Goals l Communicate the purpose, significance, state, adoption, & future of Grid technology l Persuade you that Grid technology represents a significant opportunity u u Grids aren’t only (or particularly) about science and servers—themes of virtualization, service discovery, service management, and Qo. S delivery are universal Rapid uptake in industry & science represents an exceptional opportunity for impact foster@mcs. anl. gov ARGONNE ö CHICAGO

31 For More Information l The Globus Project™ u l Context & research articles u l www. mcs. anl. gov/~foster Open Grid Services Architecture u l www. globus. org/ogsa Global Grid Forum www. gridforum. org u Edinburgh, July 22 -24 u Chicago, Oct 15 -17 u foster@mcs. anl. gov ARGONNE ö CHICAGO

32 OGSA Implementation 1) OGSA builds on infrastructure u u Plumbing: WSDL, WS-Security, WS-Routing/Referral, reliable messaging, transactions, etc. Hosting environments 2) to enable virtualization via u Service description u Service provisioning 3) Standard container avoids implementing OGSI features in every service instance foster@mcs. anl. gov OGSI/OGSA Interfaces service description, service provisioning, … Standard OGSI container Web services various Hosting Environment Resource virtualization and Qo. S support ARGONNE ö CHICAGO

33 Building an OGSI Container l Service data mgmt, query, subscription u Container should provide simple interface for interacting with an instance’s implementation to get and manage dynamic service data l u Container should handle query processing l u Service instance = CLR object. NET support for XPath & Xquery allows for rich functionality Container manages notification subscriptions, and drives asynchronous notification messages l Soft-state lifetime management l Soft-state registration foster@mcs. anl. gov ARGONNE ö CHICAGO