Grid Computing and the Open Grid Service Architecture

Grid Computing and the Open Grid Service Architecture Ian Foster Argonne National Laboratory University of Chicago http: //www. mcs. anl. gov/~foster October 2, 2002 Grid, Globus Toolkit, and OGSA 2 nd IEEE Intl Symp. on Network Computing & Applications, Boston, April 17, 2003

Partial Acknowledgements l 3 Open Grid Services Architecture design – Carl Kesselman, Karl Czajkowski @ USC/ISI – Steve Tuecke @ANL – Jeff Nick, Steve Graham, Jeff Frey @ IBM l Grid services collaborators at ANL – Kate Keahey, Gregor von Laszewski – Thomas Sandholm, Jarek Gawor, John Bresnahan l Globus Toolkit R&D also involves many fine scientists & engineers at ANL, USC/ISI, and elsewhere (see www. globus. org) l Strong links with many EU, UK, US Grid projects l Support from DOE, NASA, NSF, IBM, Microsoft foster@mcs. anl. gov ARGONNE ö CHICAGO

4 Overview l Grid: why and what l Evolution of Grid technology – Open Grid Services Architecture l Future directions – Towards lightweight VOs: dynamic trust relationships – Towards global knowledge communities: virtual data and dynamic workspaces foster@mcs. anl. gov ARGONNE ö CHICAGO

Why the Grid? (1) Revolution in Science l 5 Pre-Internet – Theorize &/or experiment, alone or in small teams; publish paper l Post-Internet – Construct and mine large databases of observational or simulation data – Develop simulations & analyses – Access specialized devices remotely – Exchange information within distributed multidisciplinary teams foster@mcs. anl. gov ARGONNE ö CHICAGO

Why the Grid? (2) Revolution in Business l 6 Pre-Internet – Central data processing facility l Post-Internet – Enterprise computing is highly distributed, heterogeneous, inter-enterprise (B 2 B) – Business processes increasingly computing- & data-rich – Outsourcing becomes feasible => service providers of various sorts foster@mcs. anl. gov ARGONNE ö CHICAGO

New Opportunities Demand New Technology 7 “Resource sharing & coordinated problem solving in dynamic, multiinstitutional virtual organizations” “When the network is as fast as the computer's internal links, the machine disintegrates across the net into a set of special purpose appliances” foster@mcs. anl. gov (George Gilder) ARGONNE ö CHICAGO

8 Grid Communities & Technologies l Yesterday – Small, static communities, primarily in science – Focus on sharing of computing resources – Globus Toolkit as technology base l Today – Larger communities in science; early industry – Focused on sharing of data and computing – Open Grid Services Architecture emerging l Tomorrow – Large, dynamic, diverse communities that share a wide variety of services, resources, data – New issues: Trust, distributed RM, knowledge foster@mcs. anl. gov ARGONNE ö CHICAGO

9 NSF Tera. Grid l NCSA, SDSC, Argonne, Caltech l Unprecedented capability – 13. 6 trillion flop/s – 600 terabytes of data – 40 gigabits per second – Accessible to thousands of scientists working on advanced research l www. teragrid. org foster@mcs. anl. gov ARGONNE ö CHICAGO

10 foster@mcs. anl. gov ARGONNE ö CHICAGO

Data Grids for High Energy Physics l l 11 Enable international community of 1000 s to access & analyze petabytes of data Harness computing & storage worldwide Virtual data concepts: manage programs, data, workflow Distributed system management foster@mcs. anl. gov ARGONNE ö CHICAGO

NEESgrid Earthquake Engineering Collaboratory 12 U. Nevada Reno www. neesgrid. org foster@mcs. anl. gov ARGONNE ö CHICAGO

Grid Computing 13 Grid Computing By M. Mitchell Waldrop May 2002 Hook enough computers together and what do you get? A new kind of utility that offers supercomputer processing on tap. Is Internet history about to repeat itself? foster@mcs. anl. gov ARGONNE ö CHICAGO

Industrial Perspective on Grids: A Wide Range of Applications 14 Grid Services Market Opportunity 2005 Unique by Industry with Common Characteristics Manufacturing Financial Services Energy Derivatives Analysis Seismic Analysis Statistical Analysis Reservoir Analysis Portfolio Risk Analysis Mechanical/ Electronic Design LS / Bioinformatics Other Entertainment Process Simulation Cancer Research Finite Element Analysis Drug Discovery Digital Rendering Protein Folding Massive Multi-Player Games Failure Analysis Protein Sequencing Streaming Media Web Applications Weather Analysis Code Breaking/ Simulation Academic “Gridified” Infrastructure Sources: IDC, 2000 and Bear Stearns- Internet 3. 0 - 5/01 Analysis by SAI foster@mcs. anl. gov ARGONNE ö CHICAGO

15 Overview l Grid: why and what l Evolution of Grid technology – Open Grid Services Architecture l Future directions – Towards lightweight VOs: dynamic trust relationships – Towards global knowledge communities: virtual data and dynamic workspaces foster@mcs. anl. gov ARGONNE ö CHICAGO

16 Open Grid Services Architecture l Service-oriented architecture – Key to virtualization, discovery, composition, local-remote transparency l Leverage industry standards – Internet, Web services l Distributed service management – A “component model for Web services” l A framework for the definition of composable, interoperable services “The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration”, Foster, Kesselman, ARGONNE Nick, Tuecke, 2002 foster@mcs. anl. gov ö CHICAGO

17 Web Services l l l XML-based distributed computing technology Web service = a server process that exposes typed ports to the network Described by the Web Services Description Language, an XML document that contains – Type of message(s) the service understands & types of responses & exceptions it returns – “Methods” bound together as “port types” – Port types bound to protocols as “ports” l A WSDL document completely defines a service and how to access it foster@mcs. anl. gov ARGONNE ö CHICAGO

18 OGSA Structure l A standard substrate: the Grid service – Standard interfaces and behaviors that address key distributed system issues – A refactoring and extension of the Globus Toolkit protocol suite l … supports standard service specifications – Resource management, databases, workflow, security, diagnostics, etc. – Target of current & planned GGF efforts l … and arbitrary application-specific services based on these & other definitions foster@mcs. anl. gov ARGONNE ö CHICAGO

19 Open Grid Services Infrastructure Client Introspection: • What port types? • What policy? • What state? Grid Service Handle handle resolution Grid Service Reference Lifetime management • Explicit destruction • Soft-state lifetime Grid. Service (required) Data access Service data element Other standard interfaces: factory, notification, collections Service data element Implementation Hosting environment/runtime (“C”, J 2 EE, . NET, …) foster@mcs. anl. gov ARGONNE ö CHICAGO

20 Open Grid Services Infrastructure GWD-R (draft-ggf-ogsi- gridservice-23) Open Grid Services Infrastructure (OGSI) http: //www. ggf. org/ogsi-wg Editors: S. Tuecke, ANL K. Czajkowski, USC/ISI I. Foster, ANL J. Frey, IBM S. Graham, IBM C. Kesselman, USC/ISI D. Snelling, Fujitsu Labs P. Vanderbilt, NASA February 17, 2003 Open Grid Services Infrastructure (OGSI) foster@mcs. anl. gov ARGONNE ö CHICAGO

Example: Reliable File Transfer Service Client 21 Client Request and manage file transfer operations Notf’n Policy File Grid Service Transfer Source Fault Monitor Perf. Monitor Query &/or subscribe to service data Pending Performance Policy Faults interfaces service data elements Internal State Data transfer operations foster@mcs. anl. gov ARGONNE ö CHICAGO

Open Grid Service Architecture: Next Steps 22 ü Technical specifications – Open Grid Services Infrastructure is complete – Security, data access, Java binding, common resource models, etc. , in the pipeline ü Implementations and compliant products – Here: OGSA-based Globus Toolkit v 3, … – Announced: IBM, Avaki, Platform, Sun, NEC, HP, Oracle, UD, Entropia, Insors, …, … 6 Rich set of service defns & implementations foster@mcs. anl. gov ARGONNE ö CHICAGO

Globus Toolkit v 3 (GT 3) Open Source OGSA Technology l Implements OGSI interfaces l Supports primary GT 2 interfaces 23 – High degree of backward compatibility l Multiple platforms & hosting environments – J 2 EE, Java, C, . NET, Python l New services – SLA negotiation, service registry, community authorization, data management, … l Rapidly growing adoption and contributions: “Linux for the Grid” foster@mcs. anl. gov ARGONNE ö CHICAGO

24 Overview l Grid: why and what l Evolution of Grid technology – Open Grid Services Architecture l Future directions – Towards lightweight VOs: dynamic trust relationships – Towards global knowledge communities: virtual data and dynamic workspaces foster@mcs. anl. gov ARGONNE ö CHICAGO

25 Future Directions l Grids are about computers, certainly – “On-demand” access to computing, etc. – Challenging future issues here: e. g. , scale foster@mcs. anl. gov ARGONNE ö CHICAGO

26 CMS Event Simulation Production l Production Run on the Integration Testbed – Simulate 1. 5 million full CMS events for physics studies: ~500 sec per event on 850 MHz processor – 2 months continuous running across 5 testbed sites – Managed by a single person at the US-CMS Tier 1 foster@mcs. anl. gov ARGONNE ö CHICAGO

27 CMS Event Simulation Production s! l t s s Production Run on the Integration t. Testbed i n c e for – Simulate 1. 5 million full CMS events physics i v s. MHz processor studies: ~500 sec per event E on 850 y ) h n s P 5 testbed o – 2 months continuous running across r sites i a S l e Tier 1 M – Managed by a singlel person at the US-CMS y C i M o U P t 5 C. 1 ed 0 3 r e y v l i r l a e De n ( foster@mcs. anl. gov ARGONNE ö CHICAGO

28 Future Directions l Grids are about computers, certainly – “On-demand” access to computing, etc. – Challenging future issues here: e. g. , scale l But they are ultimately about people, their activities, and their interactions – New interaction modalities supported by ondemand formation of lightweight VOs – New technologies needed: e. g. , trust, security, data and knowledge integration l Convergence of interest between “Compute” and “Collaboration” Grids? foster@mcs. anl. gov ARGONNE ö CHICAGO

Global Knowledge Communities foster@mcs. anl. gov 29 ARGONNE ö CHICAGO

30 Example Issue: Trust and Security l Effective VO operation depends critically on – Trust: can I rely on you? – Protection mechanisms to govern actions l l Suffers from VO-organization policy mismatch Goal: collaborations no longer defined by slow centralized mechanisms but can – form spontaneously; – be managed in a distributed manner; and – be protected by an infrastructure that maintains and enforces trust relationships foster@mcs. anl. gov ARGONNE ö CHICAGO

31 Grid Security Services Requestor's Domain Trust Service Attribute Service Audit/ Secure-Logging Service Provider's Domain Authorization Service Privacy Service Trust Service Attribute Service Audit/ Secure-Logging Service Privacy Service Credential Validation Service Bridge/ Translation Service Requestor Application WS-Stub Secure Conversation WS-Stub Credential Validation Service Provider Application Credential Validation Service Authorization Service Attribute Service Trust Service VO Domain foster@mcs. anl. gov ARGONNE ö CHICAGO

Understanding and Enhancing VO Trust and Security Usability analysis Community Social network analysis Other analyses Monitoring for reputation, compliance, intrusion detection, etc. trust ( Tr, Te, As, L ) <- Cs; recommend ( Rr, Re, As, L ) <- Cs; Trust Establishment, enhancement, maintenance, verification Workflow analysis Risk analysis allowed (S, O, A, C) Policy Factoring wrt environment Mechanism foster@mcs. anl. gov 32 VOTA, PKI, VPN, etc. Feasibility analysis wrt cost, legality, etc. ARGONNE ö CHICAGO

Virtual Data for Collaborative Science l 33 Much collaboration is concerned with the development & use of knowledge, whether – Programs for data analysis and generation – Computations involving those programs – Metadata concerning data, programs, computations—and their interrelationships l In a distributed, heterogeneous, fractal (? ) environment with widely varying – Data and analysis program formats – Degrees of formality and scale – Scientific goals and sharing policies foster@mcs. anl. gov ARGONNE ö CHICAGO

Sloan Digital Sky Survey Production System foster@mcs. anl. gov 34 ARGONNE ö CHICAGO

Virtual Data Concept l 35 Capture and manage information about relationships among – Data (of widely varying representations) – Programs (& their execution needs) – Computations (& execution environments) l Apply this information to, e. g. – Discovery: Data and program discovery – Workflow: Structured paradigm for organizing, locating, specifying, & requesting data – Explanation: provenance – Planning and scheduling – Other uses we haven’t thought of foster@mcs. anl. gov ARGONNE ö CHICAGO

36 “I’ve come across some interesting data, but I need to understand the nature of the corrections applied when it was constructed before I can trust it for my purposes. ” Motivations Data created-by Transformation execution-of “I want to search an astronomical database for galaxies with certain characteristics. If a program that performs this analysis exists, I won’t have to write one from foster@mcs. anl. gov scratch. ” “I’ve detected a calibration error in an instrument and want to know which derived data to recompute. ” consumed-by/ generated-by Derivation “I want to apply an astronomical analysis program to millions of objects. If the results already exist, I’ll save weeks. ARGONNE of computation. ” ö CHICAGO

Example: Sloan Galaxy Cluster Analysis 37 DAG Sloan Data Galaxy cluster size distribution foster@mcs. anl. gov Jim Annis, Steve Kent, Vijay Sehkri, Fermilab; Michael ARGONNE ö CHICAGO Milligan, Yong Zhao, Chicago

38 Integrating Provenance Data foster@mcs. anl. gov ARGONNE ö CHICAGO

39 Summary l Yesterday – Small, static communities, primarily in science – Focus on sharing of computing resources – Globus Toolkit as technology base l Today – Larger communities in science; early industry – Focused on sharing of data and computing – Open Grid Services Architecture emerging l Tomorrow – Large, dynamic, diverse communities that share a wide variety of services, resources, data – New issues: Trust, distributed RM, knowledge foster@mcs. anl. gov ARGONNE ö CHICAGO

40 For More Information l The Globus Project™ – www. globus. org l Technical articles – www. mcs. anl. gov/~foster l Open Grid Services Arch. – www. globus. org/ogsa l Chimera – www. griphyn. org/chimera l Global Grid Forum – www. gridforum. org foster@mcs. anl. gov ARGONNE ö CHICAGO