Open Grid Services Architecture A Tutorial Ian Foster
- Slides: 138
Open Grid Services Architecture: A Tutorial Ian Foster Argonne National Laboratory University of Chicago Globus Alliance www. mcs. anl. gov/~foster
Thanks l For slides and/or comments: Malcolm Atkinson, David Foster, Carl Kesselman, Lisa Childers, Lee Liming, Steve Tuecke Open Grid Services Architecture 2
Overview l Grid background l Open Grid Services Architecture l Open Grid Services Infrastructure l Beyond OGSI: other OGSA services l Globus Toolkit v 3 implementation l Early GT 3 performance measurements l Scientific and commercial perspectives l Summary Open Grid Services Architecture 3
The Grid l Infrastructure (“middleware” & “services”) for establishing, managing, and evolving multi-organizational federations u u l Dynamic, autonomous, domain independent On-demand, ubiquitous access to computing, data, and services Mechanisms for creating and managing workflow within such federations u u New capabilities constructed dynamically and transparently from distributed services Service-oriented, virtualization Open Grid Services Architecture 4
What is a Grid? l Three key criteria u u u l Coordinates distributed resources … using standard, open, general-purpose protocols and interfaces … to deliver non-trivial qualities of service. What is not a Grid? u u A cluster, a network attached storage device, a scientific instrument, a network, etc. Each may be an important component of a Grid, but by itself does not constitute a Grid Open Grid Services Architecture 5
e. Science is the Initial Motivator … l l New approaches to enquiry based on u Deep analysis of huge quantities of data u Interdisciplinary collaboration u Large-scale simulation u Smart instrumentation Enabled by an infrastructure that enables access to, and integration of, resources & services without regard for location … but e. Business is catching up rapidly Open Grid Services Architecture 6
Resource Integration as a Fundamental Challenge Many sources of data, services, computation Security & policy must underlie access & management decisions Discovery R RM Access Registries organize services of interest to a community RM RM Security service Data integration activities may require access to, & exploration/analysis of, data at many locations Open Grid Services Architecture RM Resource management is needed to ensure progress & arbitrate competing demands Policy service Exploration & analysis may involve complex, multi-step workflows 7
Distributed Computing l The concept of sharing distributed resources is not new u u u In 1965, MIT's Fernando Corbató and the other Multics OS designers envisioned a computer facility operating “like a power company or water company” In their 1968 article “the computer as a communications device, ” J. C. R. Licklider and Robert W. Taylor anticipated Grid-like scenarios Since the late 1960 s, much work has been devoted to developing distributed systems, with mixed success Open Grid Services Architecture 8
Overview l Grid background l Open Grid Services Architecture l Open Grid Services Infrastructure l Beyond OGSI: other OGSA services l Globus Toolkit v 3 implementation l Early GT 3 performance results l Scientific and commercial perspectives l Summary Open Grid Services Architecture 9
Why Open Standards Matter l l Ubiquitous adoption demands open, standard protocols u Standard protocols enable interoperability u Avoid product/vendor lock-in u Enables innovation/competition on end points Further aided by open, standard APIs u u l Standard APIs enable portability Allow implementations to port to different vendor platforms Internet and Web as exemplars Open Grid Services Architecture 10
Increased functionality, standardization The Emergence of Open Grid Standards Managed shared virtual systems Computer science research Open Grid Services Arch Web services, etc. Internet standards Custom solutions 1990 Real standards Multiple implementations Globus Toolkit Defacto standard Single implementation 1995 Open Grid Services Architecture 2000 2005 2010 11
Open Grid Services Architecture l Service orientation to virtualize resources u l l è Everything is a service From Web services u Standard interface definition mechanisms u Evolving set of other standards: security, etc. From Grids (Globus Toolkit) u Service semantics, reliability & security models u Lifecycle management, discovery, other services A framework for the definition & management of composable, interoperable services “The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration”, Foster, Kesselman, Nick, Tuecke, 200212 Open Grid Services Architecture
Web Services l XML-based distributed computing technology l Web service = a server process that exposes typed ports to the network l Described by the Web Services Description Language, an XML document that contains u l Type of message(s) the service understands & types of responses & exceptions it returns u “Methods” bound together as “port types” u Port types bound to protocols as “ports” A WSDL document completely defines a service and how to access it Open Grid Services Architecture 13
WSDL Example <wsdl: definitions target. Namespace=“…”> <wsdl: types> <schema> <xsd: element name=“foo. Input” …/> <xsd: element name=“foo. Output” …/> </schema> </wsdl: types> <wsdl: message name=“foo. Input. Message”> <part name=“parameters” element=“foo. Input”/> </wsdl: message> <wsdl: message name=“foo. Output. Message”> <part name=“parameters” element=“foo. Output”/> </wsdl: message> <wsdl: port. Type name=“foo. Interface”> <wsdl: operation name=“foo”> <input message=“foo. Input”/> <output message = “foo. Output”/> </wsdl: operation> </wsdl: port. Type> </wsdl: definitions> Open Grid Services Architecture 14
Transient Service Instances l “Web services” address discovery & invocation of persistent services u l l Interface to persistent state of entire enterprise In Grids, must also support transient services, created/destroyed dynamically u Interfaces to the states of distributed activities u E. g. workflow, video conf. , dist. data analysis Significant implications for how services are managed, named, discovered, and used u In fact, much of our work is concerned with the management of services Open Grid Services Architecture 15
OGSA Structure l A standard substrate: the Grid service u u l A Grid service is a Web service … supports standard service specifications u u l Standard interfaces and behaviors that address key distributed system issues: naming, service state, lifetime, notification Agreement, data access & integration, workflow, security, policy, diagnostics, etc. Target of current & planned GGF efforts … and arbitrary application-specific services based on these & other definitions Open Grid Services Architecture 16
Overview l Grid background l Open Grid Services Architecture l Open Grid Services Infrastructure l Beyond OGSI: other OGSA services l Globus Toolkit v 3 implementation l Early GT 3 performance results l Scientific and commercial perspectives l Summary Open Grid Services Architecture 17
OGSI Specification l Defines WSDL conventions and extensions u u l Working with W 3 C WSDL working group to drive OGSI extensions into WSDL 1. 2 Defines fundamental interfaces (using extended WSDL) and behaviors that define a Grid Service u l For describing and naming services A unifying framework for interoperability & establishment of total system properties http: //www. ggf. org/ogsi-wg Open Grid Services Architecture 18
Fundamental Interfaces & Behaviors l OGSI defines basic patterns of interaction, which can be combined with each other and with custom patterns in a myriad of ways l OGSI Specification focuses on: u Atomic, composable patterns in the form of port. Types/interfaces l u A model for how these are composed l l Define operations & associated service data elements Compatible with WSDL 1. 2 Complete service descriptions are left to other groups that are defining real services Open Grid Services Architecture 19
OGSI: Standard Web Services Interfaces & Behaviors l Naming and bindings (basis for virtualization) u l l l Every service instance has a unique name, from which can discover supported bindings Lifecycle (basis for fault resilient state management) u Service instances created by factories u Destroyed explicitly or via soft state Information model (basis for monitoring & discovery) u Service data (attributes) associated with GS instances u Operations for querying and setting this info u Asynchronous notification of changes to service date Service Groups (basis for registries & collective svcs) u Group membership rules & membership management l Base Fault type Open Grid Services Architecture 20
OGSI Service Data l Attributes: Publicly visible state of the service l Want to bring full power of XML to attributes u l get. XXX/set. XXX is too limiting l How to get/set multiple? l Want richer queries across attributes (e. g. , join) u Use XML Schema, XPath, XQuery, XSLT, etc. u OGSI service data: l Attributes defined using XML Schema l Attributes combined into a single (logical) document within the service l Rich pull/push/set operations against service data document Should declare attributes in WSDL interface Open Grid Services Architecture 21
Open Grid Services Infrastructure Client Introspection: • What port types? • What policy? • What state? Grid Service Handle handle resolution Lifetime management • Explicit destruction • Soft-state lifetime Grid. Service (required) Data access Service data element Grid Service Reference Other standard interfaces: factory, notification, collections Service data element Implementation Hosting environment/runtime (“C”, J 2 EE, . NET, …) Open Grid Services Architecture 22
Open Grid Services Infrastructure (OGSI) Resource allocation Authentication & authorization are applied to all requests Create Service factory Grid Service Handle Service data Keep-alives Notifications Service invocation Service instances Interactions standardized using WSDL Open Grid Services Architecture Register Service requestor (e. g. user application) Service discovery Service registry 23
GWSDL l OGSI requires interface extension/composition l We worked within W 3 C WSDL working group to define standard interface extension in WSDL 1. 2 that meets OGSI requirements l But could not wait for WSDL 1. 2 l So defined gwsdl: port. Type that extends WSDL 1. 1 port. Type with: l u WSDL 1. 2 port. Type extension u WSDL 1. 2 open content model Define GWSDL 1. 1 & 1. 2 mappings Open Grid Services Architecture 24
GWSDL Example <wsdl: definitions> <wsdl: types>…</wsdl: types> <wsdl: message>…</wsdl: message> … <gwsdl: port. Type name=“foo” extends=“ns: bar ogsi: Grid. Service”> <wsdl: operation name=“op 1”>…</wsdl: operation> <wsdl: operation name=“op 2”>…</wsdl: operation> <ogsi: service. Data … /> </gwsdl: port. Type> … </wsdl: definitions> Open Grid Services Architecture 25
Example: Reliable File Transfer Service Client Request and manage file transfer operations File Notf’n Policy Grid Service Transfer Source Fault Monitor Perf. Monitor Query &/or subscribe to service data Pending Performance Policy Faults interfaces service data elements Internal State Data transfer operations Open Grid Services Architecture 26
OGSI Implementations l Globus Toolkit version 3. 0 (Java, C client) l U Virginia OGSI. NET (. NET) l LBNL py. Globus (Python) l U Edinburgh (. NET) l U Manchester (PERL) l Fujitsu Unicore (Java) Open Grid Services Architecture 27
Overview l Grid background l Open Grid Services Architecture l Open Grid Services Infrastructure l Beyond OGSI: other OGSA services l Globus Toolkit v 3 implementation l Early GT 3 performance results l Scientific and commercial perspectives l Summary Open Grid Services Architecture 28
Open Grid Services Architecture Users in Problem Domain X Application & Integration Technology for Problem Domain X Generic Virtual Service Access and Integration Layer Job Submission Brokering Registry Banking Workflow Authorisation OGSA Structured Data Integration Data Transport Resource Usage Transformation Structured Data Access OGSI: Interface to Grid Infrastructure Web Services: Basic Functionality Compute, Data & Storage Resources Distributed Structured Data Relational XML Virtual Integration Architecture Open Grid Services Architecture Semi-structured - 29
OGSA Standardization & Implementation l OGSI defines core interfaces and behaviors for manageable services u l Supported by strong open source technology & major commercial vendors Efforts are underway within GGF, OASIS, and other bodies to define standards for u Agreement negotiation u Common management model u Data access and integration u Security and policy u Etc. , etc. Open Grid Services Architecture 30
Transactions & Contexts l WS-Coordination & WS-Transaction u l l IBM/MS (not in standards org) WS-CAF (Coordinated Application Framework) u Sun/Oracle/Arjuna/Fujitsu (not in standards org) u WS-CTX (Context) u WS-CF (Coordination Framework) u WS-TXM (Transaction Management) Both take a “contextualization” approach u Context (id) threaded through SOAP header u OGSI for context creation, naming & lifecycle? ? ? Open Grid Services Architecture 31
Security Standards l Many core security standards are from IETF u X. 509, Kerberos, etc. u X. 509 Proxy Certificates (RFC soon hopefully) l l l Used by Globus Toolkit GSI OASIS appears to be leader in Web services security standards u WS-Security: SOAP message security u SAML: signed assertions using XML u XACML: access control lists using XML GGF OGSA Security WG evaluating security specifications for applicability to OGSA Open Grid Services Architecture 32
IBM/Microsoft WS Security Architecture l Large set of specifications for doing Web services security, most of which should be appropriate for OGSA l Announced April 2002 l Initial spec in July 2002 (WS-Security) u l Submitted to OASIS New crops of specs arrive periodically u WS-Policy*, WS-Trust, WS-Federation, etc. u But… Not yet in any standards organization Open Grid Services Architecture 33
WS Security Current/Proposed WSS-specs WS-Secure Conversation WS-Federation WS-Authorization WS-Policy WS-Trust WS-Privacy WS-Security In progress SOAP Foundation proposed promised Open Grid Services Architecture 34
OASIS SAML & XACML l SAML: Security Assertion Markup Language u l XACML: e. Xtensible Access Control Markup Language u l Good for asserting properties such as group membership, etc For defining access control policies These are gaining considerable momentum, but WS-Policy* leaves them in question Open Grid Services Architecture 35
Project Liberty Alliance l l V 1. x specifications for identity federation u Allows cross-organization identification u Privacy preserving model Based on SAML Open Grid Services Architecture 36
WS-Agreement l Recall key criteria of a Grid: u u u l Coordinates resources that are not subject to centralized control … using standard, open, general-purpose protocols and interfaces … to deliver non-trivial qualities of service. Implies need to express and negotiate agreements that govern the delivery of services to clients u Agreement = what will be done, Qo. S, billing, compliance monitoring Open Grid Services Architecture 37
WS-Agreement Contents l Standard agreement language u u l A composition of a set of terms that govern a service’s behavior with respect to clients Agreement language uses WS-Policy (currently) Standard attributes for terms that express current state of negotiation Other groups define specific terms Standard agreement negotiation protocol u Establish, monitor, re-negotiate agreement u Expressed using OGSI GWSDL interfaces u Each agreement represented by a service Open Grid Services Architecture 38
WS-Agreement Applicability l All interesting Web/Grid services interactions will be governed by agreements! l WS-Agreement (language and interfaces) should be used by specifications that define domain-specific services u Data services u Job submission u Specialized services u Etc. Open Grid Services Architecture 39
Platform’s Community Scheduling Framework & WS-Agreement GT 3. 0 Queuing Service GT 3. 0 RM Adapter LSF Internet Job Service Reservation Service Site A Site B GT 3. 0 RM Adapter PBS Site C Reservation Agreement Exchange GT 3. 0 RM Adaptor Open Grid Services Architecture SGE 40
WSDM / WSMF / CMM l OASIS Web Services Distributed Management (WSDM) technical committee u u l HP submitted its Web Services Management Framework (WSMF) to WSDM in July 2003 l WS-Events: event schema, subscription, message queues l WSMF-Foundation: management using Web services l WSM: management of Web services GGF Common Management Model (CMM) WG u l Management using/of Web Services IBM submission overlaps WSMF-Foundation Working to bring WSDM & CMM together, based on OGSI foundation Open Grid Services Architecture 41
Data as Service: OGSA Data Access & Integration l l Service-oriented treatment of data appears to have significant advantages u Leverage OGSI introspection, lifetime, etc. u Compatibility with Web services Standard service interfaces being defined u Service data: e. g. , schema u Derive new data services from old (views) u Externalize to e. g. file/database format u Perform queries or other operations Open Grid Services Architecture 42
Data Services l GGF Data Access and Integration Svcs (DAIS) u u l OGSI-compliant interfaces to access relational and XML databases Needs to be generalized to encompass other data sources (see next slide…) Generalized DAIS becomes the foundation for: u Replication: Data located in multiple locations u Federation: Composition of multiple sources u Provenance: How was data generated? Open Grid Services Architecture 43
“OGSA Data Services” (Foster, Tuecke, Unger, eds. ) l l Describes conceptual model for representing all manner of data sources as Web services u Database, filesystems, devices, programs, … u Integrates WS-Agreement Data service is an OGSI-compliant Web service that implements one or more of base data interfaces: u u Data. Description, Data. Access, Data. Factory, Data. Management These would be extended and combined for specific domains (including DAIS) Open Grid Services Architecture 44
Data Access & Integration Services 1 a. Request to Registry for sources of data about “x” 1 b. Registry responds with Factory handle SOAP/HTTP Registry service creation API interactions 2 a. Request to Factory for access to database Factory Client 2 c. Factory returns handle of GDS to client 3 a. Client queries GDS with XPath, SQL, etc 3 c. Results of query returned to client as XML 2 b. Factory creates Grid. Data. Service to manage access Grid Data Service XML / Relational database 3 b. GDS interacts with database Open Grid Services Architecture Slide Courtesy Malcolm Atkinson, UK e. Science Center 45
Overview l Grid background l Open Grid Services Architecture l Open Grid Services Infrastructure l Beyond OGSI: other OGSA services l Globus Toolkit v 3 implementation l Early GT 3 performance results l Scientific and commercial perspectives l Summary Open Grid Services Architecture 46
The Globus Alliance l A group of people with a common mission: u l Argonne/U. Chicago, USC-ISI, EPCC, & KTH-PDC u u u l “Make Grid computing an everyday reality. ” Led by Ian Foster (Argonne), Carl Kesselman (ISI), Malcolm Atkinson (EPCC), Lennart Johnsson (PDC) Includes researchers, software developers, software architects & designers, systems engineers, & others Collaborations (or at least acquaintances) with most Grid activities in the world All activities contribute to our common mission u u Research Software development (prototypes, reference implns) Application consulting Infrastructure consulting Open Grid Services Architecture 47
Globus Toolkit: A Story of Evolution l Definition of Grid problem has been stable since original Globus Project proposal in 1995 u l l Though we’ve gotten better at articulating it But our approach to its solution has evolved: u From APIs and custom protocols… u to standard protocols… u to Grid services (OGSA) Driven by experience implementing and deploying the Globus Toolkit, and building real applications with it Open Grid Services Architecture 48
Globus Toolkit® v 3. 0 l All of the GT v 2. 4 services and clients l Complete Java implementation of OGSI v 1. 0 l l u Rich, container-based implementation u Built on Apache Axis Globus “proprietary” services built on OGSI u Managed Jobs (akin to GT 2 GRAM) u Reliable File Transfer (RFT) u Index Services (akin to GT 2 GIIS) Some services not yet OGSI-fied: u Grid. FTP, Replica Location Services (RLS) Open Grid Services Architecture 49
GT 3 Distribution GT 2 Components RLS GT-OGSA Grid Service Infrastructure The focus of this presentation Open Grid Services Architecture 50
GT-OGSA Grid Service Infrastructure Grid Service Container User-Defined Services Base Services System-Level Services OGSI Spec Implementation Security Infrastructure Web Service Engine Hosting Environment Open Grid Services Architecture 51
GT-OGSA Grid Service Infrastructure Grid Service Container User-Defined Services Base Services System-Level Services OGSI Spec Implementation Security Infrastructure Web Service Engine Hosting Environment Open Grid Services Architecture 52
OGSI Implementation l GT 3 includes a set of primitives that fully implement the interfaces and behaviors defined in the OGSI Specification u l Defines how entities can create, discover and interact with a Grid service The OGSI Specification defines a protocol: GT 3 provides a programming model for that protocol Open Grid Services Architecture 53
Implementation of the Grid. Service port. Type: Grid. Service. Impl and Persistent. Grid. Service. Impl l destroy() l set. Service. Data() l find. Service. Data() l request. Termination. After() l request. Termination. Before() add. Operation. Provider() l (See docs for complete set of methods) Implementation of the OGSI Spec Additional functionality can be added to a Grid Service using Operation. Providers Deployment descriptor <parameter name=“operation. Providers” value=“<class. Name>”> Open Grid Services Architecture 54
Building an OGSI-Compliant Grid Service using GT 3 Write service-specific logic that also implements the GT 3 Operation. Provider interface Open Grid Services Architecture 55
Building an OGSI-Compliant Grid Service using GT 3 Write service-specific logic that also implements the GT 3 Operation. Provider interface Combine with one of the two GT 3 implementations of base Grid. Service functionality: Grid. Service. Impl or Persistent. Grid. Service. Impl Open Grid Services Architecture 56
Building an OGSI-Compliant Grid Service using GT 3 Write service-specific logic that also implements the GT 3 Operation. Provider interface Combine with one of the two GT 3 implementations of base Grid. Service functionality: Grid. Service. Impl or Persistent. Grid. Service. Impl Open Grid Services Architecture An OGSI-Compliant grid service 57
Building an OGSI-Compliant Grid Service using GT 3 Operation. Providers are configured at deployment time or added at runtime Write service-specific logic that also implements the GT 3 Operation. Provider interface Combine with one of the two GT 3 implementations of base Grid. Service functionality: Grid. Service. Impl or Persistent. Grid. Service. Impl Open Grid Services Architecture An OGSI-Compliant grid service 58
A Grid Service Can be Composed of Multiple Operation. Providers l OPs can be designed as atomic bits of functionality to facilitate reuse l OP approach eases the task of bringing legacy code into OGSI-compliance l OPs allow Grid Services to be formed dynamically (in contrast to the inheritance approach) Open Grid Services Architecture 59
Several Operation. Providers are Included in the GT 3 Distribution Notification. Source. Provider Handle. Resolver. Provider Service. Group. Registration. Provider Factory. Provider Open Grid Services Architecture 60
OGSI Specification: Notifications l Our Notification. Source. Provider implementation allows any Grid Service to become a sender of notification messages l A subscribe request on a Notification. Source triggers the creation of a Notification. Subscription service l A Notification. Sink can receive notification msgs from Notification. Sources u l Sinks are not required to implement the Grid. Service port. Type Notifications can be set on SDEs Open Grid Services Architecture 61
OGSI Specification: Factory port. Type l Factories create services l Factories are typically persistent services l Factory is an optional OGSI interface (Grid Services can also be instantiated by other mechanisms) Open Grid Services Architecture 62
OGSI Specification: Service Groups Service group port. Types l A Service. Group is a grid service that maintains information about a group of other grid services l The classic registry model can be implemented with the Service. Group port. Types l A grid service can belong to more than one Service. Group l Members of a Service. Group can be heterogeneous or homogenous l Each entry in a service group can be represented as its own service l Service group port. Types are optional OGSI interfaces Open Grid Services Architecture 63
OGSI Specification: Handle Resolver Handle. Resolver port. Type l Defines a means for resolving a GSH (Grid Service Handle) to a GSR (Grid Service Reference) u A GSH points to a Grid Service (GT 3 uses a hostname-based GSH scheme) u A GSR specifies how to communicate with the Grid Service (GT 3 currently supports SOAP over HTTP, so GSRs are in WSDL format) Service locator Includes 0 or more Grid Service Handles (GSHs) Includes 0 or more Grid Service References (GSRs) Open Grid Services Architecture 64
A Service Creation Scenario* Registry 1. From a known registry, the client discovers a factory by querying the service data of the registry Client * All scenarios are offered as examples and are not prescriptive Open Grid Services Architecture 65
A Service Creation Scenario Registry 1. From a known registry, the client discovers a factory by querying the service data of the registry 2. The client calls the create. Service operation on the factory Factory Client Open Grid Services Architecture 66
A Service Creation Scenario Registry 1. From a known registry, the client discovers a factory by querying the service data of the registry Client Open Grid Services Architecture 2. The client calls the create. Service operation on the factory Factory 3. The factory creates a service Service 67
A Service Creation Scenario Registry 1. From a known registry, the client discovers a factory by querying the service data of the registry Client Open Grid Services Architecture 2. The client calls the create. Service operation on the factory 4. The factory returns a locator to the newly created service Factory 3. The factory creates a service Service 68
A Service Creation Scenario Registry 1. From a known registry, the client discovers a factory by querying the service data of the registry 2. The client calls the create. Service operation on the factory 4. The factory returns a locator to the newly created service Client Factory 3. The factory creates a service Service 5. The client and service interact Open Grid Services Architecture 69
A Notification Scenario Notification Source 1. Notification. Sink calls the subscribe operation on Notification. Source Notification Sink Open Grid Services Architecture 70
A Notification Scenario Notification Source 1. Notification. Sink calls the subscribe operation on Notification. Source Notification Sink Open Grid Services Architecture 2. Notification Source creates a subscription service Notification Subscription 71
A Notification Scenario Notification Source 1. Notification. Sink calls the subscribe operation on Notification. Source 3. Notification Source returns a locator to the subscription service Notification Sink Open Grid Services Architecture 2. Notification Source creates a subscription service Notification Subscription 72
A Notification Scenario 4. a deliver. Notification stream continues for the lifetime of Notification. Subscription 1. Notification. Sink calls the subscribe operation on Notification. Source 3. Notification Source returns a locator to the subscription service Notification Sink 4. b The Notification. Sink and Subscription service interact to perform lifetime management Open Grid Services Architecture Notification Source 2. Notification Source creates a subscription service Notification Subscription 73
A Notification Scenario 4. a deliver. Notification stream continues for the lifetime of Notification. Subscription The sole mandated cardinality: 1 to 1 1. Notification. Sink calls the subscribe operation on Notification. Source 3. Notification Source returns a locator to the subscription service Notification Sink 4. b The Notification. Sink and Subscription service interact to perform lifetime management Open Grid Services Architecture Notification Source 2. Notification Source creates a subscription service Notification Subscription 74
GT-OGSA Grid Service Infrastructure Grid Service Container User-Defined Services Base Services System-Level Services OGSI Spec Implementation Security Infrastructure Web Service Engine Hosting Environment Open Grid Services Architecture 75
Security Infrastructure l Transport Layer Security/Secure Socket Layer (TLS/SSL) u l SOAP Layer Security u l l To be deprecated Based on WS-Security, XML Encryption, XML Signature GT 3 uses X. 509 identity certificates for authentication It also uses X. 509 Proxy certificates to support delegation and single sign-on, updated to conform to latest IETF/GGF draft Open Grid Services Architecture 76
GT-OGSA Grid Service Infrastructure Grid Service Container User-Defined Services Base Services System-Level Services OGSI Spec Implementation Security Infrastructure Web Service Engine Hosting Environment Open Grid Services Architecture 77
System-Level Services l General-purpose services that facilitate the use of Grid services in production environments l The 3. 0 distribution includes three system-level services u An administration service u A logging service u A management service Open Grid Services Architecture 78
GT-OGSA Grid Service Infrastructure Grid Service Container User-Defined Services Base Services System-Level Services OGSI Spec Implementation Security Infrastructure Web Service Engine Hosting Environment Open Grid Services Architecture 79
Grid Service Container Includes the OGSI Implementation, security infrastructure and system-level services, plus: l Service activation, deactivation, construction, destruction, etc. l Service data element placeholders that allow you to fetch service data values dynamically, at query time l Evaluator framework (supporting By. XPath and By. Name notifications and queries) l Interceptor/callback framework (allows one to intercept certain service lifecycle events) Open Grid Services Architecture 80
Grid Service Container (cont. ) Layers in the Web Services Model Interface Layer OGSI Spec is here Transport Layer Transport/binding layer (e. g. , GT 3: SOAP over HTTP) Implementation Layer Container is here Open Grid Services Architecture 81
GT-OGSA Grid Service Infrastructure Grid Service Container User-Defined Services Base Services System-Level Services OGSI Spec Implementation Security Infrastructure Web Service Engine Hosting Environment Open Grid Services Architecture 82
Hosting Environment GT 3 currently offers support for four Java hosting environments: l Embedded l Standalone l Servlet l EJB Open Grid Services Architecture 83
Virtual Hosting Environment Framework l Virtual hosting allows Grid services to be distributed across several remote containers l Useful in implementing solutions for problems common to distributed computing u Load balancing u User account sandboxing Open Grid Services Architecture 84
GT-OGSA Grid Service Infrastructure Grid Service Container User-Defined Services Base Services System-Level Services OGSI Spec Implementation Security Infrastructure Web Service Engine Hosting Environment Open Grid Services Architecture 85
Resource Management l GRAM Architecture rendered in OGSA l The MMJFS runs as an unprivileged user, with a small highly-constrained setuid executable behind it l Individual user environments are created using virtual hosting User 1 Master User MMJFS Open Grid Services Architecture MJS MJS User 2 MJS User 3 MJS MMJFS: Master Managed Job Factory Service MJS: Managed Job Service User Hosting Environment 86
GRAM Job Submission Scenario Index Service 1. From an index service, the client chooses an MMJFS Client Open Grid Services Architecture 87
GRAM Job Submission Scenario Index Service 1. From an index service, the client chooses an MMJFS 2. The client calls the create. Service operation on the factory and supplies RSL MMJFS Client Open Grid Services Architecture 88
GRAM Job Submission Scenario Index Service 1. From an index service, the client chooses an MMJFS 2. The client calls the create. Service operation on the factory and supplies RSL Client Open Grid Services Architecture MMJFS 3. The factory creates a Managed Job Service MJS 89
GRAM Job Submission Scenario Index Service 1. From an index service, the client chooses an MMJFS 2. The client calls the create. Service operation on the factory and supplies RSL Client Open Grid Services Architecture 4. The factory returns a locator MMJFS 3. The factory creates a Managed Job Service MJS 90
GRAM Job Submission Scenario Index Service 1. From an index service, the client chooses an MMJFS 2. The client calls the create. Service operation on the factory and supplies RSL 4. The factory returns a locator Client MMJFS 3. The factory creates a Managed Job Service MJS 5. The client subscribes to the MJS’ status SDE and retrieves output Open Grid Services Architecture 91
Information Services l Index service as caching aggregator u l Caches service data from other Grid services Index service as provider framework u Serves as a host for service data providers that live outside of a Grid service to publish data Open Grid Services Architecture 92
Reliable File Transfer l OGSI-compliant service exposing Grid. FTP control channel functionality u l Recoverable Grid service u l 3 rd-party transfer between Grid. FTP servers Automatically restarts interrupted transfers from the last checkpoint Progress and restart monitoring RFT JDBC Open Grid Services Architecture Grid. FTP Server 1 Grid. FTP Server 2 93
Reliable File Transfer Service Client Request and manage file transfer operations File Notf’n Policy Grid Service Transfer Source Fault Monitor Perf. Monitor Query &/or subscribe to service data Pending Performance Policy Faults interfaces service data elements Internal State Data transfer operations Open Grid Services Architecture 94
GT-OGSA Grid Service Infrastructure Grid Service Container User-Defined Services Base Services System-Level Services OGSI Spec Implementation Security Infrastructure Web Service Engine Hosting Environment Open Grid Services Architecture 95
GT 3 User-Defined Services l GT 3 can be viewed as a Grid service development kit that includes: u u u Primitives, APIs, tools and runtime services designed to ease the task of building OGSIcompliant services Primitives for provisioning security Base services that provide an infrastructure with which to build higher-level services Open Grid Services Architecture 96
GT 3 User-Defined Services (cont. ) GT 3 build files User source files Grid service executable files ANT (Diagram inspired by Borja Sotomayor’s excellent GT 3 tutorial) Open Grid Services Architecture User build file 97
Overview l Grid background l Open Grid Services Architecture l Open Grid Services Infrastructure l Beyond OGSI: other OGSA services l Globus Toolkit v 3 implementation l Early GT 3 performance results l Scientific and commercial perspectives l Summary Open Grid Services Architecture 98
Early GT 3 Performance Analysis l l Goal: explore GT 3 under heavy load u Maximal throughput/rate of GT 3 services u Identify limiting factors Three GT 3 Grid services measured u GRAM u Dummy. Service u Index. Service l Note: Preliminary results, performance continues to improve rapidly l Work performed by Massimo Lamanna, David Foster, et al. , at CERN Open Grid Services Architecture 99
GT 3 GRAM Performance: Setup l GRAM in GT 3 standalone container l Managed-job-globusrun clients started simultaneously on up to 32 client nodes (lxplus) in non-batch mode used to submit jobs to GT 3 GRAM l GRAM hardware: 2 * Intel Pentium III 600 MHz processors, 256 MB RAM l Note: 1 managed-job-globusrun client is able to submit 1 job Open Grid Services Architecture 100
GT 3 GRAM performance l Results: service node u Saturation throughput for job submission on the service node: 3. 8 jobs/minute with an average CPU user+system usage of 62% u u l Comments: u Scalability issue for heavily used servers Open Grid Services Architecture 101
GT 3 GRAM performance l Results: client node u l Using a 2 * Intel Pentium III 600 MHz processors, 256 MB RAM client node, a managed-job-globusrun client consumes at average 16 seconds CPU user+system time (on both CPU’s) for 1 job Comment: u Lightweight clients (e. g. , written in "C") needed Open Grid Services Architecture 102
Dummy. Service performance l Setup (1) u u l Each Dummy. Service client executes these steps: 1. Calls Dummy. Service. Factory to create a Dummy. Service instance 2. Executes 2 simple methods (echo and get. Time) on the Dummy. Service instance 3. Calls Dummy. Service instance to destroy itself Up to 1000 clients talking to the Dummy. Service from up to 45 client nodes (lxplus) With & without GSI message level authentication Grid service node hardware: 2 * Intel Pentium III 600 MHz processors, 256 MB RAM Setup (2) u Services Same, Architecture but repeats Open Grid step #2 one hundred times 103
Dummy. Service performance l Preliminary Results setup authentication service saturation container throughput average CPU u+s usage, % 1 - GT 3 standalone 41 services/s 89 1 yes GT 3 standalone 1. 3 services/s 88 1 - Tomcat 60 services/s 89 1 yes Tomcat 1. 2 services/s 88 2 - GT 3 standalone 300 method calls/s 96 2 yes GT 3 standalone 10 method calls/s 72 2 - Tomcat 290 method calls/s 96 2 Yes Tomcat 13 method calls/s 79 l Security overhead needs further investigation l Cross check our implementation/setup with Globus team foreseen Open Grid Services Architecture 104
Dummy. Service Performance l Conclusions u Security overhead needs further investigation u More tests on more powerful machines u l Container: depending on the setup the Tomcat container may be a bit slower or up to 50% faster, compared to the standalone container Notes: u u Results table shows top saturation rates With varying number of clients throughput goes down by up to 30% and the average CPU u+s usage varies accordingly Open Grid Services Architecture 105
Open Grid Services Architecture 106
l The first time the client contacts the Dummy. Service. Factory, creates a Dummy. Service instance, and calls the first method, it takes about 10 s to accomplish it l These actions take about 1 s on subsequent occasions Open Grid Services Architecture 107
Index. Service performance l Setup (1): Index. Service acting as a notification source (pushing data) u u u l Multiple notification sinks subscribe to the Index. Service "Host" Service Data Element (SDE), and are notified about each update of "Host" SDE, happening at a fixed rate No security Grid service node hardware: 2 * Intel Pentium III 600 MHz processors, 256 MB RAM Setup (2): Index. Service responding to find. Service. Data requests (pulling data) u u u Multiple ogsi-find-service-data clients are run sequentially and in parallel asking for Index. Service "Host" Service Data Element No security Grid service node hardware: 2 * Intel Pentium III 600 MHz processors, 256 MB RAM Open Grid Services Architecture 108
Index. Service Performance setup service saturation container throughput average CPU u+s usage, % 1 GT 3 standalone 10 -15 notifications/s 1 Tomcat - 2 GT 3 standalone 200 requests/s 88 2 Tomcat 200 requests/s 90 Open Grid Services Architecture 81 – 87 109
Index. Service Performance l Saturation throughput with find. Service. Data is ~13 -20 times higher than with notifications l Setup (1) measurement using Tomcat failed due to a thread bug, is fixed, fix announced to appear in next (4. 1. 28) Tomcat version l Preliminary measurement with a faster service node (2 * Intel(R) Xeon(TM) 2. 40 GHz processors, 1 GB RAM): u Saturation throughput for setup (1) was about 32 notifications/s for 800 listeners compared to 10 notifications/s with 400 listeners on the 2 * 600 MHz machines – not quite 4 times faster Open Grid Services Architecture 110
Open Grid Services Architecture 111
Open Grid Services Architecture 112
Overview l Grid background l Open Grid Services Architecture l Open Grid Services Infrastructure l Beyond OGSI: other OGSA services l Globus Toolkit v 3 implementation l Early GT 3 performance results l Scientific and commercial perspectives l Summary Open Grid Services Architecture 113
Scientific Perspectives l Importance of Grid technologies growing with “e. Science” & “cyberinfrastructure” l OGSA offers big opportunities l u Leverage commercial technologies u Achieve collaborative software ecosystem But also concerns u Potentially competing demands of industry u Sometimes slow pace of standards definition u Critical mass of R, D, and support Open Grid Services Architecture 114
NEESgrid Earthquake Engineering Collaboratory U. Nevada Reno www. neesgrid. org Open Grid Services Architecture 115
NEESgrid Core Capabilities l Tele-control and tele-observation of experiments l Data cataloging and sharing l Remote collaboration and visualization tools and services l Simulation execution and integration Open Grid Services Architecture 116
NEESgrid High-level Structure Open Grid Services Architecture 117
Architecture of NEESgrid Equipment Site Open Grid Services Architecture 118
Tele-Control Services l l l Transaction-based protocol and service (NTCP) to control physical experiments and computational simulations. OGAI based implementation (GT 3. 0) Plug-ins to interface the NTCP service A computational simulation written in Matlab u Shore Western control hardware u MTS control hardware (via Matlab and x. PC) u l Security architecture, including GSI authentication and a flexible, plug-in-based authorization model. Open Grid Services Architecture 119
Multi-site Online Simulation Test (MOST) l A large-scale experiment conducted in multiple geographical locations. u l Combine physical experiments with numerical simulation Psuedo-dynamic testing u Ideal for network latency reasons Open Grid Services Architecture 120
Substructure Pseudo-dynamic Testing Numerical Simulation Physical Tests Structural Test Geotechnical Test at Another Lab Open Grid Services Architecture Structural FE Simulation 1 Total System Geotechnical FE Simulation 2 121
Test Structure for MOST Experiment UIUC Experimental Model U. Colorado Experimental Model NCSA Computational Model m 1 f 2 f 1 m 1 f 2 Note: for ease of programming, all computational models were written in Matlab. Open Grid Services Architecture 122
MOST: A Grid Perspective UIUC U. Colorado Experimental Model F 1 e F 2 m 1 , q 1 f 1 , x 1 = f 2 NTCP SERVER SIMULATION COORDINATOR NCSA NTCP SERVER m 1 Computational Model f 1 Open Grid Services Architecture f 2 123
Tele-Control Services Open Grid Services Architecture 124
Industry Trends l l Economies of scale and outsourcing u Computing center outsourcing u Application Service Providers (ASPs) u Storage Service Providers (SSPs) Services orientation u l Web Services (and CORBA) influences Tantalizing dreams of B 2 B u u Still a sense that B 2 B market is huge If only enough inter-organization problems can be overcome … Open Grid Services Architecture 125
The Value of Grid Computing: IBM Perspective Increased Efficiency Higher Quality of Service Increased Productivity & ROI Reduced Complexity & Cost Improved Resiliency Open Grid Services Architecture 126
IBM Grid Product Offerings l IBM Grid Toolbox: u u u l l l Includes GT 2. 2 IBM value-added extras Commercial support! Websphere Application Server (supports OGSIcompliant Grid services) Grid education services IT consulting/infrastructure planning Compute/storage hardware provision Integrated solutions for market sectors Open Grid Services Architecture 127
Butterfly. net: Enterprise Optimization l. A scalable, resilient infrastructure for creating & running massive multiplayer games l Developers l Improved avoid upfront costs end-user experience l Demonstrated 8 x increase in profitability over centralized model l Dynamic provisioning & on demand capacity l Uses Globus Toolkit & runs on IBM Global Services hosting environment Open Grid Services Architecture 128
“The HP Stack” HP’s value add is the UDC, management and security, and consulting services to enhance and build upon industry standards Applications Globus/OGSA Web Services Grid-Enabled Resources & Resource Utilities (UDC) Open Grid Services Architecture HP Consulting Services Management and Security l 129
On the Road to Planetary Computing virtual data center computing utility or GRID value programmable data center switch compute fabric storage grid-enabled systems UDC Tru 64, HP-UX, Linux clusters Open VMS clusters, Tru. Cluster, MC Service. Guard today Open Grid Services Architecture shared, traded resources 130
HP Grid Product Offerings l Globus Toolkit for HP-UX, Tru 64 Unix, Linux Globus Toolkit 2. 4 u Tested and optimized for HP platforms u Commercial support! u l HP Utility Data Center (UDC) Data Center operation & construction services u Service-centric design u Intent is to construct UDC products using OGSI-compliant services u l Enterprise Grid consulting Open Grid Services Architecture 131
Oracle’s Use of Grid Technology l Use Grid technology to build better products u Oracle Database 10 g l l u Oracle Application Server 10 g l l l Enhanced scalability, relocation, & distributed SQL Max database size -> 8 exabytes Already based on J 2 EE/Web Services Extending to include OGSI yields powerful capabilities Improves scalability and flexibility Increases in both scalability and efficiency Improves competitiveness of existing products Open Grid Services Architecture 132
Oracle Grid Product Offerings l Oracle Database 10 g u u u l Transportables Distributed SQL Managed using OGSI-compliant interfaces(? ) Oracle Application Server 10 g u u u Hosting for OGSI-compliant Grid services Development environment Application Server can be managed and configured using OGSI-compliant interfaces(? ) Open Grid Services Architecture 133
Platform Symphony: Real-Time Online Processing • Automatically connect applications to services • Dynamic & intelligent provisioning Applications: Delivery Application Virtualization Application Services: Distribution Infrastructure Virtualization Servers: Execution Open Grid Services Architecture • Dynamic & intelligent provisioning • Automatic failover 134
Overview l Grid background l Open Grid Services Architecture l Open Grid Services Infrastructure l Beyond OGSI: other OGSA services l Globus Toolkit v 3 implementation l Early GT 3 performance results l Scientific and commercial perspectives l Summary Open Grid Services Architecture 135
Summary: Standards l Standards are critical to Grid success l Grid and Web services are merging u Grid is an aggressive use case of Web Services l Web services standards landscape is in great flux, and OGSI will need to evolve with it l Grid services standards landscape heating up u W 3 C, OASIS, GGF are key standards orgs l Uncertain status of security & policy standards continues to be a big source of concern l Open source software important for adoption Open Grid Services Architecture 136
Summary l Grid: resource sharing, integration, virtualization in network systems l OGSA is defining a standards-based Grid infrastructure l u Based on Web services u Core specification, OGSI, is complete u Numerous other specifications underway GT 3 provides tools for building OGSI-based Grid services u Available today Open Grid Services Architecture 137
For More Information l The Globus Alliance® u l Global Grid Forum u l www. ggf. org Background information u l www. globus. org www. mcs. anl. gov/~foster Globus. WORLD 2004 u www. globusworld. org u Jan 20– 23, San Francisco 2 nd Edition: November 2003 Open Grid Services Architecture 138
- Ogsa in grid computing
- Ian foster university of chicago
- Foster open science
- Open innovation open science open to the world
- Lga vs pga
- Jenkins and selenium grid
- Clis
- Sgeee
- Open science grid
- Open grid service infrastructure
- Open science grid computer
- Smart grid architecture model
- Overview of grid architecture
- Open xml sdk tutorial
- Open compute project tutorial
- Openplc tutorial
- Open scad
- Open inventor tutorial
- Computer architecture tutorial
- Software architecture tutorial
- Software architecture diagram tutorial
- Mips sw example
- Rindx
- Sql master data manager
- Foster claire keegan
- The scream surrealism
- Foster fuch spots
- Dr andrew foster deaf
- Foster fuchs spot
- Effects of multiple placements in foster care
- Effects of multiple placements in foster care
- Elizabeth foster fsu
- Mizzz foster
- Va medical foster home
- Dr andrew foster deaf
- Andrea foster mack
- Holston homes foster care
- Reparation agreement magsaysay
- Foster and freeman vsc 80
- Foster lifelong engagement
- James foster gwu
- Sian foster
- Adoption and foster care analysis and reporting system
- A poem to my foster dog
- John h. chafee foster care independence program eligibility
- Medical foster home program
- How foster children
- Portal arg
- Ekstrasystoler foster
- Tiburcio de espadaña
- Social impulses foster infant language
- Social impulses foster infant language
- Cultural relativism tends to foster
- Intelligence cycle
- Prayer of relinquishment foster
- Foster freeman
- Foster freeman
- Foster and maintain business relationships
- Florida probe disadvantages
- Baldur god
- Hannah webster
- Stayz amsterdam
- Nancy foster aha
- Saliromani
- Foster youth college success initiative
- Rebekah foster
- Horse whisperer meaning
- Foster parent association
- Ros foster
- Nancy foster aha
- Trauma informed care for foster youth
- Dwight eisenhower apush
- New foundations home for child anderson sc
- Mizzz foster
- "foster business school"
- "foster business school"
- Normally open timed open symbol
- Open hearts open hands
- Open architecture examples
- Open virtual switch architecture
- Open platform communications unified architecture
- Modular open systems architecture
- Cisco cloud architecture
- Open architecture
- Open architecture
- The open group architecture framework
- Autosar lin
- Open architecture api
- Open architecture radar
- Open agent architecture
- Open control architecture
- The open group architecture framework
- Open services for lifecycle collaboration
- Cycle
- Return architecture
- What is product architecture
- Types of product architecture
- Bus architecture in computer organization
- Layered architecture for web services and grids
- Sql server reporting services architecture
- Rfc 1633
- Integrated services vs differentiated services
- Wake county human services community services center
- Ian milborrow pwc
- Ian juliano
- Cloud ral
- Dr ian arnott
- Ian scott international
- Ian lipio
- Ian coddington
- Contractive autoencoder
- Ian crotty
- Ian slater dwf
- Ian lindsley
- Albert tomasso
- Ian rice md
- A mcpherson pro reunion
- Ian thurgood
- Ian jarman
- Dr ian mundy
- Ian dawson history
- Ian gordon lawyer
- Goodfellow mlschifferzoeschiffer
- Gans goodfellow
- Ian tibbetts
- Ian white ucl
- Ian hartwig
- Ian hembry
- Dr ian sando
- Ian watt the rise of the novel summary
- Stefanie molin
- Ian tharp
- Oak tree care services
- Ian axelrod
- Dr ian gallimore
- Ian lawson md
- Ian walker economics
- Professor ian wilcox
- Ian viney