Universit degli Studi di Bari Corso di Laurea
Università degli Studi di Bari – Corso di Laurea Specialistica in Informatica “Tecnologia dei Servizi “Grid e cloud computing” A. A. 2009/2010 Giorgio Pietro Maggi giorgio. maggi@ba. infn. it, http: //www. ba. infn. it/~maggi Lezione 2 a - 14 ottobre 2009 Il materiale didattico usato in questo corso è stato mutuato da quello utilizzato da Paolo Veronesi per il corso di Griglie Computazionali per la Laurea Specialistica in Informatica tenuto nell’anno accademico 2008/09 presso l’Università degli Studi di Ferrara. Paolo Veronesi paolo. veronesi@cnaf. infn. it, pveronesi@unife. it http: //www. cnaf. infn. it/~pveronesi/unife/ Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 0
Grid Architecture Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 1
Grid Architecture: Definition (1/2) p The Grid Architecture: n n identifies the fundamental system components (services and VOs); specifies purpose and function of these components; indicates how these components interact with each other. Functions: It identifies key areas in which services are required; p It defines standard protocols and APIs to facilitate creation of interoperable Grid systems and portable applications. Grid architecture protocols define the basic communication mechanisms by which: p p n n p Virtual Organization users and resources negotiate, establish, manage and exploit sharing relationships; different services interact. Grid architecture services are the components whose standardization facilitates extensibility, interoperability, portability and code sharing. Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 2
Grid Architecture: Definition (2/2) U 8 VO 2 U 7 U 6 COMPONENTS protocol S 2 U 1 S 3 protocol S 1 U 2 protocol 1. VOs and their users 2. Services 3. Protocols: - between services and VOs protocol S 4 VO 1 U 5 U 3 Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 3
Resource Sharing Address security and policy concerns of resource owners and users p Are flexible enough to deal with many resource types and sharing modalities p Scale: p n n to large number of resources, many participants, many program components to large data management tasks and computing tasks Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 4
Service Sharing 1) Need for interoperability when different groups want to share resources n n Diverse components, policies, mechanisms E. g. , standard notions of identity, means of communication, resource descriptions 2) Need for shared infrastructure services to avoid repeated development, installation n n E. g. , one port/service/protocol for remote access to computing, not one per tool and/or application E. g. , use of shared Certificate Authorities as they are expensive to run A common need for protocols & services Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 5
Layered Grid Architecture (by Analogy to the Internet Architecture) Grid Internet Architecture “Coordinating multiple resources”: ubiquitous infrastructure services, app-specific distributed services “Sharing single resources”: negotiating access, controlling use “Talking to Grid elements”: communication (Internet protocols) & security “Controlling elements locally”: Access to, & control of, resources Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a Collective Application Resource Connectivity Transport Internet Fabric Link 6 Internet Protocol Architecture Application
Layered Grid Architecture p p Layers are ment to group services of similar nature in the same class They do not intend to prescribe how services at different layers interact. Communication between services at the same and/or different layer is possible, depending on the scenario. n n Example 1: The workload management service (Collective layer) uses the Resoure discovery service (still Collective layer) to find resources that match the user’s rquirements Example 2: an application may submit a job: p p p By invoking the Workload manager service (re-submission, logging and bookkeping, input/output management, Collective) By invoking the resource layer through a standardized API (Resource) By submitting directly to a specific local resource instance (Fabric) Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 7
Protocols, Services, and APIs Occur at Each Level Applications Languages/Frameworks Collective Service APIs and SDKs Collective Services Resource APIs and SDKs Resource Services Collective Service Protocols Resource Service Protocols Connectivity APIs Connectivity Protocols Local Access APIs and Protocols Fabric Layer Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 8
Fabric Layer (1/2) p Examples: n n n p p p Computing farm Mass storage device file catalogs network link file systems sensors, etc. It provides access to the resources to which shared access is mediated by Grid protocols. Fabric components implement local, resource-specific operations (through internal protocols), which are transparent to the upper layers thanks to connectivity and resource level protocols, that define interfaces, not the physical internal characteristics. Operations at the fabric layer are triggered by resource sharing operations at a higher level Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 9
Fabric Layer: Capabilities (2/2) p Examples of capabilities n Computational resource: p p n Start programs Monitor execution and corresponding processes Management of resources (e. g. advance reservation) Enquiry of hw/sw capabilities, current load, queue waiting time, etc. Storage resource: p p p Put/get files Third party and high-performance data transfer Read/write subset of file Management of disk space, disk bandwidth, network bandwidth, CPU, etc. Enquiry of hw/sw characteristics, available space, bandwidth utilziation, etc. Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 10
Connectivity Layer: Protocols & Services p p It supports secure communication between Fabric-layer resources by defining the core communication and authentication protocols required for grid-specific network functions. Communication: n p transport (e. g. IP), naming (e. g. DNS), routing, etc. (the TCP/IP protocol stack) Security: the Grid Security Infrastructure (GSI) n Uniform authentication, authorization, and message protection mechanisms in multi-institutional setting n Single sign-on: users must be able to authenticate just once and then have access to multiple resources, without further user intervention; n Delegation: a user’s program can access the resources on which the program owner is authorized. n Integration with local security solutions: Grid security needs to interoperate with local security solutions adopted by the resource managers n User-based trust relationship: a user can access to resources in different domains without requiring security administrators in each domain to interact with others in different domains. In other words, authorization is only based on the user’s certificate, not on the authentication/authorization performed by other servers in different domains. n Public key cryptography: Secure Socket Layer (SSL) n Supporting infrastructure: Certificate Authorities, certificate & key management Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 11
Resources Layer: Protocols & Services p Resource layer defines protocols, APIs, and SDKs for: n n n p p Resource layer implementation relies on the Fabric layer functions. Two main components define this layer: n n p secure negotiations, initiation, monitoring control, accounting, and payment of sharing operations on individual resources. information protocols: used to obtain the information about the structure and state of the resource, e. g. : configuration, current load and usage policy. management protocols: used to negotiate access to the shared resource, specifying for example Qo. S, advanced reservation, etc. Examples: n n n Access to compute cluster, storage, network information Invocation of resource service providers Access to local scheduler Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 12
Collective Layer: Protocols & Services p p The Collective Layer contains protocols and services that coordinate multiple resources concurrently, i. e. they capture interactions among a collection of resources. It supports a variety of sharing behaviors without placing new requirements on the resources being shared. Examples: n n n Directory services: they allow VO participants to discover the existence and/or properties of VO resources. Co-allocation, scheduling: they allow VO participants to request the allocation of one or more resource for a specific purpose and the scheduling of tasks on the appropriate resources. Monitoring: support the monitoring of VO resources (intrusion, failure, load, . . ) Data replication: supports the management of VO storage resources to maximize data access performance with respect to metrics such as response time, reliability, etc. Workload management: description, use and management of complex workflows (e. g. submission of jobs with mutual dependencies) Authorization servers, accounting, collaboratory services (synchronous/asynchronous exchange of messages) Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 13
Application Layer p p It includes user applications that operate within a Virtual Organization environment. Applications are constructed such that invocation of services and use of protocols defined at any layer are possible. For example, a given user can submit jobs by invoking services at different layers: n n Collective Layer: by submitting to a Workload Manager (it can provide: internal queue in: case of submission failure, periodic re-submission, access to logging and bookkeeping information, input/output file management) Resource Layer: by submitting directly to a compute resource scheduler (e. g. a local cluster) Connectivity Layer: by submitting to a remote cluster Fabric Layer: by submitting to a local cluster Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 14
Three Generations of Grid Source: Charlie Catlett • Local “metacomputers“ – Distributed file systems – Site-wide single sign-on • "Metacenters" explore inter-organizational integration • Totally custom-made, top-to-bottom: proofs of concept • Utilize software services and communications protocols ! developed by grid projects: ere h e r a to deliver complete solution • Need significant customization e W • Interoperability is still very difficult! – Condor, Globus, UNICORE, Legion, etc. • Common interface specifications support interoperability of discrete, independently developed services • Competition and interoperability among applications, toolkits, and implementations of key services Standardization is key for third-generation grids! Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 15
Who Defines Standard for Grid p OGF: Open Grid Forum n n p OASIS: Organization for the Advancement of Structured Information Standards n n p http: //www. oasis-open. org not-for-profit consortium that drives the development, convergence and adoption of open standards for the global information society DMTF: Distributed Management Task Force n n p http: //ogf. org community of users, developers, and vendors leading the global standardization effort for Grid computing http: //www. dmtf. org industry organization leading the development, adoption and promotion of interoperable management standards and initiatives W 3 C: World Wide Web n n http: //www. w 3. org develops interoperable technologies (specifications, guidelines, software, and tools) to lead the Web to its full potential Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 16
Open Grid Forum http: //www. ggf. org http: //www. gridalliance. org • Birthed in high performance • Birthed in enterprise data computing community in 1998 center community in 2004 • Merged with European & Asian grid efforts in 2001 • Raised awareness of grids in “end-user” organizations • 52 groups and 81 documents • 5 groups and several important documents Merger completed, June ’ 06; OGF Launched September of ‘ 06 Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 17
Broad and Narrow Grids p OGF distinguishes between broad and narrow Grids p A Broad Grid is any collection of distributed services n p Marketing-oriented, not rigorous definition A Narrow Grid n n n Collaboration Grid Data Center Grid Cluster Grid Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 18
Collaboration Grids Multiple institutions, secure, widely distributed, VOs p Collaborative agreements & commercial partnerships p Financial Model: Increase overall revenue p Examples: p n EGEE http: //www. eu-egee. org Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 19
Data Center Grid Centralized management of multiple platforms p Aggregation of enterprise resources and applications p Financial Model: Reduce Total Cost Ownership (TCO) p Examples: p n Google, e. Bay, Amazon Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 20
Cluster Grid p Networks of Workstations, Blades, etc. p Cycle scavenging, Homogeneous workload p Financial Model: n Lower marginal costs Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 21
How Are Grids Used? High-performance computing Collaborative design E-Business High-energy physics Financial modeling Life sciences Data center automation Drug discovery E-Science Collaborative data-sharing Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 22
OGSA The Open Grid Services Architecture (OGSA) architecture describes the functionality required to implement a Grid infrastructure, and the rendering of this functionality into service definitions Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 23
OGSA Capabilities OGSA “profiles” Web services foundation Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 24
OGSA Capabilities Execution Management • Job description & submission • Scheduling • Resource provisioning OGSA “profiles” Web services foundation Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 25
OGSA Capabilities Execution Management • Job description & submission • Scheduling • Resource provisioning Data Services • Common access facilities • Efficient & reliable transport • Replication services OGSA “profiles” Web services foundation Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 26
OGSA Capabilities Execution Management • Job description & submission • Scheduling • Resource provisioning Data Services • Common access facilities • Efficient & reliable transport • Replication services Resource Management • Discovery • Monitoring • Control OGSA “profiles” Web services foundation Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 27
OGSA Capabilities Execution Management • Job description & submission • Scheduling • Resource provisioning Data Services • Common access facilities • Efficient & reliable transport • Replication services Resource Management Self-Management • Discovery • Monitoring • Control • Self-configuration • Self-optimization • Self-healing OGSA “profiles” Web services foundation Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 28
OGSA Capabilities Execution Management • Job description & submission • Scheduling • Resource provisioning Data Services • Common access facilities • Efficient & reliable transport • Replication services Resource Management Self-Management • Discovery • Monitoring • Control • Self-configuration • Self-optimization • Self-healing OGSA Information Services • Registry • Notification • Logging/auditing OGSA “profiles” Web services foundation Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 29
OGSA Capabilities Execution Management • Job description & submission • Scheduling • Resource provisioning Data Services • Common access facilities • Efficient & reliable transport • Replication services Resource Management Self-Management • Discovery • Monitoring • Control • Self-configuration • Self-optimization • Self-healing OGSA Information Services Security • Registry • Notification • Logging/auditing • Cross-organizational users • Trust nobody • Authorized access only OGSA “profiles” Web services foundation Tecnologia dei Servizi “Grid e cloud computing” - Lezione 002 a 30
- Slides: 31