grid computation and web services 1 Agenda Grid

grid computation and web services 1

Agenda • Grid computation. – Motivation, a very simple example. – Architecture and implementation problems. • Web Services – What are they? , why do we need them? – Some technologies involved XML, SOAP, WSDL, UDDI. • Putting it all together grid computation and web services 2

Motivation (some history) • 95% of CPU power in the world is idle. • Multiple scientific tasks like require supercomputing power (weather forecast, weapons simulations, etc). • The cost of this supercomputers is millions of dollars. • Why not using existing resources to tackle this tasks. grid computation and web services 3

The Idea • Use the CPU power available with the help of volunteers ready to contribute to your project. • Connect the computers throw existing networks (internet). • Get a de facto supercomputer. • Save millions. grid computation and web services 4

SETI@home • SETI (Search for Extra-Terrestrial Intelligence). • SETI@Home connects more than 4. 000 computers throughout the world having a workpower bigger than any existing super-computer. • SETI Statistics. – – Users: 4. 763. 661 (the original target was 300. 000). Total CPU time: 1706429. 454 years. FPO: 3. 979678 e+21. 24 hours FPO 61. 86 TFLOPS/sec. • The earth simulator (350 million US$) has 40 TFLOPS/sec. grid computation and web services 5

CRASS (1) • The problem. – More than 7000 observed orbiting objects larger than 10 cm. – Between 70. 000 to 120. 000 objects larger than 1 cm. – High speeds can make collisions fatal, making the debris problem worse. – 1 on 10 shuttle missions have performed collisions avoidance maneuvers. grid computation and web services 6

CRASS (2) • USS space com DB tracks the position of the debris. • CRASS uses the DB resources and a model of the forces acting to predict possible collisions. • CRASS master assigns to each node the operational spacecraft and some debris. grid computation and web services 7

CRASS (3) • The nodes calculate the debris orbit and issue collisions warnings in a specific time scope. • Master receives the results and assign another period of time till the simulation is completed. grid computation and web services 8

CRASS (4): Architecture MASTER Scheduler node grid computation and web services node DB -Test speed of the nodes -Distribute the debris accordingly node 9

Conclusions for CRASS • CRASS works because the debris does not interact between them. • No need of communication between nodes. • Overhead of data is small (the number of operational spacecraft is small in relation to the debris). • The problem is easily distributed. • But there are other not so simple cases. • Grids are more than CRASS. grid computation and web services 10

A Grid is more than that • A grid is a seamless computing information environment composed of: – Storage systems. – Networks. – Data servers. – On-line instruments (telescopes, particle accelerators, etc). – Computing processors. grid computation and web services 11

Implementing a Grid • Putting together so different elements is a complex task. – Need of a common language between them. – Need of communications protocols. – Security issues. • Grid applications are usually distributed and thus hard to program. grid computation and web services 12

Implementing a Grid (2) • Component (object) oriented programming has helped. – Reusability of code. – OO design subdivides problems in small ones. Suitable approach to deploy big applications. • Making objects work together is still a hard task involving many issues. • CORBA was supposed to be the solution grid computation and web services 13

More difficulties • • Slow communications between computers. Wide geographical area. Different users platforms (WIN, UNIX, MAC). Some problems require strong interaction between the different CPUs. – For these problems supercomputers cannot be replaced (for example: chain reactions, weather simulations. . . ). – So we still need supercomputers. grid computation and web services 14

Dividing the problems • It’s very important to divide developers and users. – Scientific programmers (no knowledge of grid technologies). – Grid developers (no knowledge of the scientific problem). – End users (no knowledge at all). • Each group needs appropriate tools to solve their problems. • We need tools to connect the elements. grid computation and web services 15

Dividing the problems (2) • Throughout the presentation we will focus in the second group problems (grid architecture). • Some new arising tools called hosting environments help also to deploy highly distributed applications. – The programs (elements, objects) do not run over the OS but over a hosting environment. grid computation and web services 16

Grid Portals • Grid portals are a solution for end users. • They provide an easy way to use then resources of a grid with no special knowledge of computing. • The user will be able to use the portal for: – Locate grid resources. – Set in the portal the task he wants to perform. – Define execution parameters (number of processors, memory needs, etc). grid computation and web services 17

Nimrod Portal Applet grid computation and web services 18

Grid Architecture elements • Security Protocols – Authentication and privacy. • Information Services – users need to see which resources are available for use. • Schedulers – multiple resources can be scheduled concurrently. grid computation and web services 19

An Example of a Grid Portal Architecture Web Browser (2 ) (1 ) (4 ) Certificates Server Web Server Messages DB (3 ) Application Manager grid computation and web services (4)The Grid (1)The user browser (2)The Server (3)The Grid connects to the checks if the user is Application Manager Portal Server. a registered grid is notified of the connects to the user presence through a messages channel 20

Towards Grid Services • The grid model is developing. – Not only scientific cooperation. – Business want to link their different departments maybe in a very wide geographical area using grid technologies. – Outsourcing: companies are using specialized companies and reducing their IT departments. • Economies of scale. • Price/performance ratio. grid computation and web services 21

Towards Grid Services (2) • We would like to make grids more open. Other people to use the grid capabilities. • Redefine the grid elements as services. • service: network-enabled entity that provides some capability. • The key issue is interoperability between services. grid computation and web services 22

Towards Grid Services (3) • Redefine the architecture elements of a grid as services: – Security services: authentication and privacy. – Information services: what resources do we have in the grid (data, processors, instruments). – Job-submission resources. – Co-scheduling service: multiple resources. – Caching: in large grids caching data can improve performance dramatically. grid computation and web services 23

Towards Grid Services (4) • The interoperability problem basically consists in: – Defining what the service does. – Giving instructions for it’s use (invocation APIs) • We would also like a mechanism to find appropriate services. • We need semantic information about the services. grid computation and web services 24

It’s Not Easy • We need specific protocols (hopefully universally accepted) to define our services. • XML helps with it’s semantic approach but it’s not enough. • If we use the Internet as a transport channel how do we avoid firewalls? – RPCs are not firewall-friendly. grid computation and web services 25

Agenda • Grid computation. – Basic ideas. – Implementation problems. • Web Services – What are they? , why do we need them? – Some technologies involved XML, SOAP, WSDL, UDDI. • Putting it all together grid computation and web services 26

What the Web Can Offer • Web sites offer a myriad of services. Buy books, check stocks, bank transactions… • We would like to be able to do that automatically. • Amazon could buy book from providers when it’s DB shows stock is low. • Business would like to talk to each other. • We need a ways to define and provide services. grid computation and web services 27

Some History • Earlier attempts to B 2 B failed because HTML was not able to give enough semantic information about the services provided. • In 2001 several companies got together with W 3 C and came up a precise set of XML based standards in order to help B 2 B. grid computation and web services 28

Some Standards • WSDL Web Services Description Language. – An XML Schema that defined the service capabilities and how to invoke it. • UDDI Universal Description, Discovery and Integration. – Registry specification on how to publish WSDL documents. • WSIL Web Services Inspection Language. – How to find WSDL documents on a site. • The idea is that these tools will help a user to find a WS and use it (automatically). grid computation and web services 29

A Possible Scenario USER User chooses the price User checks prices offered Registry sends tobest user the by USER wants to buy books and buys the book by the services matching services ISBN, searches the registry using WSIL Buy books by isbn WS WS Registry Amazon. com BN. com Harper. Collins. com Registry (written in UDDI) contains WSDL documents describing the services grid computation and web services 30

Using a WS • There are two types of web services. – RPC (remote procedure call): the user calls a function provided by the service and gets a value. – DOC: the user invokes the service and gets a documents as a response. • In order to use the service we need to know how to call it, i. e under which protocol is it working (TCP, SMTP, HTTP or other). grid computation and web services 31

Understanding a WS • A WS is a collection of endpoints. • An endpoint is a combination of a binding and an address (URI). • A binding is a concrete protocol and data format for a port type • The set of message exchanges is called an operation in WSDL terms: – error messages, encoding, parameters type, etc. grid computation and web services 32

Understanding a WS (2) • Port types are interfaces (in wsdl 1. 2 they are changing the term). • Related concrete end points can be grouped in interfaces and thus provide abstract endpoints, also called services grid computation and web services 33

Messages TCP HTTP SMTP endpoints interface operation SERVICE Application (resource) grid computation and web services 34

SOAP • Communication with the WS is done through messages, however we do not want to write the XML messages ourselves. • SOAP simple object access protocol solves that. • SOAP is simply a standard to send messages. There are others, but Microsoft, W 3 C and Sun use SOAP so it’s a de facto standard. grid computation and web services 35

SOAP (2) • We send a message in a SOAP envelope. Using whatever protocol we like (FTP, HTTP, JMS, . . . ) • The SOAP processor is the one in charge of understanding the SOAP message and call the WS. • There are Apache packages for java and Microsoft packages designed to use SOAP… we’ll see some examples. grid computation and web services 36

SOAP (3) SOAP request SOAP response Client XML Parser Application HTTP Firewalls grid computation and web services XML Parser (SOAP translator) Server Application 37

Using SOAP to Invoke a WS • Barnes and Noble provides a service to get a books price with it’s ISBN – method name: get. Price(String isbn). – The service is located in the Web Server of Xmethods. • Send though HTTP a SOAP message requesting the price of isbn = 0439139597 (Harry Potter 5). grid computation and web services 38

The SOAP envelope header: information about the SOAP specification SOAP request <SOAP-ENV: Envelope xmlns: SOAP-ENV= "http: //schemas. xmlsoap. org/soap/envelope/" The information about the method xmlns: xsi="http: //www. w 3. org/1999/XMLSchemainstance" invoked and the encoding type xmlns: xsd="http: //www. w 3. org/1999/XMLSchema"> The parameter needed (isbn) <SOAP-ENV: Body> <ns 1: get. Price xmlns: ns 1="urn: xmethods. BNPrice. Check" SOAP-ENV: encoding. Style= "http: //schemas. xmlsoap. org/soap/encoding/"> <isbn xsi: type="xsd: string">0439139597</isbn> </ns 1: get. Price> </SOAP-ENV: Body> </SOAP-ENV: Envelope> grid computation and web services 39

SOAP answer <SOAP-ENV: Envelope xmlns: SOAP-ENV= "http: //schemas. xmlsoap. org/soap/envelope/" xmlns: xsi="http: //www. w 3. org/1999/XMLSchemainstance" xmlns: xsd="http: //www. w 3. org/1999/XMLSchema"> The return value of the method <SOAP-ENV: Body> <ns 1: get. Price. Response xmlns: ns 1="urn: xmethods. BNPrice. Check" SOAP-ENV: encoding. Style= "http: //schemas. xmlsoap. org/soap/encoding/"> <return xsi: type="xsd: float">15. 57</isbn> </ns 1: get. Price. Response> </SOAP-ENV: Body> </SOAP-ENV: Envelope> grid computation and web services 40

Using JAVA with SOAP • SOAP packages for JAVA give the possibility to: – Create SOAP envelopes. – Set the envelope data (URL, URI, method name, parameters, … – SEND the envelope. – GET the answer (also a SOAP envelope). – PARSE the envelope and get the relevant data. grid computation and web services 41

Java SOAP import java. net. URL; import java. util. Vector; import org. apache. soap. *; import org. apache. soap. rpc. *; public class Client { public static void main(String[] args) throws Exception { The URN of the method and the URL url = new invoked method name URL("http: //xmethods. vet/sd/2001/"+ ”BNQuote. Service"); // Build the call. Call call = new Call(); call. set. Target. Object. URI("urn: xmethods. BNPrice. Check"); call. set. Method. Name(”get. Price"); grid computation and web services 42

Java SOAP (2) call. set. Encoding. Style. URI(Constants. NS_URI_SOAP_ENC); Vector params = new Vector(); params. add. Element(new Parameter("name", String. class, args[0], null)); call. set. Params(params); // Invoke the call. Response resp = null; Setting args[0] as the isbn to look try { resp = call. invoke(url, ""); } catch( SOAPException e ) { System. err. println("Caught SOAPException (" + e. get. Fault. Code() + "): "+ e. get. Message()); System. exit(-1); } grid computation and web services 43

Java SOAP (3) //Check the response. if( !resp. generated. Fault} ( () Parameter ret = resp. get. Return. Value; () Object value = ret. get. Value ; () Object value = ret. get. Value(); System. out. println(value; ( System. out. println(value); { } else { Fault fault = resp. get. Fault ; () Fault fault = resp. get. Fault(); Getting the returned value System. err. println("Generated fault; (" : System. err. println("Generated fault: "); System. out. println (" Fault Code+ " = System. out. println (" Fault Code = " + fault. get. Fault. Code ; (() fault. get. Fault. Code()); System. out. println (" Fault String+ " = System. out. println (" Fault String = " + fault. get. Fault. String; (() fault. get. Fault. String()); grid computation and web services 44

Agenda • Grid computation. – Basic ideas. – Implementation problems. • Web Services – What are they? , why do we need them? – Some technologies involved XML, SOAP, WSDL, UDDI. • Putting it all together grid computation and web services 45

The Idea • Use the WS standards to solve the implementation problems of a grid. • WS are platform-independent and programming-language independent. • Communication protocols are widely accepted (firewalls-friendly). • WSDL offers a way to publish the services. • UDDI offers a way to find the wanted service. grid computation and web services 46

Grid Services As WS (2) • However there are grid needs that are not provided by the WS specifications and tools. We need new interfaces. – Grids need dynamic creation of application instances. (The application instance is created when the user needs it). – Grids need lifetime management. operations. – WS are usually stateless. grid computation and web services 47

Grid Factories • The factory creates an instance of a new WS in some resource. – Dynamic creation of WS is supported is hosting environments as J 2 EE Server, . NET and AXIS. • An application manager instance is created and publishes a WSDL document. • The client can contact the new instance through the WSDL directly (we do not need the factory any more). – A messages channel is opened. • Keep alive messages, destroy. . . grid computation and web services 48

Grid Factories (2) Grid Factory Service User Resource Broker Service Application Instance Service grid computation and web services Authentication Service Grid users DB 49

Lifetime Management • Normally a transient service will be created and will run till it’s task termination. However errors can (and do) happen. • Need a ways to kill idle WS that consume resources. A possible solution will be: – The service is created with a specific lifetime. – The user can request to extend the lifetime. – The user needs to send keep-alive messages, or explicitly kill it. grid computation and web services 50

User Storage Factory Service user DB Storage Service Instance Storage Mining Factory Service Mining Service Instance - The user invokes “create service” from the 2 arethe Keep As no Alive more messages keep-alive and messages status messages are received sent factories. Storage room is allocated. services kill themselves freeing all their resources. grid computation and web services 51

The role of hosting environments • A grid service architecture should be independent of the implementations issues. • Nowadays most grid application rely in native OS (for example the creation of a new service means the creation of a process). • This fact means for example that the factory service has to be platform-aware. grid computation and web services 52

The role of hosting environments (2) • WS can be created with more sophisticated tools called hosting environments or containers as J 2 EE (Sun), Web. Sphere (IBM) or. NET (Microsoft) – These containers work in a componentbased basis. • Hosting environments build complex applications by using components that adhere to specific interfaces (i. e. EJB). grid computation and web services 53

The role of hosting environments (3) – Building applications which can have components distributed in different machines and places. – Communications are handled by the hosting environment. – Easy security. – Is not the component that handles communication or creation of instances but the hosting environment. – Messaging between components is also handled by the host no matter where the are. grid computation and web services 54

The role of hosting environments (4) An accountancy. The bean askslooks container beans the bean host in communicate thefor wants directory a handle to as getifto some the where they sales-bean. where data the bean from running is the sales and in the returns department same a machine. handle DB. Communications and security issues are managed from the host Container Clients DB Sales bean Accountancy bean grid computation and web services 55

The role of hosting environments (5) • This approach affects also the third level of users (scientific programmers). • The programmer needs to follow specific interfaces for his application. • In other words if he is a java programmer he needs to deliver beans. grid computation and web services 56

Conclusion • WS provide a powerful tool to implement grid services. • New interfaces (hopefully widely accepted) should be created to support specific grid needs: – Factories. – Lifetime management. – Security. . . grid computation and web services 57

References • Articles. – The anatomy of the grid. www. globus. org/research/papers/anatomy. pdf. – The physiology of the grid. www. globus. org/research/papers/physiology. pdf. – Programming the grid: distributed software components, P 2 P and grid web services for scientific applications. http: //citeseer. nj. nec. com/gannon 02 programming. html. • Grid services for distributed system integration. www. globus. org/research/papers/ieee-cs-2. pdf. grid computation and web services 58

References (2) • Some links – – SOAP: www. w 3. org/TR/SOAP/ UDDI: www. uddi. org WDSL: www. w 3. org/TR/wsdl To see some WS in action: www. xmethods. net – Microsoft’s WS page: http: //msdn. microsoft. com/webservices/defa ult. aspx grid computation and web services 59