JRA 1 ITCZ Meeting 14 15122004 www euegee

JRA 1 IT-CZ Meeting, 14 -15/12/2004 www. eu-egee. org Distributed Superscheduling Matteo Mordacchini INFN – Padova Università Ca’ Foscari di Venezia EGEE is a project funded by the European Union under contract IST-2003 -508833

Distributed Superscheduling Problem • When submitting a job, the UI uses a list of known RBs • If the first RB fails to satisfy the request, the UI will resubmit the request to the second RB on the list, and so on. Problems • Maybe not all the RBs with resources useful for the user are included in the list. • If the RBs capable of satisfying the request are not on top of the list, there is a great waste of time and resources before reaching the right site. • Some brokers could be overloaded (while other brokers could be under-loaded) JRA 1 IT-CZ Meeting, 14 -15/12/2004 - 2

Distributed Superscheduling Problem Aim • Find a (fast) way to allow RBs to collaborate in order to satisfy a user’s request. • Don’t affect the current scheduling methods in the case the first RB already has the resources capable of processing the job. • Forwarding of job requests to another WMS must be analysed in detail, but seems feasible JRA 1 IT-CZ Meeting, 14 -15/12/2004 - 3

Distributed Superscheduling Some Considerations • This “collaboration” among brokers should be transparent to the end-user • Having *all* resources (CEs) available on the Grid seen by a single broker (or by all brokers) can have scalability problems Solutions in literature • Construct a distributed index of the resources and organize them into a overlay network • Different brokers considering different subsets of the existing Grid resources JRA 1 IT-CZ Meeting, 14 -15/12/2004 - 4

Distributed Superscheduling Indexing • Some solutions suggest to organize the resources into a multidimensional space, where every attribute represents a dimension of the space. • The space will be divided into zones. • Every index entry points to one of this zone and then it is used to access all the resources that lay in that region of space. JRA 1 IT-CZ Meeting, 14 -15/12/2004 - 5

Distributed Superscheduling Overlay Networks • Given the above space subdivision, each region is assigned to a node in the network. • Every node is responsible of giving the information about the resources that belong to its region § routing request that affect other regions, passing it to the neighbour that is the closest to the interested regions. § • The network is self-organizing: the presence of a resource is communicated to only one node; then it will collaborate with the other nodes in order to find the right region (and associated node) where the resource has to be indexed JRA 1 IT-CZ Meeting, 14 -15/12/2004 - 6

Distributed Superscheduling - Proposal • Use some of the nodes in the Grid to construct an overlay network similar to that described above. • Instead of indexing the single resources (CEs), we suggest to index RBs, in order to find the best set of RBs that are (with high probability) able to solve a given user request. RB s RB s JRA 1 IT-CZ Meeting, 14 -15/12/2004 - 7

Distributed Superscheduling - Proposal • In order to index RBs, we need to create a sort of “RB profile” § Considering in some way the content of the ISM • The creation of the “RB profile” is done as follows: § The resources of the RB are grouped into sets based on the similarities of the characteristics of the CEs. § A representative of every set is created. § The RB profile is formed by the representatives of the sets. • The RB will be indexed in as much regions of the space as the number of different representative he has in its profile JRA 1 IT-CZ Meeting, 14 -15/12/2004 - 8

Distributed Superscheduling - Proposal Search operations • If a RB cannot satisfy a user request, it asks to a node in the overlay network to find a set of RB that can process the given request. • Once it has this set, it could ask to the RBs in the set to check if they really can process the request (they really have the proper CEs available); then, he will transfer the request to one of the RBs that have answered positively. JRA 1 IT-CZ Meeting, 14 -15/12/2004 - 9

Distributed Superscheduling - Proposal Advantages • For registration and query operations, every RB needs only to know the address of one node in the overlay network. • RB profiles could be updated less frequently than the descriptions of CEs. • The process is completely transparent to the user. Disadvantages • In order to improve performances, it is better to replicate the nodes in the overlay network. • Staging of Input- and Output-Sandbox • Communications to the LB. JRA 1 IT-CZ Meeting, 14 -15/12/2004 - 10

Distributed Superscheduling References • S. Ratnasamy, P. Francis, M. Handley, R. Karp, “A Scalable Content Addressable Network”, ACM SIGCOMM’ 01, Aug. 27 -31, 2001 • I. Stoica, R. Morris, D. Karger, M. Frans Kaashoek, H. Balakrishnan, “Chord: A Scalable Peer-to-Peer Lookup Service for Internet Applications”, ACM SIGCOMM’ 01, Aug. 27 -31, 2001 • A. Andrzejak, Z. Xu, “Scalable Efficient Range Queries for Grid Information Services”, 2 nd IEEE International Conference on Peer-to-Peer Computing, Sept. 5 -7, 2002, Linköping, Sweden • P. Ganesan, B. Yang, H. Garcia-Molina, “One Torus to Rule Them All: Multi-dimensional Queries in P 2 P Systems”, 7 th International Workshop on the Web and Databases (Web. DB 2004), June 17 -18, 2004, Paris, France JRA 1 IT-CZ Meeting, 14 -15/12/2004 - 11

Distributed Superscheduling • C. Schmidt, M. Parashar, “Flexible Information Discovery in Decentralized Distributed Systems”, 12 th IEEE International Symposium on High Performance Distributed Computing (HPDC'03), June 22 - 24, 2003, Seattle, Washington • M. Cai, M. Frank, J. Chen, P. Szekely, “MAAN: A Multi-Attribute Addressable Network for Grid Information Services”, 4 th International Workshop on Grid Computing, Phoenix, Arizona, 2003 • K. Aberer, P. Cudré-Mauroux, A. Datta, Z. Despotovic, M. Hauswirth, M. Punceva, R. Schmidt, “P-Grid: A Self-organizing Structured P 2 P System”, ACM SIGMOD Record, Vol. 32, No. 3, September 2003 • A. M. Ouksel, G. Moro, “G-Grid: A Class of Scalable and Selforganizing Data Structures for Multi-dimensional Querying and Content Routing in P 2 P Networks”, Technical Report no. DEIS-LIA 002 -04, February 2004, Univ. of Bologna, Univ. of Illinois at Chicago JRA 1 IT-CZ Meeting, 14 -15/12/2004 - 12

Distributed Superscheduling • E. Tanin, A. Harwood, H. Samet, “Indexing Distributed Complex Data for Complex Queries”, Proceedings of the National Conference on Digital Government Research, pp. 81 -90, Seattle, Washington, 2004 JRA 1 IT-CZ Meeting, 14 -15/12/2004 - 13