Solving Interoperability Challenges between GARUDA grid and European

Solving Interoperability Challenges between GARUDA grid and European grid (EGI) in CHAIN-REDS Experiments Author: Mr Henry Sukumar henrys@cdac. in Presented by: Sarat Chanda Babu ISGC 2015 Centre for Development of Advanced Computing Bangalore, INDIA 19 March 2015, ISGC 2015, Taiwan 1

Outline 1. 2. 3. 4. 5. 6. Introduction GARUDA- National grid computing initiative CHAIN REDS Interoperability challenges Conclusion Reference 19 March 2015, ISGC 2015, Taiwan 2

C-DAC Presence & Size 11 Centres across India Workforce of 3000 members 19 March 2015, ISGC 2015, Taiwan 3

The Garuda grid GARUDA – National grid computing initiative bringing together academic, scientific and research communities for developing their data and compute intensive applications. Partners – More than 70 institutes 19 March 2015, ISGC 2015, Taiwan 4

CHAIN-REDS: Co-ordination & Harmonisation of Advanced e-Infrastructures for Research and Education Data Sharing CHAIN-REDS science gateway has been built in the context of EUCHAIN and CHAIN-REDS projects to demonstrate how the science gateway paradigm and standard adoption can make e-Infrastructure worldwide, based on different middleware and architectures (Grid, HPC, Cloud or simply local clusters), interoperable amongst each other, at user application level. Courtesy: chain-project. eu 19 March 2015, ISGC 2015, Taiwan 5

Role of CDAC in CHAIN Specific Goals of the collaboration are: To enhance the interoperation capacities of both infrastructures with the focus on Virtual Organization's using, or interested to use, both infrastructures. To provide Local and Global operational services as needed to support the members of such virtual organizations To cooperate and exchange information about common operation activities; To participate in the Security Policy Group to contribute to the development of the security policies fabric of the infrastructure. 19 March 2015, ISGC 2015, Taiwan 6

Collaborators: Ø Ø Ø Ø Ø CNGrid, China EGI In. Spire, Europe ENEA-CRESCO EUMEDGRID-Support, Egypt Future. Grid GARUDA, India GISELA, Grid Initiatives for e-Science virtual communities in Europe and Latin America IGI (Italian Grid Infrastructure) SAGrid – South African National Grid 19 March 2015, ISGC 2015, Taiwan 7

Interoperability “ The ability of a system or a product to work with other systems or products without special effort on the part of the customer ” The International Electrotechnical Commission (IEC) has identified two types of interoperability: Syntactic interoperability as the ability of two or more systems to communicate and exchange data. Semantic interoperability as the ability of two or more systems to automatically interpret the exchanged data.

GARUDA – CHAIN Interoperability Setup 19 March 2015, ISGC 2015, Taiwan 9

Overview of CHAIN environment Science gateway, a web portal with embedded applications for users to submit jobs. The Standard-based (SAGA) middleware independent Grid Engine that can interoperate with wide range of Grid middleware such as Globus, GLITE, Unicore etc… The help desk service is handled using Global Grid User Support (GGUS), allows users to create ticket and track its status from creation to final resolution 19 March 2015, ISGC 2015, Taiwan 10

Science Gateway Science gateway, a web portal with embedded applications for users to submit jobs. A Science Gateway is a community-developed set of tools, applications, and data that are integrated via a portal or a suite of applications, usually in a graphical user interface, that is further customized to meet the needs of a specific community. Gateways enable entire communities of users associated with a common discipline to use national resources through a common interface that is configured for optimal use. Researchers can focus on their scientific goals and less on assembling the cyberinfrastructure they require. Gateways can also foster collaborations and the exchange of ideas among researchers. Ref: https: //www. xsede. org/gateways-overview 19 March 2015, ISGC 2015, Taiwan 11

Scientific Gateways (cntd/-) Depending on the needs of the communities, a gateway may provide any of the following features: High-performance computation resources Workflow tools General or domain-specific analytic and visualization software Collaborative interfaces Job submission tools Education modules Example: OSDD portal on Garuda 19 March 2015, ISGC 2015, Taiwan 12

Science Gateways (cntd/-) Science Gateways commonly take one of these three forms: A gateway packaged as a web portal, with users using the front end to access digital services provided through the back end. A grid-bridging gateway, with communities running their own grids devoted to their areas of science (in these cases, a Science Gateway is a mechanism for extending the reach of the community’s grid by coupling it with digital services from another organization). A gateway that lets users run a desktop or mobile application to access remote digital services. 19 March 2015, ISGC 2015, Taiwan 13

Science Gateways (cntd/-) Science Gateways commonly take one of these three forms: A gateway packaged as a web portal, with users using the front end to access digital services provided through the back end. A grid-bridging gateway, with communities running their own grids devoted to their areas of science (in these cases, a Science Gateway is a mechanism for extending the reach of the community’s grid by coupling it with digital services from another organization). A gateway that lets users run a desktop or mobile application to access remote digital services. Ref: http: //sciencegateways. org/what-is-a-science-gateway/ 19 March 2015, ISGC 2015, Taiwan 14

OSDD – Garuda Crowdsourcing approach to genome scale re-annotation of Mycobacterium Tuberculosis Identification of novel drug targets Predictive modeling Unique research initiative with large scale community participation in collaborative experiments Hosts a central repository of synthesized chemicals for screening against TB & Malaria Large Student community from colleges and universities are cloning, expressing and purifying the genes Computational models run by students on HPC resources of the Garuda Grid Uses Galaxy Workflow 15

CHAIN REDS - Science Gateway https: //science-gateway. chain-project. eu/ 19 March 2015, ISGC 2015, Taiwan 16

Simple API for Grid Applications (SAGA) The Simple API for Grid Applications (SAGA) is a family of related standards specified by the Open Grid Forum to define an application programming interface (API) for common distributed computing functionality. To hide Middleware heterogenity - The Standard-based (SAGA) middleware independent Grid Engine that can interoperate with wide range of Grid middleware such as Globus, GLITE, Unicore etc… To hide infrastructures heterogenity- (EGI, IGI, Garuda, . . ) Java implementation of SAGA is JSAGA A high level API for efficiently submitting Jobs 19 March 2015, ISGC 2015, Taiwan 17

SAGA Core API specification covers the following areas: security and session management permission management asynchronous operations monitoring asynchronous notifications attribute management I/O buffer management 19 March 2015, ISGC 2015, Taiwan 18

SAGA Architecture The SAGA core implementation defines the Typical SAGA architecture: a light-weight runtime system dispatches API calls from the application to the middleware through a set of plug-ins or adaptors. 19 March 2015, ISGC 2015, Taiwan 19

Global Grid User Support (GGUS) GGUS is a Help. Desk (At CHAIN side) single entry point for reporting problems Distributed infrastructure with central coordination A first line support team supports generic grid pronblems, while the specialised units answer to middleware, network deployment, Virual organization specific problems etc. . 19 March 2015, ISGC 2015, Taiwan 20

Overview of GARUDA Environment GARUDA Grid is built on top of Globus Toolkit and Gridway Metascheduler. The users can submit jobs via Grid Portal and command-line using Gridway scheduler commands. Request Tracker (RT) is used for helpdesk service. (RT is a tickettracking system used to coordinate tasks and manage requests among a community of users) PBS: Portable Batch System; SGE: Sun Grid engine 19 March 2015, ISGC 2015, Taiwan 21

CHAIN-GARUDA Interoperability Two Main challenges How the users registered at Science Gateway can submit a job to GARUDA? and vice versa, how the users of GARUDA can use Science Gateway to submit jobs. How the users of Science Gateway will raise a Job submission issue to GARUDA Administrators? and vice versa, how the users of GARUDA can raise an issue to Science Gateway administrators? 19 March 2015, ISGC 2015, Taiwan 22

GARUDA – CHAIN Job Management Interface 19 March 2015, ISGC 2015, Taiwan 23

GARUDA – CHAIN Job submission Interface The Job Management between CHAIN and GARUDA was handled with the help of Grid Engine (at CHAIN) and Grid Gateway (at GARUDA). The Generic Grid Engine is based on JSAGA Uniform data and job management across different grid infrastructure/middleware. It’s Globus at GARUDA Supports WS-GRAM adaptor and Pre-WS GRAM adaptor Written in Java – works on both Windows and Linux • • • Grid Engine – Based JSAGA in Science gateway JSAGA - Java implementation of the Simple API for Grid Applications (SAGA) PBS – Portable Batch System – Torque used at GARUDA WS-GRAM – Web service Grid Resource Allocation Manager (GRAM) Pre-WS – Non web service based Grid Resource Allocation Manager (GRAM) 24

JSAGA to Globus, Glite, Unicore 19 March 2015, ISGC 2015, Taiwan 25

Generic Grid Engine at Science Gateway Slide Courtesy : Roberto Barbera, CHAIN 19 March 2015, ISGC 2015, Taiwan • • Grid Engine – Based JSAGA in Science gateway JSAGA - Java implementation of the Simple API for Grid Applications (SAGA) 26

Grid Gate. Way at GARUDA JSAGA WS-GRAM adaptor to GARUDA JSAGA WS-GRAM adaptor provides remote job submission to GARUDA. But Gridway Meta scheduler does not provide remote job submission directly. Solution : To make Gridway to accept remote job submission from JSAGA, we need a component called Grid Gateway. Grid Gate. Way enables the remote access to Grid. Way's metascheduling capabilities through a WSRF interface Advantage : permits the virtualization of a grid (accessible through the encapsulated Grid. Way) through a WS-GRAM service 19 March 2015, ISGC 2015, Taiwan 27

Grid Engine remote job submission to GGW Users Implementation part of CHAIN-REDS project Globus Science Gateway WS-GRAM Adaptor Grid Gateway Gridway JSAGA CHAIN HPC Grid Globus PBS PBS Cloud GARUDA • • • GGW – Grid Gate. Way Grid Engine – JSAGA API in Science gateway JSAGA - Java implementation of the Simple API for Grid Applications (SAGA) PBS – Portable Batch System – Torque used at GARUDA WS-GRAM – Web service Grid Resource Allocation Manager (GRAM) 19 March 2015, ISGC 2015, Taiwan 28

CHAIN – GARUDA Job submission Interface - Status Currently, the setup is being tested with the job submission from Grid Engine to Grid. Gate. Way. Challenges faced… Cross Certification JSAGA WS-GRAM and Globus 4. 0. 7 version compatibility issue Need to use JSAGA Pre-WS GRAM adaptor instead of WS-GRAM Pre-WS Globus support at GARUDA Globus 4. 0. 7 does not support SHA-2 based grid certificate. Data transfer of job outputs from GARUDA to CHAIN • • JSAGA - Java implementation of the Simple API for Grid Applications (SAGA) WS-GRAM – Web service Grid Resource Allocation Manager (GRAM) 19 March 2015, ISGC 2015, Taiwan 29

GARUDA - CHAIN Job submission Interface - Status GROMACS version 4. 5. 3 was supported by Science Gateway. GARUDA users were in need of GROMACS 5. 0, but only Gromacs 4. 6. 5 was supported • • JSAGA - Java implementation of the Simple API for Grid Applications (SAGA) WS-GRAM – Web service Grid Resource Allocation Manager (GRAM) 19 March 2015, ISGC 2015, Taiwan 30

GARUDA – CHAIN Help Desk Interoperability 19 March 2015, ISGC 2015, Taiwan 31

Help desk Interoperability CHAIN handles user support using Global Grid User Support (GGUS) system GARUDA uses Request Tracker (RT) Challenge Interoperability of GGUS and RT will Science Gateway users to raise issue ticket in GARUDA RT via GGUS And vice versa, GARUDA users to raise issue ticket in GGUS via RT The integration should be seamless and should not require additional authentication and authorization 19 March 2015, ISGC 2015, Taiwan 32

CHAIN – GARUDA Helpdesk Interoperability Implementation part of CHAIN-REDS project GGUS at CHAIN SOAP services RT at GARUDA • The main scope of this task is the implementation of the automatic communication between the GGUS and GARUDA Request Tracker (RT) systems via the exchange of SOAP messages. • GGUS has already implemented web services, so the necessary web services must have been implemented at the GARUDART GGUS - Global Grid User Support RT – Request Tracker SOAP - Simple Object Access Protocol 19 March 2015, ISGC 2015, Taiwan 33

Helpdesk Interoperability – 2 Possible scenarios • Scenario 1: A ticket is created at the GGUS, where the ticket submitter or the TPM on duty team assigns the ticket at the NGI_INDIA support unit, using the “Assign to ROC/NGI” field of the “Submit ticket” form of the GGUS. In this scenario, a corresponding ticket must be created at the appropriate queue of the GARUDA RT. • Scenario 2: A ticket is created at the GARUDA RT, where the ticket submitter assigns the ticket at the GGUS support unit. In this scenario, a corresponding ticket must be created at the GGUS - Global Grid User Support RT – Request Tracker SOAP - Simple Object Access Protocol 19 March 2015, ISGC 2015, Taiwan 34

GGUS – RT Implementation Sample implementation details and scripts can be download from Hellasgrid (successfully integrated GGUS with Hellasgrid RT) http: //wiki. hellasgrid. gr/wiki/bin/view/Hellas. Grid/GOC/GGUSAnd. Hellas. Grid. R TSystems. Integration 19 March 2015, ISGC 2015, Taiwan 35

CHAIN – GARUDA Interoperability Status • Currently, the ticket transfer from GGUS to RT has been tested successfully. Further work is in progress - the attachment synchronization and establishing ticket transfer from RT to GGUS - Global Grid User Support RT – Request Tracker 19 March 2015, ISGC 2015, Taiwan 36

Conclusions The challenges addressed during the interoperation of CHAIN-REDs and GARUDA Grid were connectivity, Middleware support, resource information aggregation, data transfer between the resource elements, monitoring, help desk integration and cross certification. 19 March 2015, ISGC 2015, Taiwan 37

Team Mr. S, Henry Sukumar (C-DAC) Mr. J, Santhosh (CDAC) Ms. M. G, Divya (CDAC) Dr. Subrata, Chattopadhyay (C-DAC) Mr. GIUSEPPE, La Rocca (INFN) Mr. MARIO, Torrisi (INFN) Mr. GUENTER, Grein (CHAIN) Mr. GKAMAS, Vasileios (CHAIN-REDS) Mr. ROBERTO, Barbera (CHAIN-REDS) 19 March 2015, ISGC 2015, Taiwan 38

References http: //www. cadc. in http: //garudaindia. in http: //chain-project. eu 19 March 2015, ISGC 2015, Taiwan 39

Thank You 19 March 2015, ISGC 2015, Taiwan 40