Technical Forum Madrid 16 th20 th Sept 2013

Technical Forum Madrid, 16 th-20 th Sept. 2013 Advantages of adopting late-binding techniques through standardised interfaces for workflow managers. A. J. Rubio-Montero 1, M. Plociennik 2, I. Marín-Carrión 3, T. Zok 2, M. A. Rodríguez-Pascual 1, R. Mayo-García 1, M. Owsiak 2, E. Huedo 3, F. Castejón 1, 4 and B. Palak 2. 1 CIEMAT, Centro de Investigaciones Energéticas Medioambientales y Tecnológicas, Avda. Complutense, 40, 28040 Madrid, Spain. 2 PSNC, Poznań Supercomputing and Networking Center, IBCh, PAS, Noskowskiego 12/14, 61 -704 Poznań, Poland. 3 Fac. Informática, UCM, Prof. José García Santesmases s/n. 28040 MADRID, Spain. 4 Euratom-CIEMAT Association, Avda. Complutense, 40, 28040 Madrid, Spain. Introduction Visual workflow systems such as Kepler[1] are fundamental tools to implement and maintain complex scientific simulations. These workflow managers allow directly interfacing HPC platforms, databases, repositories or grid resources in a single framework[2]. However, this approach means to be tied to middleware implementations and to bear the performance slowdown due to waiting queues. A way to improve the grid execution performance is the use of pilot-job techniques, which were successfully tested with workflows managers in approaches as Pegasus[3] or P-Grade[4]. Nevertheless, they inherit the issues from current pilot systems that lack some features such as easy-installing, user-sharing or the absence of standardised interfaces to remotely access which make them unfeasible for many projects. The aim of this work is to offer an unified, simplified, efficient, flexible, standardised and portable new mechanism to remotely manage pilot jobs and the tasks that compound a Kepler workflow. For this purpose, the new pilot system framework GWpilot[5] based on the Grid. Way[6] metascheduler is utilised and accessed through the standard OGSA-BES interface[7]. Objectives • To take advantage of the Grid. Way modularity to build a new multi-level scheduling framework based on the performance and reliability provided by Gwpilot. • To achieve portability and interoperability required by EGI and OGF through the OGSA-BES interface. • To preserve backward compatibility with legacy middleware by means of Kepler actors and Grid. Way drivers. Advantages Standardised remote task delegation through OGSA. • A Kepler simulation that can be check-pointed and carried out on laptops. • Several Kepler instances (or other applications) can also use the platform. Underneath management of user tasks and pilot jobs without control loss of provisioning: • Task requirements will automatically be translated to pilot provisioning. • Two ways for data allocation: via customisation tasks or directly accessing to storage services. • Pre-allocation, reservation, data-location awareness, and multi-user coordination are allowed. Benefits and impact • The creation, maintenance and execution workflows with pilots jobs are simplified. of • It allows implementing complex scheduling policies to improve specific workflows. [1] I. Altintas, C. Berkley, E. Jaeger, M. Jones, B. Ludaescher and S. Mock. Kepler: An Extensible System for Design and Execution of Scientific Workflows, Proc. of SSDBM 2004. pp. 423– 424. DOI: 10. 1109/SSDM. 2004. 1311241 [2] L. Cabellos, I. Campos, E. Fernández-del-Castillo, M. Owsiak, B. Palak and M. Plociennik. Scientific workflow orchestration interoperating HTC and HPC resources, Comp. Phys. Comm. 182(4) (2011) 890– 897. DOI: 10. 1016/j. cpc. 2010. 12. 020 [3] G. Juve, E. Deelman, K. Vahi, G. Mehta, Experiences with resource pro-visioning for scientific workflows using Corral, Scientific Programming 18 (2) (2010) 77– 92. DOI: 10. 3233/SPR-2010 -0300 [4] P. Kacsuk, T. Kiss, G. Sipos. Solving the grid interoperability problem by P-GRADE portal at workflow level, Future Generation Computer Systems 24(7) (2008) 744– 751 DOI: 10. 1016/j. future. 2008. 02. 008 [5] E. Huedo, R. S. Montero, I. M. Llorente, A modular meta-scheduling architecture for interfacing with pre-WS and WS Grid resource management services, Future Generation Computer Systems 23 (2) (2007) 252– 261. DOI: 10. 1016/j. future. 2006. 07. 013 [6] A. J Rubio-Montero, E. Huedo, F. Castejón, R. Mayo-García. GWpilot: Enabling multi-level scheduling in distributed infrastructures with Grid. Way and pilot jobs. To appear. [7] I. Marín, E. Huedo, I. M. Llorente. Interoperating Grid Infrastructures with the Grid. Way Metascheduler, Concurrency and Computation: Practice and Experience. (2012). In press. DOI: 10. 1002/cpe. 2971 This work is the result of a coordinate effort supported by the EGI SA 1, SA 3 and IGE WP 4 Tasks.