Independent Software Vendor ISV Remote Computing Primer Steven

The Problem…. • Growth of compute oriented applications • Research applications with source available

Goals • How can standards enable access to these distributed resources? • What scenarios

Specification Toolbox • • • WS-Addressing Security specifications and profiles Job Submission Description Language

Core Specifications • WS-Addressing • Encapsulates service in Endpoint Reference (EPR) in XML •

Job Submission Description Language • Core JSDL specification (GFD 56 136) has many implementations

Running jobs • Basic Execution Service (BES) (GFD 108) • Provides an interface to

Other OGF Specifications • Byte. IO (GFD 87) • Access through a web service

Application Scenarios 1. 2. 3. 4. Run a job through a standard API Run

Run a job • The current ‘state of the art’ • Application calls an

Run a job using a standard API • Application internally calls a standard API

Run a job through a web service • HPCBP client embedded in the application

Extension: File Staging • Use the HPCBP File Staging specification • Can include ftps,

Extend: Run-time interaction • Use Byte. IO in streaming mode to: • Send control

Summary • Core set of established interoperable specs. • Demonstrated interoperability • Commercial and

Acknowledgements • Use Case Workshop participants • University of Virginia, March 2008 • Core

Slides: 18

Download presentation

The Problem…. • Growth of compute oriented applications • Research applications with source available • Commercial Applications from ISVs have no source • ISV: Independent Software Vendors • Applications becoming bound by the desktop • Multi-core pushing out this boundary • But ensemble/parameter sweeps increases demand • Exploit resources provided within the enterprise • Growing adoption of HPC into ISV applications • How to deal with different schedulers, clusters & OS? © 2008 Open Grid Forum

Goals • How can standards enable access to these distributed resources? • What scenarios need to be supported? • Deal with realistic network environments • Firewalls and NAT’ed networks • Move from desktop to mobile clients • Clients are not always connected to the network • Clients will not have files on shared network • Embrace different scheduler interfaces © 2008 Open Grid Forum

Specification Toolbox • • • WS-Addressing Security specifications and profiles Job Submission Description Language (JSDL) JSDL Single Process Multiple Data Application Extension JSDL Parameter Sweep Extension Basic Execution Service (BES) HPCP-Application Extension HPC Basic Profile (HPCBP) File Staging Extension to the HPC Basic Profile Byte. IO GLUE (Resource Description) Distributed Resource Management Application API (DRMAA) • Resource Namespace Service (RNS) © 2008 Open Grid Forum

Core Specifications • WS-Addressing • Encapsulates service in Endpoint Reference (EPR) in XML • From a client perspective • Security specifications and profiles • Builds on core standard WS-Security infrastructure • WS-Secure Addressing (GFD 131): • Profile on WS-Addressing and WS-Security Policy • Embeds into EPR how to access a service • WS-Secure Communication (GFD 132): • Profile on mechanisms to enable easier interoperability © 2008 Open Grid Forum

Job Submission Description Language • Core JSDL specification (GFD 56 136) has many implementations • Describes what you want to run (the job) • Describes what you need to run it on (the resource) • Extensions • JSDL Single Process Multiple Data (SPMD) Application Extension (GFD 115) • JSDL Parameter Sweep Extension • Now entering public comment • HPCP-Application Extension (GFD 111) • Supported in HPC Basic Profile © 2008 Open Grid Forum

Running jobs • Basic Execution Service (BES) (GFD 108) • Provides an interface to submit, monitor & manage an activity • HPC Basic Profile (HPCBP) (GFD 114) • Specialises BES (& other specifications) for HPC • File Staging Extension to the HPC Basic Profile (GFD 135) • Profiles JSDL to allow transfers in & out of a cluster © 2008 Open Grid Forum

Other OGF Specifications • Byte. IO (GFD 87) • Access through a web service to a file abstraction • Support for random access & streaming patterns • GLUE (Finishing public comment) • Information model for grids • Focus on services & resources for compute & storeage • Expose virtual organization & usage controls • Distributed Resource Management Application API (DRMAA) (GFD 22) • Client API to enable access to DRMs • Resource Namespace Service (RNS) (GFD 101) • Naming of resources in a hierarchical namespace © 2008 Open Grid Forum

Application Scenarios 1. 2. 3. 4. Run a job through a standard API Run a job through a web service with file staging 5. Run a job through a web service and have bi -directional interaction with it © 2008 Open Grid Forum

Run a job • The current ‘state of the art’ • Application calls an external script • Script invokes a local job submission command • Requires: • Installation of local scheduler library • Customization of scripts to access local scheduler © 2008 Open Grid Forum

Run a job using a standard API • Application internally calls a standard API • DRMAA specification with multiple implementation • Within DRMAA invoke scheduler specific Application plugin DRMAA Interface • Requires: DRMAA plug-in Scheduler • Installation of local scheduler library • Installation of DRMAA plugin to call the library © 2008 Open Grid Forum

Run a job through a web service • HPCBP client embedded in the application • Formulates the job using JSDL & HPCP Application • HPCBP service used to access the cluster • Available for many schedulers • Requires: • Installation of HPCBP service on the cluster • Integration of the HPCBP client in the application • Note: Cross-platform interoperability demonstrated at SC 06 & SC 07 © 2008 Open Grid Forum

Extension: File Staging • Use the HPCBP File Staging specification • Can include ftps, ftp, scp sftp, and other protocols • By using an intermediary file store the client can disconnect & reconnect later • • Client copies the files to the intermediary store Client then submits the job to the cluster The cluster reads/writes files to the intermediary store Once the job is complete the files are retrieved • Requires: • Support of standard file protocols on client, cluster & store © 2008 Open Grid Forum

Extend: Run-time interaction • Use Byte. IO in streaming mode to: • Send control requests from the client to the application • Return data from the application to the client • Byte. IO service needs to be accessible from the client and the cluster’s compute nodes • E. g. the cluster’s head nodes • Requires: • Byte. IO service and integration into application & client © 2008 Open Grid Forum

Summary • Core set of established interoperable specs. • Demonstrated interoperability • Commercial and open-source implementations • ‘Toolbox’ of specifications enables applications • Allows ISVs to build applications that: • Can connect to any ‘standard’ infrastructure • Run from mobile clients in network environments © 2008 Open Grid Forum

Acknowledgements • Use Case Workshop participants • University of Virginia, March 2008 • Core scenario: Narfi Stefansson, Mathworks • Vigorous discussion over the two days • Further contribution & feedback at OGF 23 • And on mailing list and OGSA calls • Resulting document is now being published • Editors: Steven Newhouse & Andrew Grimshaw © 2008 Open Grid Forum