Condor Bird Bath SOAP Interface to Condor Charaka
Condor Bird. Bath SOAP Interface to Condor Charaka Goonatilake Department of Computer Science University College London c. goonatilake@cs. ucl. ac. uk 1
Background 2
Motivations § Adding Web Service interfaces to Condor daemons § Facilitate the development of third-party applications capable of interacting with Condor (remotely). – E. g. build higher-level application specific scheduler that submits jobs to multiple Condor pools based on application semantics – These can be built using a wide range of languages/SOAP packages – Bird. Bath has been tested on: • • Java (Apache Axis, XSUL) Python (ZSI) C# (. Net) C/C++ (g. SOAP) § Condor accessible from platforms where its command-line tools are not supported/installed 3
The (Hypothetical) Big Picture Site D Globus/grid. SAM Condor-G SOAP Schedd Site A Client Schedd Site B flock ing Site C Schedd 4
Condor SOAP Interfaces § Now available in Condor 6. 7. 5 § Begin/commit/abort transaction § Submit Job § Obtain Job Class. Ads § Get/Send file § Removed/hold/release job Schedd § Obtain resource Class. Ads § Obtain all daemon Class. Ads Execution machine Collector Negotiator Shadow Submission machine(s) Central manager Execution machine 5
Clients for Bird. Bath Custom Client hold. Job() Submission machine(s) SOAP library Schedd § WSDLs available that describe each daemon interface § SOAP libraries (stub code) can be generated automatically from WSDLs 6
Example: Query Collector § Obtain information (Class. Ads) about all resources that have over 512 MB RAM collector. Locator = new Condor. Collector. Locator(); collector = locator. getcondor. Collector(new URL(“http: //kotturoti. cs. ucl. ac. uk: 9618”); class. Ads = collector. query. Startd. Ads(“Memory>512”); 7
Example: Job Submission 1. 2. 3. 4. 5. 6. Create New Transaction Create New Job. Id Send Files ({input files, binaries}) Create Job Description Submit Job Commit Transaction § Transaction based process § Uses two phase commit to promote reliability and robustness § Files transferred in chunks – Base 64 encoded SOAP attachments § DAGMan jobs supported 8
OMII Collaboration: grid. SAM plugin § Job Submission & Monitoring Web Service that submits to range of resource managers (e. g. Condor, SGE, Globus) § Uses JSDL (Job Submission Description Language) for standardised job description Client JSDL SOAP Grid. SAM Condor SGE Client <? xml version="1. 0" encoding="UTF-8"? > <Job. Definition xmlns="http: //…"> <Job. Description> <Application> JSDL <Executable>/bin/echo</Executable> <Argument>hello world</Argument> SOAP </Application> </Job. Description> </Job. Definition> Grid. SAM Fork SGE … Fork … 9
grid. SAM plugin Client § Existing Condor plugin uses shell-based submission via command-line tools and monitoring of log files JSDL SOAP Middle-tier Grid. SAM § Our plugin accesses Condor via the SOAP interfaces § Building JSDL to Class. Ad conversion libraries Condor-WS Condor SGE Fork … Class. Ad SOAP Schedd Submission machine(s) 10
Future Work § Bundle plug-in into grid. SAM – Also provide separate JSDL to Class. Ad conversion library § Extensions to other daemons (what functionality to expose? ) – For example, allow checkpoint data to be retrieved through the shadow in order to migrate jobs across pools § Roll out daemons on e-minerals mini-grid 11
- Slides: 11