The g Lite Workload Management System Giuseppe LA

The g. Lite Workload Management System Giuseppe LA ROCCA INFN Catania giuseppe. larocca@ct. infn. it ACGRID-II School 2 -14 November 2009 Kuala Lumpur - Malaysia

Outline • Introduction to the Job Description Language (JDL); – Main attributes; – Examples • Overview to the g. Lite Workload Management System (WMS); – Main components; – WMProxy Server; – Command Line Interface; – Troubleshooting; • References • Hands on

Job Description Language • The Job Description Language (JDL) is a high-level language based on the Classified Advertisement (Class. Ad) language, used to describe jobs and aggregates of jobs with arbitrary dependency relations. – The JDL is used in WLCG/EGEE to specify the desired job characteristics and constraints, which are taken into account by the WMS to select the best resource to execute the job. – A job description is a file (called JDL file) consisting of lines having the format: attribute = expression; – Expressions can span several lines, but only the last one must be terminated by a semicolon.

Job Description Language • The character “ ‘ ” cannot be used in the JDL. • Comments must be preceded by a sharp character (#) or a double slash (//) at the beginning if each line. • Multi-line comments must be enclosed between “/*” and “*/”. Attention! The JDL is sensitive to blank characters and tabs. No blank characters or tabs should follow the semicolon at the end of a line.

Simple JDL example Executable = "/bin/hostname"; Std. Output = "std. out"; Std. Error = "std. err"; The Executable attribute specifies the command to be run by the job. If the command is already present on the WN, it must be expressed as a absolute path; if it has to be copied from the UI, only the file name must be specified, and the path of the command on the UI should be given in the Input. Sandbox attribute. Executable = "test. sh"; Input. Sandbox = {"/home/doe/test. sh"}; Std. Output = "std. out"; Std. Error = "std. err";

• The Arguments attribute can contain a string value, which is taken as argument list for the executable: Arguments = "file. A 10"; • In the Executable and in the Arguments attributes it may be necessary to use special characters, such as &, , |, >, <. These characters should be preceded by triple in the JDL, or specified inside quoted strings e. g. : Arguments = "-f file 1\&file 2"; • The attributes Std. Output and Std. Error define the name of the files containing the standard output and standard error of the executable, once the job output is retrieved.

• If files have to be copied from the UI to the execution node, they must be listed in the Input. Sandbox attribute: Input. Sandbox = {"test. sh", . . , "file. N"}; • The files to be transferred back to the UI after the job is finished can be specified using the Output. Sandbox attribute: Output. Sandbox = {"std. out", . . , "std. err"}; • The shell environment of the job can be modified using the Environment attribute. Environment = {"CMS_PATH=$HOME/cms", "CMS_DB=$CMS_PATH/cmdb"};

• Wildcards are allowed only in the Input. Sandbox attribute; • Absolute paths cannot be specified in the Output. Sandbox attribute; • The Input. Sandbox cannot contain two files with the same name, even if they have a different absolute path, as when transferred they would overwrite each other.

Job. Type – Normal (simple, sequential job), Interactive, MPICH, Checkpointable, Partitionable, Parametric “Interactive”, “MPICH” deprecated! – A combination of values • Checkpointable, Interactive • Checkpointable, MPI Job. Type = {“MPI”, ”Checkpointable”};

• The Requirements attribute can be used to express constraints on the resources where the job should run. – Its value is a Boolean expression that must evaluate to true for a job to run on that specific CE. • Note: Only one Requirements attribute can be specified (if there are more than one, only the last one is considered). If several conditions must be applied to the job, then they all must be combined in a single Requirements attribute. • For example, let us suppose that the user wants to run on a CE using PBS as batch system, and whose WNs have at least two CPUs. He will write then in the job description file: Requirements = other. Glue. CEInfo. LRMSType == "PBS" && other. Glue. CEInfo. Total. CPUs > 1;

• The WMS can be also asked to send a job to a particular queue in a CE with the following expression: Requirements = other. Glue. CEUnique. ID == "lxshare 0286. cern. ch: 2119/jobmanager-pbs-euasia"; Requirements = other. Glue. CEInfo. Host. Name == "lxshare 0286. cern. ch"; • It is also possible to use regular expressions when expressing a requirement. – Let us suppose for example that the user wants all his jobs to run on any CE in the domain cern. ch. This can be achieved putting in the JDL file the following expression: Requirements = Reg. Exp("cern. ch", other. Glue. CEUnique. ID); – The opposite can be required by using: Requirements = (!Reg. Exp("cern. ch", other. Glue. CEUnique. ID));

• If the job duration is significant, it is strongly advised to put a requirement on the maximum CPU time, or the wallclock time (expressed in minutes), needed for the job to complete. – For example, to express the fact that the job needs at least 8 CPU hours Requirements = other. Glue. CEPolicy. Max. CPUTime > 480; Note: if a job exceeds the time limits of the queue where it is running, it will be killed by the batch system.

Specifying OS and architecture of the CE The user might need to be able to specify which types of CE are to be used to process the job. Requirements = (other. Glue. Host. Operating. System. Name=="Cent. OS" || other. Glue. Host. Operating. System. Name=="Red. Hat. Enterprise. AS“ || other. Glue. Host. Operating. System. Name=="Scientific. SL" || other. Glue. Host. Operating. System. Name=="Scientific. CERNSLC") && (other. Glue. Host. Operating. System. Release >= 4. 0 && other. Glue. Host. Operating. System. Release < 5. 0); or to make the Requirements more readable use … SN = other. Glue. Host. Operating. System. Name; SR = other. Glue. Host. Operating. System. Release; RHEL 4 = (SN=="Cent. OS" || SN=="Red. Hat. Enterprise. AS" || SN=="Scientific. SL" || SN=="Scientific. CERNSLC") && (SR>=4. 0 && SR<5. 0); Requirements = RHEL 4;

Specifying requirements on a close SE In order to specify requirements on the SE “close” to the CE where the job should run use the following expression: Member("castor. pic. es", other. Glue. CESEBind. Group. SEUnique. ID);

• If the job must run on a CE where a particular experiment software is installed and this information is published by the CE, something like the following must be written: Requirements = Member(“BLAST-1. 0. 3", other. Glue. Host. Application. Software. Run. Time. Environment); Note: the Member operator is used to test if its first argument (a scalar value) is a member of its second argument (a list). In fact, the Glue. Host. Application. Software. Run. Time. Environment attribute is a list of strings and is used to publish any VOspecific information relative to the CE (typically, information on the VO software available on that CE).

• It is possible to have the WMS automatically resubmitting jobs which, for some reason, are aborted by the Grid. Two kinds of resubmission are available for the g. Lite 3 WMS: the deep resubmission and the shallow resubmission (only the former is available in the LCG-2 WMS). – The resubmission is deep when the job fails after it has started running on the WN, and shallow otherwise. • The user can limit the number of times the WMS should resubmit a job by using the JDL attributes Retry. Count and Shallow. Retry. Count for the deep and shallow resubmission respectively. • For example, to disable the deep resubmission and limit the attempts of shallow resubmission to 3 : Retry. Count = 0; Shallow. Retry. Count = 3;

• The proxy renewal feature of the WMS is automatically enabled, as long as the user has stored a longterm proxy in the default My. Proxy server (usually defined in the MYPROXY SERVER environment variable. However it is possible to indicate to the WMS a different My. Proxy server in the JDL file: My. Proxy. Server = “myproxy. ct. infn. it";

How does it work ? UI 2. 1. myproxy-init job My. Proxy Server sub mit 4. d ob j 3. le u d e h c s ew i n re x pro y

• The choice of the CE where to execute the job, among all the ones satisfying the requirements, is based on the rank of the CE, a quantity expressed as a floating-point number. The CE with the highest rank is the one selected. – By default, the rank is equal to other. Glue. CEState. Estimated. Response. Time, where the estimated response time is an estimation of the time interval between the job submission and the beginning of the job execution. – Rank = other. Glue. CEState. Free. CPUs; which will rank best the CE with the most free CPUs.

Overview of g. Lite Middleware

Overview • The Workload Management System (WMS) is the g. Lite 3 component that allows users to submit jobs, and performs all tasks required to execute them, without exposing the user to the complexity of the Grid. – It is the responsibility of the user to describe his jobs and their requirements, and to retrieve the output when the jobs are finished. • In the WLCG/EGEE Grid, two different workload management systems are deployed: the legacy LCG-2 system and the new system from the EGEE project, which is an evolution of the former and therefore has more functionalities. • In the following sections, we will describe the basic concepts of the language used to describe a job and the basic command line interface to submit and manage simple jobs.

The Workload Management System (WMS) comprises a set of Grid middleware components responsible for distribution and management of tasks across Grid resources. • The Workload Manager(WM) aims to accept and satisfy requests for job management coming from its clients; – WM will pass the job to an appropriate CE for execution taking into account requirements and the preferences expressed in the job description; – The decision of which resource should be used is the outcome of a matchmaking process. • The Logging and Bookkeeping (L&B) service tracks jobs managed by the g. Lite WMS; – All the important data are fed to L&B internally from various g. Lite middleware components

g. Lite WMS Architecture

g. Lite WMS Architecture Job management requests (submission, cancellation) expressed via a Job Description Language (JDL)

g. Lite WMS Architecture Finds an appropriate CE for each submission request, taking into account job requests and preferences, Grid status, utilization policies on resources

g. Lite WMS Architecture Keeps submission requests Requests are kept for a while if no resources are immediately available

g. Lite WMS Architecture Repository of resource information available to matchmaker Updated via notifications and/or active polling on resources

g. Lite WMS Architecture Performs the actual job submission and monitoring

The WMProxy • The WMProxy is the service responsible to provide access to the WMS functionality through a Web Service Interface • The g. Lite WMProxy Server can be either accessed directly through the published WSDL, the C++ command line interface, or the API • has been designed to efficiently handle a large number of requests for job submission and control to the WMS – it provides additional features such as bulk submission and the support for shared and compressed sandboxes for compound jobs. – It’s the natural replacement of the NS in the passage to the SOA approach.

The Command Line Interface • The g. Lite WMS implements two different services to manage jobs: the Network Server and the WMProxy. – The recommended method to manage jobs is through the g. Lite WMS via WMProxy, because it gives the best performance and allows to use the most advanced functionalities • The WMProxy implements several functionalities, among which: – submission of job collections; – faster authentication; – faster match-making; – faster response time for users; – higher job throughput.

Delegating a proxy to WMProxy • Each job submitted to WMProxy must be associated to a proxy credential previously delegated by the owner of the job to the WMProxy server. – This proxy is then used any time WMProxy needs to interact with other services for job related operations (e. g. submission to the CE, a Grid. FTP file transfer etc. ) – There are two possible mechanisms to ask for a delegation of the user credentails: • asking the “automatic” delegation of the credentials during the submission operation • asking for an “explicit“ delegation

• To explicitly delegate a user proxy to WMProxy, the command to use is: glite-wms-job-delegate-proxy -d <deleg. ID> where <deleg. ID> is a string chosen by the user. For example, to delegate a proxy: $ glite-wms-job-delegate-proxy -d mydeleg. ID Connecting to the service https: //rb 102. cern. ch: 7443/glite_wms_wmproxy_server ======= glite-wms-job-delegate-proxy Success ==== Your proxy has been successfully delegated to the WMProxy: https: //rb 102. cern. ch: 7443/glite_wms_wmproxy_server with the delegation identifier: mydeleg. ID ===========================

Submitting a simple job • Starting from a simple JDL file, we can submit it via WMProxy by doing: $ glite-wms-job-submit –d mydeleg. ID test. jdl Connecting to the service https: //rb 102. cern. ch: 7443/glite_wms_wmproxy_server ==== glite-wms-job-submit Success ==== The job has been successfully submitted to the WMProxy Your job identifier is: https: //rb 102. cern. ch: 9000/v. ZKKk 3 gd. Bla 6 Ry. Sximq_v. Q =======================

Troubleshooting • To submit jobs via WMProxy, it is required to have a valid VOMS proxy, otherwise the submission will fail with an error like: Error - Operation failed Unable to delegate the credential to the endpoint: https: //rb 102. cern. ch: 7443/glite_wms_wmproxy_server User not authorized: unable to check credential permission (/opt/glite/etc/glite_wms_wmproxy. gacl) (credential entry not found) credential type: person input dn: /C=CH/O=CERN/OU=GRID/CN=John Doe

Authorization • The client must be properly authorized when interacts with the WMProxy service. • This means that either the FQAN or the DN (in case of globus-style proxies) of the client must be properly listed and authorized in the glite_wms_wmproxy. gacl file on the WMProxy machine. $ cat glite_wms_wmproxy. gacl <gacl version='0. 0. 1'> <entry <voms><fqan>bio/Role=NULL</fqan></voms> <allow><exec/></allow> </entry> </gacl>

Listing CE(s) that matching a job It is possible to see which CEs are eligible to run a job described by a given JDL using: $ glite-wms-job-list-match –d mydeleg. ID test. jdl Connecting to the service https: //rb 102. cern. ch: 7443/glite_wms_wmproxy_server ========================== COMPUTING ELEMENT IDs LIST The following CE(s) matching your job requirements have been found: *CEId* - CE. pakgrid. org. pk: 2119/jobmanager-lcgpbs-cms - grid-ce 0. desy. de: 2119/jobmanager-lcgpbs-cms - gw-2. ccc. ucl. ac. uk: 2119/jobmanager-sge-default - grid-ce 2. desy. de: 2119/jobmanager-lcgpbs-cms ==========================

Retrieving the status of a job $ glite-wms-job-status https: //rb 102. cern. ch: 9000/f. Nd. D 4 FW_Xxkt 2 s 2 a. ZJeoeg ******************************* BOOKKEEPING INFORMATION: Status info for the Job : https: //rb 102. cern. ch: 9000/f. Nd. D 4 FW_Xxkt 2 s 2 a. ZJeoeg Current Status: Done (Success) Exit code: 0 Status Reason: Job terminated successfully Destination: ce 1. inrne. bas. bg: 2119/jobmanager-lcgpbs-cms Submitted: Mon Dec 4 15: 05: 43 2006 CET ******************************* The verbosity level controls the amount of information provided. The value of the -v option ranges from 0 to 3. The commands to get the job status can have several job. IDs as arguments, i. e. : glite-wms-job-status <job. ID 1>. . . or, more conveniently, the -i <file path> option can be used to

Retrieving the output(s) $ glite-wms-job-output https: //rb 102. cern. ch: 9000/yabp 72 a. ERhof. LA 6 W 2 -Lr. Jw Connecting to the service https: //128. 142. 160. 93: 7443/glite_wms_wmproxy_server =========================== JOB GET OUTPUT OUTCOME Output sandbox files for the job: https: //rb 102. cern. ch: 9000/yabp 72 a. ERhof. LA 6 W 2 -Lr. Jw have been successfully retrieved and stored in the directory: /tmp/doe_yabp 72 a. ERhof. LA 6 W 2 -Lr. Jw =========================== The default location for storing the outputs (normally /tmp) is defined in the UI configuration, but it is possible to specify in which directory to save the output using the --dir <path name> option.

Cancelling a job $ glite-wms-job-cancel https: //rb 102. cern. ch: 9000/P 1 c 60 RFsr. IZ 9 mn. BALa 7 y. ZA Are you sure you want to remove specified job(s) [y/n]y : y Connecting to the service https: //128. 142. 160. 93: 7443/glite_wms_wmproxy_server ===== glite-wms-job-cancel Success ====== The cancellation request has been successfully submitted for the following job(s): - https: //rb 102. cern. ch: 9000/P 1 c 60 RFsr. IZ 9 mn. BALa 7 y. ZA ========================== If the cancellation is successful, the job will terminate in status CANCELLED

Possible job status

Real Time Output Retrieval • The user can enable the job perusal by setting the attribute Perusal. File. Enable to true in the job JDL. • This makes the WN to upload, at regular time intervals (defined by the Perusal. Time. Interval attribute and expressed in seconds), a copy of the output files specified using the glite-wms-job-perusal command to the WMS machine (by default), or to a Grid. FTP server specified by the attribute Perusal. Files. Dest. URI Executable = "job. sh"; Std. Output = "stdout. log"; Std. Error = "stderr. log"; Input. Sandbox = {"job. sh"}; Output. Sandbox = {"stdout. log", "stderr. log", "testfile. txt"}; Perusal. File. Enable = true; Perusal. Time. Interval = 30; Retry. Count = 0;

After the job has been submitted with glite-wms-job-submit, the user can choose which output files should be inspected: $ glite-wms-job-perusal --set -f testfile. txt https: //wms 104. cern. ch: 9000/B 02 x. R 3 EQg 9 ZHHo. Rc-1 n. Jk. Q Connecting to the service https: //128. 142. 160. 93: 7443/glite_wms_wmproxy_server ====== glite-wms-job-perusal Success ======== Files perusal has been successfully enabled for the job: https: //wms 104. cern. ch: 9000/B 02 x. R 3 EQg 9 ZHHo. Rc-1 n. Jk. Q ==============================

. . and, when the job starts, the user can see one output file: $ glite-wms-job-perusal --get -f testfile. txt https: //wms 104. cern. ch: 9000/B 02 x. R 3 EQg 9 ZHHo. Rc-1 n. Jk. Q Connecting to the service https: //137. 138. 45. 79: 7443/glite_wms_wmproxy_server ====== glite-wms-job-perusal Success ======= The retrieved files have been successfully stored in: /tmp/doe_Oo. DVm. WCAnhx_Hi. SPv. ASGsg ============================

References WMProxy User’s guide – https: //edms. cern. ch/file/674643/1/EGEE-JRA 1 TEC-674643 -WMPROXY-guide-v 0 -2. pdf JDL Attributes Specification – https: //edms. cern. ch/file/555796/1/EGEE-JRA 1 TEC-555796 -JDL-Attributes-v 0 -8. pdf – https: //edms. cern. ch/file/590869/1/EGEE-JRA 1 TEC-590869 -JDL-Attributes-v 0 -8. pdf g. Lite User’s guide – https: //edms. cern. ch/file/722398/1. 2/g. Lite-3 User. Guide. pdf

Hands-on • Connect to the training infrastructure using the information reported in the tutorial sheet • Run the hands-on available in this web link: http: //www. euasiagrid. org/wiki/index. php/GLite_Job_Ma nagement • Don’t forget to initialize the proxy before to start. • Enjoy!