SRM TEST SUITE FOR LCG2 Elena Slabospitskaya Institute

SRM TEST SUITE FOR LCG-2 Elena Slabospitskaya Institute of High Energy Physics Protvino, RUSSIA 18. 04. 2004

SRM test suite for LCG-2 Objectives: Test Suite for testing of SRM (Storage Resource Manager) interface v. 1. 1 with multiple data files into one SRM request (1 -99). Requirements: The Test Suite requires a grid servers set (UI, RM, default classic SE and SRM server). Static config files for the Replica Manager can point to the SRM servers to be used. Goal is to test all SRM methods defined in SRM v 1. 1, including multi-file requests

SRM test suite for LCG-2 The SRM test suite consists of 2 programs. The perl script is preparatory for the C-program. 1) A script in Perl, which generates data files and their replicas. File'names and sizes (1 -10 Mb) generated with a random number generator. The script creates the specified number of files in the directory srm_log by "dd". These files are copied to the local SE and registered (published). Replicas of files are made to the specified SRM. All replica names are written to the srm_log/sfns file and are used by the C-program.

SRM test suite for LCG-2 The second part of the test suite is a program written in C language. The first test programs (srm_test. Ping, srm_testget. Protocols, srm_testget. Meta. Data, srm_test. Put and srm_test. Get) had been written by Jean-Philippe Baud (baud@mail. cern. ch). All basic SRM operations are tested - get, put, Pin, Un. Pin, Mk. Permanent, set. File. Status, get. Est. Get. Time, get. Est. Put. Time, get. File. Meta. Data, get. Protocol s, Advisory. Delete.

SRM test suite for LCG-2 This program tests all SRM functions. SRMs do not perform file movement operations, but invoke middleware components (f. ex. gridftp) to perform theirs. SRM allows pinning files, releasing files, and allocating space dynamically on an "asneeded" basis. So, the C-program concentrates on put/get couple of operations.

SRM test suite for LCG-2 The main algorithm is: srm_put -- Reserve space before the file transfer (srm_put) --mk. Permanent -- set Files Status to 'Running' -- A data files put to SRM server by helps "globus-url-copy" -- set Files Status to 'Done' srm_get --pin file -- set Files Status to 'Running' --A data files get from SRM server to UI -- set Files Status to 'Done' --unpin files -- compare files before and after transfer by "diff" delete -- The program try to delete files by helps Advisory. Delete SRM function and"edg-gridftp-rm"

SRM test suite for LCG-2 The example of put code is: Srm_init(&soap); srm_ping. Response(&soap, argv[1], fp); srm_get. Protocols(&soap, argv[1], fp); printf("========THE FILE IS PUTTING VIA SRM========<n"); fprintf(fp, "======THE FILE IS PUTTING VIA SRM========<n"); for (i=0; i < fnum; i++) srm_Is. File. Caching(&soap, argv[1], pargs+i, fp); srm_get. Est. Put. Time(&soap, argv[1], pargs, fp, fnum); req. Id_put= srm_put(&soap, argv[1], pargs, fp, fnum); if (srm_Is. File. Permanent(&soap, argv[1], req. Id_put, fp, fnum) == 0) srm_mk. Permanent(&soap, argv[1], req. Id_put, pargs, fp, fnum); file. Transfer. PUT(fnames, fnum, fp, &soap, argv[1], req. Id_put); NOTE: function file. Transfer. PUT is including set. File. Status(Running) and set. File. Status(Done)

SRM test suite for LCG-2 The example of get code is: Printf("=======THE FILE IS GETTING VIA SRM========n"); fprintf(fp, "=======THE FILE IS GETTING VIASRM=========n"); for (i=0; i < fnum; i++) srm_Is. File. Caching(&soap, argv[1], &pargs[i], fp); req. Id_get= srm_get(&soap, argv[1], pargs, fp, fnum); srm_get. Est. Get. Time(&soap, argv[1], pargs, fp, fnum); if ( srm_Is. File. Pinning(&soap, argv[1], req. Id_get, fp, fnum) == 0) srm_pin(&soap, argv[1], req. Id_get, pargs, fp, fnum); file. Transfer. GET(fnames, fnum, fp, &soap, argv[1], req. Id_get); if (srm_Is. File. Pinning(&soap, argv[1], req. Id_get, fp, fnum)) srm_un. Pin(&soap, argv[1], req. Id_get, pargs, fp, fnum); srm_advisory. Delete(&soap, argv[1], pargs, fp, fnum); file. Delete(fnum, fp, argv[1]); srm_end(&soap); NOTE: function file. Transfer. GET is including set. File. Status(Running) and set. File. Status(Done)

SRM test suite for LCG-2 1) THE USAGE. /srm_SURLs. pl or OF Perl -script. /srm_SURLs-dcache. pl The script prepare replicas for srm test suite. Results story in file 'srm_log/sfns' Usage: srm_SURLs. pl [-help| -h] srm_SURLs. pl surls host Where: surls- are the number of creating files (and their replicas) host - is the name of the SRM server Example: . /srm_SURLs. pl 2 wacdr 002 d. cern. ch A dcache-server has not made replicate. File yet. A variant of perl script is -----------srm_SURLs-dcache. pl. now It made copy. And. Register of the file to SRM-server right

SRM test suite for LCG-2 The example of the Perl code $guid= `edg-rm --vo=dteam --config=$conf copy. And. Register. File file: . $file -l $lfn`; if (!$guid) { print "FAIL: Publish a file at grid on local SEn"; return; } print "OK: Publish a file at grid on local SEn ", $guid, "n"; $surl=`edg-rm --vo=dteam --config=$conf replicate. File --destinationfile $host $lfn`; if (!$surl) {print "FAIL: Don't replicate a file on $hostn"; `edg-rm --vo=dteam delete. File $lfn`; return; } print "OK: Replicate a file on $hostn ", $surl, "n"; chomp($surl); print SFN $surl, "n"; `edg-rm --vo=dteam delete. File $lfn`;

SRM test suite for LCG-2 The example of Perl script' stdout. /srm_SURLs. pl 4 wacdr 002 d. cern. ch. Preparing of file srms. CQgq 6 t 8 Xl 9 Ou 4 Hd 7 Tx 9 U 4 MMac 032 Bv 6 DPf 8 Ro with a size 3001 Kb Preparing of file srms. NXdo 1 l 9 Gj 6 Tu 5 Og 6 He 8 O 3 SKzk 0 Q 3 Ht 8 XWl 9 Je with a size 5468 Kb Preparing of file srms. RZep 6 q 3 Kt 6 Gt 6 Cy 1 Lw 7 A 3 PSmr 6 P 4 Ed 8 NVq 2 Vq with a size 6138 Kb Preparing of file srms. DXhl 7 G 2 Kt 2 Dc 2 Lm 3 Sj 6 L 2 TBmq 1 K 4 Kr 2 SPz 8 Jt with a size 9468 Kb ----------------/afs/cern. ch/user/l/lspitsky/public/srm_log/srms. CQgq 6 t 8 Xl 9 Ou 4 Hd 7 Tx 9 U 4 MMac 032 Bv 6 DPf 8 Ro lfn: srm_log/srms. CQgq 6 t 8 Xl 9 Ou 4 Hd 7 Tx 9 U 4 MMac 032 Bv 6 DPf 8 Ro OK: Publish a file at grid on local SE guid: 283 c 4606 -7838 -11 d 8 -ba 43 -86 e 4740 e 9525 OK: Replicate a file on wacdr 002 d. cern. ch srm: //wacdr 002 d. cern. ch: 8443/castor/cern. ch/grid/dteam/generated/2004 -03 -17/ file 2 e 03017 f-7838 -11 d 8 -91 fc-e 022371 e 3 b 67 The storage is mandatory for LFN lfn: srm_log/srms. CQgq 6 t 8 Xl 9 Ou 4 Hd 7 Tx 9 U 4 MMac 032 Bv 6 DPf 8 Ro ----------------/afs/cern. ch/user/l/lspitsky/public/srm_log/srms. NXdo 1 l 9 Gj 6 Tu 5 Og 6 He 8 O 3 SKzk 0 Q 3 Ht 8 XWl 9 Je lfn: srm_log/srms. NXdo 1 l 9 Gj 6 Tu 5 Og 6 He 8 O 3 SKzk 0 Q 3 Ht 8 XWl 9 Je OK: Publish a file at grid on local SE guid: 4 bfeefa 6 -7838 -11 d 8 -96 b 4 -df 8 f 88996750. . and so on. .

SRM test suite for LCG-2 THE RESULTS WERE STORIED at directory srm_log for example: ls -al. . lspitsky zp 2048 Mar 17 20: 08. lspitsky 44 2048 Mar 17 20: 05. . lspitsky zp 351 Mar 17 20: 08 sfns -lspitsky zp 5136384 Mar 17 20: 05 srms. ACqf 1 h 5 Mi 3 Kj 5 Vp 9 Eq 3 F 6 QOwm 0 W 1 At 1 BFh 5 Aq lspitsky zp 1126400 Mar 17 20: 05 srms. BKrt 8 u 2 Ck 0 Xk 1 Zt 3 Cx 7 N 8 HDik 5 f 9 Ya 8 MHa 1 Rr lspitsky zp 6520832 Mar 17 20: 05 srms. QNey 980 Cp 2 Bv 9 Bf 8 Aq 0 Y 8 FNgt 7 e 6 Gf 9 EQi 6 Mh 2) The USAGE of C-Program /srm_lib. Full Usage: . /srm_lib. Full srm_endpoint F. ex. . /srm_lib. Full http: //lxshare 0229. cern. ch: 8443/srm/managerv 1

SRM test suite for LCG-2 The example 1/1 of srm_lib. Full log file L lxshare 0285%. /srm_lib. Full http: //wacdr 002 d. cern. ch: 8443/srm/managerv 1. SRM TESTS ARE STARTING: srm_endpoint is http: //wacdr 002 d. cern. ch: 8443/srm/managerv 1. SURLs are in the file sfns, the quantity of SURLs are 3 ===Transfer'list consist of files: srms. QNey 980 Cp 2 Bv 9 Bf 8 Aq 0 Y 8 FNgt 7 e 6 Gf 9 EQi 6 Mh srms. BKrt 8 u 2 Ck 0 Xk 1 Zt 3 Cx 7 N 8 HDik 5 f 9 Ya 8 MHa 1 Rr srms. ACqf 1 h 5 Mi 3 Kj 5 Vp 9 Eq 3 F 6 QOwm 0 W 1 At 1 BFh 5 Aq ===SURL list consist of replicas: srm: //wacdr 002 d. cern. ch: 8443/castor/cern. ch/grid/dteam/generated/2004 -03 -17/file 1 bab 899 d-7846 -11 d 8 -b 5 feabed 85 e 38 d 9 a srm: //wacdr 002 d. cern. ch: 8443/castor/cern. ch/grid/dteam/generated/2004 -03 -17/file 3853815 b-7846 -11 d 8 -be 78 ddee 591 cf 72 c srm: //wacdr 002 d. cern. ch: 8443/castor/cern. ch/grid/dteam/generated/2004 -03 -17/file 5 bed 1 f 13 -7846 -11 d 8 -9 ff 2 dd 19 c 4322 a 94 OK: soap_call_tns__ping: SRM server is alive OK: soap_call_tns__get. Protocols: Supported protocols are: rfio gsiftp ========THE FILE IS PUTTING VIA SRM========< INFO: The file srm: //wacdr 002 d. cern. ch: 8443/castor/cern. ch/grid/dteam/generated/2004 -03 -17/file 1 bab 899 d-7846 -11 d 8 b 5 fe-abed 85 e 38 d 9 a PINNED INFO: The file srm: //wacdr 002 d. cern. ch: 8443/castor/cern. ch/grid/dteam/generated/2004 -03 -17/file 1 bab 899 d-7846 -11 d 8 b 5 fe-abed 85 e 38 d 9 a WAS permanent INFO: srm_Is. File. Caching: The file srm: //wacdr 002 d. cern. ch: 8443/castor/cern. ch/grid/dteam/generated/2004 -0317/file 1 bab 899 d-7846 -11 d 8 -b 5 fe-abed 85 e 38 d 9 a is caching. . . . and so on. . . .

SRM test suite for LCG-2 The example 1/2 of srm_lib. Full log file L. . K: srm_put: file. ID= 0, state = Ready, TURL = rfio: //pub 003 d//shift/pub 003 d/data 03/cg/stage/file 1 bab 899 d-784611 d 8 -b 5 fe-abed 85 e 38 d 9 a. 744627 OK: srm_put: file. ID= 1, state = Ready, TURL = rfio: //pub 001 d//shift/pub 001 d/data 09/cg/stage/file 3853815 b-784611 d 8 -be 78 -ddee 591 cf 72 c. 744628 OK: srm_put: file. ID= 2, state = Ready, TURL = rfio: //pub 002 d//shift/pub 002 d/data 05/cg/stage/file 5 bed 1 f 13 -784611 d 8 -9 ff 2 -dd 19 c 4322 a 94. 744629 INFO: srm_Is. File. Permanent: File. Id 0 has permanent at the moment INFO: srm_Is. File. Permanent: File. Id 1 has permanent at the moment INFO: srm_Is. File. Permanent: File. Id 2 has permanent at the moment STUB: soap_call_tns__mk. Permanent: Protocol not supported INFO: srm_Is. File. Pinning: file. Id 0 has pinned at the moment INFO: srm_Is. File. Pinning: file. Id 1 has pinned at the moment INFO: srm_Is. File. Pinning: file. Id 2 has pinned at the moment STUB: soap_call_tns__pin: Protocol not supported *****file. Id= 0. Transferring. . . ***** OK: srm_update. File. Status: Status of file. Id 0 have changed from: Ready to Running INFO: /opt/globus/bin/globus-url-copy file: //afs/cern. ch/user/l/lspitsky/public/srm_log/srms. QNey 980 Cp 2 Bv 9 Bf 8 Aq 0 Y 8 FNgt 7 e 6 Gf 9 EQi 6 Mh gsiftp: //wacdr 002 d. cern. ch/pub 003 d//shift/pub 003 d/data 03/cg/stage/file 1 bab 899 d-7846 -11 d 8 -b 5 feabed 85 e 38 d 9 a. 744627 error: the server sent an error response: 553 Could not determine cwdir: No such file or directory. OK: srm_update. File. Status: Status of file. Id 0 have changed from: Running to Done *****file. Id= 1. Transferring. . . ***** OK: srm_update. File. Status: Status of file. Id 1 have changed from: Ready to Running INFO: /opt/globus/bin/globus-url-copy file: //afs/cern. ch/user/l/lspitsky/public/srm_log/srms. BKrt 8 u 2 Ck 0 Xk 1 Zt 3 Cx 7 N 8 HDik 5 f 9 Ya 8 MHa 1 Rr gsiftp: //wacdr 002 d. cern. ch/pub 001 d//shift/pub 001 d/data 09/cg/stage/file 3853815 b-7846 -11 d 8 -be 78 -

SRM test suite for LCG-2 The example 3/1 of srm_lib. Full log file L lxshare 0285%. /srm_lib. Full http: //lxshare 0229. cern. ch: 8443/srm/managerv 1. SRM TESTS ARE STARTING: srm_endpoint is. http: //lxshare 0229. cern. ch: 8443/srm/managerv 1 SURLs are in the file sfns, the quantity of SURLs are 1 ===Transfer'list consist of files: srms. PPej 9 m 4 Ct 7 Wl 4 Fw 8 Jf 4 C 3 GJfl 609 Ra 8 LGa 8 Kt ===SURL list consist of replicas: srm: //lxshare 0229. cern. ch: 8443/dteam/generated/2004 -03 -08/file 3 a 07 b 97 f-711 b-11 d 8 -ab 1 df 4776 fe 48 b 24 OK: soap_call_tns__ping: SRM server is alive OK: soap_call_tns__get. Protocols: Supported protocols are: gsiftp dcap ========THE FILE IS PUTTING VIA SRM========< SOAP FAULT: SOAP-ENV: Server "could not get storage info by path : Cache. Exception(rc=666; msg=Pnfs error : can't get pnfs. Id (not a pnfsfile))" Detail: <stacktrace>java. lang. Runtime. Exception: could not get storage info by path : Cache. Exception(rc=666; msg=Pnfs error : can't get pnfs. Id (not a pnfsfile)) at disk. Cache. V 111. srm. SRMCell. get. File. Meta. Data(SRMCell. java: 1710)

SRM test suite for LCG-2 The example 3/2 of srm_lib. Full log file L STUB: soap_call_tns__get. Est. Put. Time: Protocol not supported. soap_call_tns__PUT returned reqid: 30 request state Pending FAIL: srm_put: error. Message: all file reqs are ready OK: srm_put: file. ID= 0, state = Ready, TURL = dcap: //lxshare 0229. cern. ch: 22125//pnfs/cern. ch/data/dteam/generated/2004 -0308/file 3 a 07 b 97 f-711 b-11 d 8 -ab 1 d-f 4776 fe 48 b 24 INFO: srm_Is. File. Permanent: File. Id 0 has NOT permanent now STUB: soap_call_tns__mk. Permanent: Protocol not supported INFO: srm_Is. File. Pinning: file. Id 0 has NOT pinned now *****file. Id= 0. Transferring. . . ***** FAIL: soap_call_tns__set. File. Status: Protocol not supported INFO: /opt/globus/bin/globus-url-copy file: //afs/cern. ch/user/l/lspitsky/Cern 2004/SRM_TESTS/srm_log/srms. PPej 9 m 4 Ct 7 Wl 4 Fw 8 Jf 4 C 3 GJfl 609 Ra 8 LGa 8 Kt gsiftp: //lxshare 0229. cern. ch: 22125//pnfs/cern. ch/data/dteam/generated/2004 -03 -08/file 3 a 07 b 97 f-711 b-11 d 8 ab 1 d-f 4776 fe 48 b 24 error: the server sent an error response: 553 /pnfs/cern. ch/data/dteam//lxshare 0229. cern. ch: 22125//pnfs/cern. ch/data/dteam/generated/2004 -03 -08/file 3 a 07 b 97 f-711 b-11 d 8 ab 1 d-f 4776 fe 48 b 24: No such file or directory. : /pnfs/cern. ch/data/dteam/lxshare 0229. cern. ch: 22125/pnfs/cern. ch/data/dteam/generated/2004 -03 -08 . OK: srm_update. File. Status: Status of file. Id 0 have changed from: Ready to Done

SRM test suite for LCG-2 The example 3/2 of srm_lib. Full log file L soap_call_tns__GET returned reqid: 31 . request state Pending . FAIL: srm_get: error. Message: Get. Storage. Info. Failed : Pnfs. Get. Storage. Info. Message. get. Return. Code () != 0 FAIL: state = Failed FAIL: srm_get: file. ID= 0, state = Failed, TURL = (null) STUB: soap_call_tns__get. Est. Get. Time: Protocol not supported INFO: srm_Is. File. Permanent: File. Id 0 has NOT permanent now STUB: soap_call_tns__mk. Permanent: Protocol not supported INFO: srm_Is. File. Pinning: file. Id 0 has NOT pinned now STUB: soap_call_tns__pin: Protocol not supported FAIL: file. Transfer. GET: TURL of file. Id=0 is (null). Did not launch srm_update. File. Status INFO: srm_Is. File. Pinning: file. Id 0 has NOT pinned now STUB: soap_call_tns__un. Pin: Protocol not supported OK: Advisory Delete of srm: //lxshare 0229. cern. ch: 8443/dteam/generated/2004 -03 -08/file 3 a 07 b 97 f-711 b-11 d 8 -ab 1 df 4776 fe 48 b 24 INFO: file. Delete: /opt/edg/bin/edg-gridftp-rm gsiftp: //lxshare 0229. cern. ch: 22125//pnfs/cern. ch/data/dteam/generated/2004 -03 -08/file 3 a 07 b 97 f-711 b 11 d 8 -ab 1 d-f 4776 fe 48 b 24 error the server sent an error response: 553 Permission denied, reason: Cache. Exception(rc=666; msg=Pnfs error : can't get pnfs. Id (not a pnfsfile))

SRM test suite for LCG-2 Conclusions for dcache' server L 1. Functions are working : . soap_call_tns__ping soap_call_tns__get. Protocols soap_call_tns__put soap_call_tns__set. File. Status (' Done') 2. Don't working soap_call_tns__get. Metafile. Data (Error: 'Java exception') soap_call_tns__get ( Error: 'Get. Storage. Info. Failed : Pnfs. Get. Storage. Info. Message. get. Return. Code () != 0' ) soap_call_tns__set. File. Status (' Running') 3. Stubs: soap_call_tns__get. Est. Get. Time soap_call_tns__get. Est. Put. Time soap_call_tns__pin, Unpin soap_call_tns__mk. Permanent