Update on g Lite WMS tests Andrea Sciab

Update on g. Lite WMS tests Andrea Sciabà CERN WLCG-OSG-EGEE Operations meeting September 21, 2006

Testing the g. Lite WMS n n RB installed with g. Lite 3. 0. 2 + various patches Dedicated machine at CERN (rb 102. cern. ch) n n n 2 × Xeon 3. 0 GHz 4 GB of RAM 3 RAID 1 partitions for better I/O performance Closely monitored by GD, FIO and JRA 1 people Tests run by CMS, GD, ATLAS

CMS Test description n Application n Run on CMS Tier-1’s and Tier-2’s Different submission methods n n n n Fake analysis jobs (~30’ of CPU time) Network server WMProxy Bulk submission Submission from 1 -3 UI’s in parallel VOMS proxies Myproxy renewal on Deep resubmission off Shallow resubmission ≤ 3

Latest results (I) n n No. of jobs = 3 UI × 33 CEs × 200 jobs/collection 20000 jobs ~2. 5 hours to submit all jobs n n n ~0. 5 sec/job Submission failed for 6 collections ~17 hours to dispatch all jobs n Equivalent to ~26000 jobs/day

Latest results (II) Site cclcgceli 02. in 2 p 3. fr ce 01 -lcg. cr. cnaf. infn. it ce 01 -lcg. projects. cscs. ch ce 03 -lcg. cr. cnaf. infn. it ce 04. pic. es ce 101. cern. ch ce 102. cern. ch ce 103. cern. ch ce 104. cern. ch ce 105. cern. ch ce 106. cern. ch ceitep. ru cmslcgce. fnal. gov cmsrm-ce 01. roma 1. infn. it dgc-grid-40. brunel. ac. uk egeece. ifca. org. es grid-ce 1. desy. de grid-ce 2. desy. de grid 10. lal. in 2 p 3. fr grid 109. kfki. hu gridba 2. ba. infn. it gridce. iihe. ac. be gridce. pi. infn. it gw 39. hep. ph. ic. ac. uk lcg 00125. grid. sinica. edu. tw lcg 02. ciemat. es lcg 06. sinp. msu. ru lcgce 01. gridpp. rl. ac. uk lcgce 01. jinr. ru polgrid 1. in 2 p 3. fr t 2 -ce-02. lnl. infn. it Submit 0 0 0 0 0 0 0 0 Wait Ready Sched 0 0 0 2 0 0 195 0 0 200 10 0 0 9 0 0 10 0 0 150 0 0 200 0 0 0 0 1 0 0 200 0 0 0 0 9 0 0 180 0 0 86 0 0 200 10 0 12 1 0 34 10 0 0 199 0 0 0 Run Done(S) Done(F) Abo Clear Canc 0 200 0 0 122 0 0 76 0 0 5 0 0 0 0 23 0 0 167 0 200 0 0 200 0 0 1 16 0 174 0 0 66 28 0 96 0 0 0 200 0 0 3 47 0 0 0 200 0 0 0 200 0 190 10 0 0 199 0 0 0 0 200 0 189 0 11 0 0 199 0 0 3 15 0 173 20 0 0 11 103 0 0 0 0 0 2 150 2 0 0 24 11 154 0 0 0 158 0 0 32 0 0 0 0 3 197 0 0 0 200 0 0

Failure reasons n n n Application errors Maradona errors “Got a job held event, reason: "The Periodic. Hold expression 'Matched =!= TRUE && Current. Time > QDate + 900' evaluated to TRUE" ” errors n n Jobs remaining in Waiting status while Pending events are generated every 5 minutes with error n n Mkfifo /tmp/…: File Exists Unspecified gridmanager error n n The WMS could not submit the job to a g. Lite CE Normally a batch system problem Shallow resubmission often recovers, but if the error happens again, the job is aborted (but sometimes appears as Cancelled) Authentication failed with Belgian CE (CRL expired) Negligible fractions of other errors n n n Could not upload a sandbox file Got a job held event, reason: Globus error 124: old job manager is still alive Gatekeeper unreachable

Efficiency table (I) CE cclcgceli 02. in 2 p 3. fr ce 01 -lcg. cr. cnaf. infn. it ce 01 -lcg. projects. cscs. ch ce 03 -lcg. cr. cnaf. infn. it Efficiency Main failure reason 1 0. 61 Application error 1 0. 98 Application error ce 04. pic. es 1 ce 101. cern. ch 0 Application error ce 102. cern. ch 0 Application error ce 105. cern. ch 0 Application error ce 106. cern. ch 0 Application error ceitep. ru 1 cmslcgce. fnal. gov 1 cmsrm-ce 01. roma 1. infn. it 1 dgc-grid-40. brunel. ac. uk 0 egeece. ifca. org. es 0. 95 grid-ce 0. desy. de 1 grid-ce 1. desy. de 1 grid-ce 2. desy. de 1 Application error Gatekeeper down

Efficiency table (II) CE grid 10. lal. in 2 p 3. fr Efficiency Main failure reason 0 Application error 0. 94 Application error gridba 2. ba. infn. it 0 Application error gridce. iihe. ac. be 0 CRL expired gridce. pi. infn. it 1 gw 39. hep. ph. ic. ac. uk 1 lcg 00125. grid. sinica. edu. tw 1 grid 109. kfki. hu lcg 02. ciemat. es 0. 82 Unspecified gridmanager error lcg 06. sinp. msu. ru 0. 99 Waiting (mkfifo error) lcgce 01. gridpp. rl. ac. uk 0 lcgce 01. jinr. ru 1 polgrid 1. in 2 p 3. fr 0 t 2 -ce-02. lnl. infn. it 1 Unspecified gridmanager error Application error

Conclusions n Very small fraction of failed jobs due to the WMS n n n Performance seems to indicate a maximum rate of ~26000 jobs/day n n Only those remaining in Waiting status (O(100)) All other failures are due either to the application, to the CE or to authentication problems (expired CRL) “Job Robot” jobs, it may be different for other kinds of jobs The WMS looks reasonably fine now