CMS Commissioning and First Data CMS Software Installation

CMS Commissioning and First Data CMS Software Installation Bockjoo Kim Stan Durkin U of Florida The Ohio State University Bockjoo Kim for the CMS Collaboration U of Florida CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T 3 Workshop, Fermilab 1

CMSSW Installation Sites on OSG CMS Commissioning and First Data 9 T 3 9 T 2 T 2 T 3 (Caltech) T 2 T 3 T 3 T 2 T 3 Stan Durkin The Ohio State University for the CMS Collaboration CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T 3 Workshop, Fermilab T 2 T 3 (FIU) 2

CMSSW on US T 3 Sites(as of 10/21/08) CMS Commissioning and First Data Sitename CE (OSG) T 3_US_Buffalo? ? u 2 -grid. ccr. buffalo. edu/fork T 3_US_Colorado CPU slots CMSSW 2112 2_1_10, … heposg 01. colorado. edu/? 16 2_1_10, … T 3_US_Cornell nys 1. cac. cornell. edu/fork 120 2_1_9, … T 3_US_FIT uscms 1. fltech-grid 3. fit. edu/fork 160 2_1_10, … T 3_US_FIU ? /? T 3_US_FNALLPC ? /? T 3_US_Kansas ? /? T 3_US_Minnesota ? /? T 3_US_Olemiss? ? umiss 001. hep. olemiss. edu/fork 42 2_1_10, … T 3_US_Princeton ? /? T 3_US_Rutgers ? /? T 3_US_TTU antaeus. hpcc. ttu. edu/fork 252 2_1_10, … T 3_US_UCLA saxon. hosted. ats. ucla. edu/fork 44 2_1_10, … T 3_US_UCR top. ucr. edu/condor 40 2_1_10, … T 3_US_UIowa T 3_US_UMD T 3_US_Vanderbilt Stan Durkin ? /? The Ohio State University ? /? vampire. accre. vanderbilt. edu/for the CMS Collaboration 1408 2_1_10, … k CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T 3 Workshop, Fermilab 3

Centralized CMS Installation CMS Commissioning and First Data v CMS requires CMSSW to be installed centrally for T 2 v For T 3, one can install it locally or centrally v non-CMS OSG sites, it can be done centrally v All installations are uniform ($OSG_APP/cmssoft/cms/$SCRAM_ARCH/cms/c mssw/CMSSW_X_Y_Z) v Central Installation on OSG done by me Stan Durkin v LCG/EGEE counter part. University (serveral people) is The Ohio State for. LCG/EGEE the CMS Collaboration doing this on CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T 3 Workshop, Fermilab 4

CMS Software Life Cycle Commissioning 1. CMS Software Development and First Data 2. Release Build 3. APT Packaging 4. Tagging Release in XML for Publication 5. Release Announcement 6. Deployment on Grids/Local 7. Release Deprecation Pre-Announcement 8. Tagging for Release Deprecation Release 9. Release Deprecation Announcement Stan Durkin 10. Remove Deprecated Release The Ohio State University for the CMS Collaboration CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T 3 Workshop, Fermilab 5

Features of Software Deployment Tool CMS Commissioning and First Data v Condor-G Job Submission with Customized Installation/Verfication Scripts v Central Run and Bookkeeping DB v Possible Install via Grid Proxy Based Portal: Different DN Can Install Different Site Simul. v Cronized Installation In Parallel per Release v Production CMS Soft Release on OSG T 2/T 3 v Twiki Pages : https: //twiki. cern. ch/twiki/bin/view/CMSSoft. Deploy. OSG Stan Durkin v Portal : https: //dev 01. ihepa. ufl. edu: 8443/csdogrid/ The Ohio State University for the CMS Collaboration CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T 3 Workshop, Fermilab 6

Consideration For Installation CMS Commissioning v Different Linux Flavors and First Data v 64 -bit at 32 -bit mode v Network Insulation (non-CMS sites) v Missing Tools on WN : apt-get, rpmbuild (non. CMS sites) v Different Shared File Systems (Lustre, AFS, etc) v Dedicated Slot : Needs to be done before any other CMS job starts v Disk Space Issue : ~50 GB required typically v Many files ==> Installation time gets longer Stan Durkin v Keeping many releases ==>University Deprecation The Ohio State becomes terrible should be done on time for theand CMS Collaboration CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T 3 Workshop, Fermilab 7

APT and File Systems CMS Commissioning and First Data v APT (RPM tool) uses lots of locks v At least, four different FS accros OSG sites : NFS, AFS, GPFS, lustre v GPFS, lustre, other not well-known FS requires special treatment for locks using local FS ( not many exotic FS, though) v Many files ==> Installation time gets longer v Keeping many releases ==> Deprecation Stan Durkin==> becomes very important The Ohio State University (64 -bit OS/8 GB required in the worst case) for the CMS Collaboration CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T 3 Workshop, Fermilab 8

Deprecation CMS Commissioning and First Data v Mostly same as installation v Most time consuming part is find dependency and dependency bookkeeping v NFS stale file handle and GPFS : ‘rm -rf’ v Recycle dependency calculation from one site v Deprecation is also automated and cronized Stan Durkin The Ohio State University for the CMS Collaboration CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T 3 Workshop, Fermilab 9

CMS Software Packaging Tool CMS Commissioning and First Data v CMS employs APT packaging for software distribution v CMS provides packaging tools for initial setup and update v RPMs are installed in a non-root area v CMS provides release publication : this allows deployment people for immediate Stan Durkin deployment The Ohio State University for the CMS Collaboration CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T 3 Workshop, Fermilab 10

Automated CMS software Deployment CMS Commissioning and First Data v Well-established OSG CMS sites needs software deployment promptly v CRON is used for automation v List of well-established OSG CMS sites v Database(DB) for bookkeeping v Scripts checks and executes: v New release that needs to be deployed v Deployment status from DB v Deployment job submission/resubmission Stan Durkin v Installation job self-monitoring The Ohio State University v Email notification theif CMS Collaboration v Repeatitionfor of all necessary CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T 3 Workshop, Fermilab 11

Implementation of the Deployment Tool CMS X 509 Commissioning and First Data Web Portal Grid. Cat Client Site List Site Catalog CRON Site Availability DB Check My. SQL DB Deploy Script DB Update Job Status Email Grid. Cat Execution Script CMS Grid Users Condor-G Job Local Scripts CMS Pkg Tool OSG Software Application Area Info. Publication RPMs Stan Durkin OSG Site A Remote Execution Script The Ohio State University for the CMS APT Collaboration CMS Pkg Tool RPMs Repository CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T 3 Workshop, Fermilab Tool Design OSG CMS 12

Grid Portal Screen Shot CMS Commissioning and First Data Stan Durkin The Ohio State University for the CMS Collaboration CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T 3 Workshop, Fermilab 13

SAM and SW Installation Monitoring CMS Commissioning and First Data SAM monitors SW Installations Stan Durkin Related SW Installation The Ohio with State University for the CMS Collaboration CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T 3 Workshop, Fermilab 14

List of Problems and Solutions CMS Commissioning and First Data v RPM version mismatch -> Rebuild RPM DB v rpm-wrapper error 88 -> insufficient disk v rpm-wrapper error 92 -> permission problem v“Could not get lock” -> File. System, use local disk v memory alloc (4 byte)-> Remove releases v memory alloc (8 byte) -> Use 64 -bit apt-get Stan Durkin The Ohio State University for the CMS Collaboration CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T 3 Workshop, Fermilab 15

Statistics of CMS SW Deployment CMS Commissioning Years 2006 -and 2008 First Data Installed Releases 2006 0_4_1 TO 1_2_0 2007 1_2_0 TO 1_7_5 2008 1_6_10 TO 2_1_10 27 50 41 118 Sites Installations (Releasesx. Sites) OSG T 2, T 3, and Opportune 33 sites 2006 2007 2008 126 294 297 717 2006 Stan Durkin 2007 Removals The Ohio State 15 University 325 (Releasesx. Sites) for the CMS Collaboration 1024 CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T 3 Workshop, Fermilab 2008 684 16

Timing and Files Site CMS Commissioning and First Data Tier Time to Install* FS No of Files Size (GB) XFS 1. 7 x 106 57 NFS/lustre 1. 3 x 106 48 Caltech 2 54 min Florida 2 868 min MIT 2 66 min NFS 1. 5 x 106 57 Nebraska 2 57 min NFS 1. 5 x 106 54 Purdue 2 52 min NFS 1. 6 x 106 96 SPRACE 2 100 min NFS 1. 7 x 106 53 UCSD 2 98 min NFS 1. 6 x 106 60 UERJ 2 51 min NFS 1. 4 x 106 52 Wisconsin 2 71 min AFS 1. 9 x 106 61 TTU 3 80 min lustre 1. 3 x 106 45 UCR 3 56 min NFS? 1. 5 x 106 54 Mississippi 3 186 min NFS? 1. 6 x 106 55 UCLA 3 48 min NFS? 1. 6 x 106 55 Vanderbilt 3 78 min GPFS 1. 2 x 106 47 Buffalo 3 6 43 FIT 3 6 41 Cornell 3 FIU 3 Stan 45 min Durkin NFS? 1. 1 x 10 763 min NFS? University 1. 1 x 10 The Ohio State for the CMS- Collaboration - - - CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T 3 Workshop, Fermilab 17

Summary CMS Commissioning and First Data v More than 717(1024) installations/removal for CMS software have been deployed on OSG, 2006 -2008 v Automated installation works quite efficiently with almost no problem theses days v Recently, most installation problem comes from rpm db limitation and can be fixed via 64 -bit apt-get v T 3 sites are not required to deploy CMSSW centrally. At the moment, there are 9 sites under the regular/central deploy list. v If other T 3 sites wish be included, please let me Stanto. Durkin know The Ohio State University for the CMS Collaboration CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T 3 Workshop, Fermilab 18