Open Science Grid org Installation and Configuration of

Open. Science. Grid. org Installation and Configuration of OSG Resources

Open. Science. Grid. org Outline of the Talk Open. Science. Grid. org OSG Resources Open. Science. Grid. org The OSG Software Open. Science. Grid. org Installation and Configuration 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 2

Open. Science. Grid. org OSG Resources

Open. Science. Grid. org Resources on Grid 3/OSG • A variety of resources types GRID 3 Instance – Large facilities – Small clusters at Tier 3 s • Fabric management – Many different technologies used • Large sites tend to use vendor specific technologies • But not always • Difficult to categorize – ROCKS used on US-LHC facilities • Different facilities use it differently • UFlorida, FNAL significantly modify the standard ROCKS configuration • Others use vanilla installations 9/18/2020 PDSF: facility with ~400 opportunistic CPUs, uses kickstart installation with cfengine + home grown… UFlorida: A CMS Tier 2 site, uses ROCKS configured for local usage U of Chicago: A ATLAS Tier 2 site, has one IBM cluster is managed with x. CAT 2 nd LCG Operations Workshop: Jorge L. Rodriguez 4

Open. Science. Grid. org Ex. ROCKS @ UFlorida Tier 2 ROCKS is the cluster management system from SDSC it is not a ROC “Reg. Ops. Center” • UFlorida is A US CMS Tier 2 facility • Standard ROCKS configuration is significantly modified – A single management frontend defines entire cluster – Support multiple distributions (RH 7. 3, RHEL 3, SL 3) • Creation of new clusters and service nodes simple & OO systems administration – Add to and modify ROCKS xml specification files – Manipulate the ROCKS My. SQL database 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 5

Open. Science. Grid. org More on the ROCKS cluster management system? I have another 10 slides on this if there is enough interest

Open. Science. Grid. org The OSG Software First version 0. 2. x

Open. Science. Grid. org The OSG 0. 2. x Software Cache • The cache is predominantly VDT 1. 3. 5 (6) • Plus OSG specific configuration packages • Plus OSG specific software not included in VDT http: //software. grid. iu. edu/pacman = OSG cache 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 8

Open. Science. Grid. org OSG Software Installation • All of the OSG software is packaged in a set of OSG packages from a versioned OSG cache – Services are selected from said package – The admin chooses where and what to deploy • We use pacman as met-packager – Ideally: “pacman – get OSG: OSG-CE-0. 2. x” and you’re done – But post and pre configuration steps are required • Some setup & configuration of the MIS… needed • Significant pre installation steps may be required for new authorization system (i. e. pool accounts, GUMS servers…) – Instructions are posted on various OSG twiki webs… http: //osg. ivdgl. org/twiki/bin/view 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 9

Open. Science. Grid. org The VDT in OSG 0. 2. x VDT 1. 3. 5 (6) major packages • Globus Toolkit: 3. 2. 1 pre-web release • Condor: vers 6. 7. 6 • Privilege: – PRIMA: Authorization Module, v 0. 2 – GUMS: Grid User Management System, v 1. 0. 1 – VOMS: v 1. 4. 0 and VOMS Admin Server, v 0. 7. 6 • Mon. ALISA: monitoring vers v 1. 2. 32 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 10

Open. Science. Grid. org The VDT in OSG 0. 2. x cont • GLUE and GIP: – 1. 2 vers. was added to the VDT yesterday – Next we will work to integrate into ITB testbed • Grid. FTP: New (GT 4) v 3. 9. 5 -2005 -03 -29 • Certificate and proxy management – DOE, LCG CA certificates, crl management, PPDG scripts … – KX 509, My. Proxy • Plus more, see http: //www. cs. wisc. edu/vdt 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 11

Open. Science. Grid. org OSG Specific Packages • MIS-CI: MIS-CI-0. 2. 6 RC 2 – OSG information collector for Grid. CAT, ACDC monitoring … • osg-auto-0. 1. x – OSG’s VDT ‘configurator’ package – provides the osg release notes and version stamp • osg-grid 3 -info-prov – Grid 3 -OSG information provider shell script 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 12

Open. Science. Grid. org Installation and Configuration

Open. Science. Grid. org Installing a Computing Element • Preliminaries: – The typical cluster configuration consists of • A headnode/gatekeeper + workernodes • A batch system that’s already deployed – OSG resources use a variety: PBS, condor, LSF… – Some even use the condor packaged with the VDT – WAN access for WNs is a resource issue • Some sites provide it via public IPs: for example FNAL • Lots of other sites provide it through NAT firewalls • But some don’t allow WAN access at all (U. Buffalo) 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 14

Open. Science. Grid. org Services Deployed on the CE • Typical gatekeeper services voms-servi – GSI/GRAM: Globus jobmanagers voms-servj • jobmanager-fork: stage in jobs + more • jobmanager-<batch>: access the WNs – gtk client tools • globus-job-run, globus-url-copy, gridproxy-init… headnode monitor • Monitoring and Information System – MIS-CI: OSG information provider – Mon. ALISA: JINI, java, WDSL, SOAP – MDS: GIP 1. 2 is being added now! compute-01 gums-serv CE compute-N • Authorization/authentication system – New Privilege Project is being deployed on many sites • Some components can be placed outside of the CE itself – But sites can still use the legacy Grid 3 system • gridmap-file generated by edg-makegridmap 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 15

Open. Science. Grid. org The Privilege Project Application of a Role Based Access Control model for OSG – A Joint US-CMS US-Atlas project – Provides an advanced authorization and authentication mechanism 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 16

Open. Science. Grid. org The Privilege Project Provides • Centralized, dynamic mapping of grid users to local OS qualifiers – VOMSes are still used to contain grid identities – Gone are the static grid-mapfiles • Improvement to AC policy implementation – The GUMS service – Access rights granted based on user’s • VO membership • User selected role(s) 9/18/2020 Grid Identity Unix ID Certificate DN <-----> local user VO Role(s) <-----> local group(s) 2 nd LCG Operations Workshop: Jorge L. Rodriguez 17

Open. Science. Grid. org Privilege Project Components User Management (VOMSRS) Client server (UI) Client tool for role selection: VOMS-proxy-init 1. VOMS-proxy-init request with specified role A VO service VOMS Server 2. Retrieves VO membership and role attribute VOMS Attribute Repository Server with VDT 1. 3 based on gt 3. 2 3. Standard globus-job-runrequest with VOMS-extended proxy VO membership synchronization 4. HTTPS/SOAP Request: SAML Query: grid. FTP & Gatekeeper Gridmap callout PRIMA module May user “Markus Lorch” of “VO=USCMS / Role=prod” access this resource? 5. HTTPS/SOAP Response: SAML Statement: Decision=Permit, with obligation local UID=xyz, GID=xyz 6. instantiates job-manager Servers with VDT based gt 3. 2 Server with VDT 1. 3 based onon GT 3. 2 Server with VDT 1. 3 based on GT 3. 2 9/18/2020 An OSG site 2 nd LCG Operations Workshop: Jorge L. Rodriguez GUMS Identity Mapping Service (manages user accounts on resources, incl. dynamic allocation) Web-Service Container 18

Open. Science. Grid. org Component Dependencies • VOMS and versions – VOMS version 1. 2. 19 or 1. 3. 1 but not 1. 3. 0 – VOMS-proxy-init available from User Interface • PRIMA module requirements – globus authz. callout shipped with gt 3. 2 – Outgoing HTTPS connectivity to appropriate GUMS server • GUMS requirements – Tomcat web service container – Incoming HTTPS connectivity opened to appropriate gatekeeper(s) – My. SQL database – Synchronization of DB with all VO VOMS servers 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 19

Open. Science. Grid. org Authen. /Authorization in OSG will support multiple modes of operation • Will work with legacy client and/or server combinations – Full legacy server: VOMS+edg-makegridmap • Privilege enabled client requests (VOMS-proxy-init) will be mapped to local user as previously: VOMS extensions ignored – Full Privilege server: All Privilege components enabled • Legacy client requests are supported but user can not be mapped to a different VO or assume a different role under its own VO • Also a Grid 3 compatibility mode will be supported – Supports privilege operations but without the globus PRIMA callout and thus can not support the “role” based mapping – The gatekeeper/grid. FTP server is operated with legacy Grid 3 stack (gtk 2. 4 servers…) – Automatically provides reverse maps for Grid 3/OSG accounting 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 20

Open. Science. Grid. org Grid 3 Compatibility mode User Management (VOMSRS) Client server (UI) Client tool for role selection: VOMS-proxy-init A VO service 1. VOMS-proxy-init request 2. Retrieves VO membership and role attribute 3. Standard globus-job-runrequest with VOMS-extended proxy grid. FTP & Gatekeeper to OSG accounting 4. instantiates VOMS Server cron job reverse map gridmapfile Server with VDT 1. 3 based on gt 3. 2 VO membership synchronization A. Periodically queries GUMS server May user “Markus Lorch” of “VO=USCMS / Role=prod” access this resource? The role however, is ignored B. Response create static grid-mapfile DN mapping to a local UID=xxx, but no role based assignment job-manager Legacy Grid 3 Servers Serverwith VDT 1. 3 based on. GT 3. 2 Server VDT on 9/18/2020 VOMS Attribute Repository An OSG site with legacy gatekeeper or grid. FTP server 2 nd LCG Operations Workshop: Jorge L. Rodriguez GUMS Identity Mapping Service (manages user accounts on resources, incl. dynamic allocation) Web-Service Container 21

Open. Science. Grid. org Storage Elements in OSG • SRM managed SE for the OSG 0. 2. x – d. Cache/SRM based SE at FNAL • srm v 1. 1: implicit space reservation, staged transfers • Multi-VO access: at the VO level of granularity – OSG monitoring • Grid. CAT and BDII: looks at GRIS running on FNAL’s LCG 2 CE • “Tactical storage”: workspace for applications – Currently our SE consist of grid. FTP service with access to a shared file system • In Grid 3 some sites deployed along with the CE gatekeeper – Plans are underway to add SRM features • d. Cache servers at USCMS sites soon and planned for a future OSG release, d. Cache client (dcp) is being deployed even sooner • DRM service SRM over nfs for a future OSG release 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 22

Open. Science. Grid. org SE Services in OSG Sites The canonical OSG site’s (Tactical Storage) – Access via Grid. FTP – Local shared disks across all components of the CE fileserver headnode POSIX nfs, afs…gps • nfs is typical – (CMS work flow scales 500+ jobs) compute-01 CE compute-N • But other technologies employed – gpfs, IBRIX, afs, gfs, pvfs… – The Grid 3/OSG workspaces • • $APP = application store, file lifetime one to many runs $DATA = data staging in&out, file lifetime ~ run $TMP = data staging in&out, file lifetime ~ job $WNTMP = temp data local to WNs (not shared) – This is usually deployed as part of the CE 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 23

Open. Science. Grid. org OSG’s TG-Storage • OSG has an active Storage Technical Group – Several activities being discussed – Two of them DRM and d. Cache are under way and expect to begin deployment in future releases • Other SE technologies also being discussed – NFS version 4: University of Michigan – LOCI-IBP: Vanderbuilt • ACCRE sponsored project to integrate into OSG a new type of advanced storage element – – Mirroring, Striping and encryption of data Agnostic to location specific details Completely Firewall friendly (verified) Very easy to set up, dependencies are minimal • Vanderbuilt group is now working on adding a certificate based authorization/authentication mechanism and SRM inteface 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 24

Open. Science. Grid. org FINI

Open. Science. Grid. org Extra Slides

Open. Science. Grid. org Discussion Topics • Interoperability – Storage Elements – CE: • Monitoring • Batch systems • Application use cases in LCG vs OSG – OSG describe MOP… 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 27

Open. Science. Grid. org ROCKS the 5 Minute Tour ROCKS builds on top RH kickstart technology – A quick Kickstart digression • A kickstart file is a simple ASCII script – Lists single and/or groupings of RPMS to be installed – Provides staged installation sections %pre, %main %post … • Anaconda is the installer – Parses and processes kickstart commands and installs the operating system on a blank server This is in fact what you interact with when you install a RH or SL on your laptop 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 28

Open. Science. Grid. org ROCKS the 5 Minute Tour cont. ROCKS enhances kickstart by – Providing machinery to push installations to servers • DHCP: provides nodes with their identity via its MAC address • cgi script: generates kickstart on the fly • https: protocol used to exchange data (kickstart, images, RPM) – ROCKS also provides a kickstart management system • ROCKS’ kickstart generator parses user defined XML spec. files and combines that with node specific information stored in a My. SQL database • Complete system description is packaged in components grouped into logical object modules 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 29

Open. Science. Grid. org My. SQL & XML in ROCKS Physical servers: are the objects ROCKS’ My. SQL Data. Base XML graphs: appliances are like the classes uflorida-frontend ufgrid 01 uflorida. DGT grinux 01 grinux 40 grinux. N Global Variables, MAC address, node names, distribution, membership, appliances… uflorida. PG frontend gatekeeper-pg gatekeeper-grid 01 gatekeeper gaterkeeper-dgt compute-pg compute-grid 01 comput e compute-dgt grinux. M 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 30

Open. Science. Grid. org ROCKS the 5 Minute Tour cont. A vanilla ROCKS graph describes a single cluster with service nodes Note: use of “rolls” 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 31

Open. Science. Grid. org ROCKS at UFlorida • Current installation is based on ROCKS 3. 2. 0 • But we’ve significantly modified things – A single ROCKS frontend manages everything – Support multiple distributions RH 7. 3, RHEL 3 & SL 3 – Frontend’s role is changed too • Primary role is for cluster management only • No user logins, services moved off, heavily firewalled… – Routinely interact with ROCKS My. SQL database – Extensive modification to vanilla xml trees and graph • We roll our own “uflorida” graphs, one for each distribution • Many changes & additions to the stnd XMLs • The XML tree is based on an older ROCKS vers. 2. 3. 2 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 32

Open. Science. Grid. org The uflorida XML graphs uflorida. xml for RHEL 3 This graph is also used for SC 3 uflorida. xml for RH 7. 3 servers 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 33

Open. Science. Grid. org The Meta-Cluster AMD 64 griphyn WAN ultralight alachua RH 7. 3 SL 3 archer melrose RHEL 3 hawthorne micanopy ufgrid 03 These are the facility's development nodes ufgrid 04 ufgrid 02 phedex pool 01 ufdcache pool 02 uflorida-frontend uflorida. PG ufgrid 01 grinux 46 grinux 41 grinux 40 grinux. N grinux 45 pool 03 uflorida. DGT uflorida. IGT private LAN gumserv gatoraid 2 gatoraid 1 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez cleo 01 34

Open. Science. Grid. org The Meta-Cluster AMD 64 WAN griphyn ultralight alachua RH 7. 3 SL 3 archer melrose RHEL 3 hawthorne micanopy ufgrid 03 The big kahuna: Our stripped down. Gri. Phy. N, UL & HEP Analyses User Interface machines version of the ROCKS frontend • User login machines • Grid User login • User environment ufgrid 02 • • Access to grid via condor. G ROCKS administration node • Ultralight development • RPM server phedex uflorida. PG uflorida. DGT uflorida. IGT ufgrid 01 ufdcache • ROCKS DB server • Kickstart generation • DHCP server private LAN grinux 46 grinux 41 grinux 01 • Admin nfs server pool 01 pool 02 pool 03 • etc • No users, no logins, no home … Other Services uflorida-frontend • webserver gumserv • Strict firewall rules • Primary DNS server • cvs pserver gatoraid 2 ufgrid 04 gatoraid 1 9/18/2020 grinux 40 grinux. N 2 nd LCG Operations Workshop: Jorge L. Rodriguez grinux 45 cleo 01 35

Open. Science. Grid. org Computing Services AMD 64 RH 7. 3 SL 3 CMS griphyn WAN ultralight alachua ufgrid 03 CMS analysis batch configuration: • Condor pool on analysis nodes ufgrid 04 flock to ufgrid 01 cluster* • Jobs will archer melrose RHEL 3 CDF CLEO hawthorne micanopy * When WN -> SL 3 ufgrid 02 phedex ufdcache uflorida. PG ufgrid 01 uflorida. DGT grinux 01 pool 02 pool 03 Gatekeeper + master nodes • uflorida-frontend Grid access only • GRAM, GSIFTP… gumserv • Other services minimum gatoraid 2 grinux 46 grinux 41 gatoraid 1 grinux. N grinux 45 9/18/2020 grinux 40 2 nd LCG Operations Workshop: Jorge L. Rodriguez uflorida. IGT private LAN cleo 01 36

Open. Science. Grid. org Storage Services RH 7. 3 AMD 64 WAN griphyn Data Movement • Pub. DB ufgrid 03 • Dataset Service • Phe. DEX ufgrid 04 server ultralight pool 01 ufdcache pool 02 uflorida-frontend melrose hawthorne micanopy d. Cache Administrator “admin door” SRM & d. Cache webserver All RAID 5 fileservers, large pool on admin node uflorida. PG Total of uflorida. DGT ufgrid 01 25 TB of storage (19. 2 TB onuflorida. IGT order) Standard d. Cache configuration grinux 01 pool 03 gumserv gatoraid 2 gatoraid 1 9/18/2020 archer RHEL 3 d. Cache with SRM: Virtual file system (pnfs) with cache’ing ufgrid 02 phedex alachua SL 3 grinux 40 private LAN grinux 46 grinux 41 NFS servers • User’s home area • User’s scratch & data space pool nodes • Grid 3: $DATA, d. Cache $TMP, $APP 40 x 50 GB partitions • gridftp server (SE? ) Resilient d. Cache configuration grinux. N 2 nd LCG Operations Workshop: Jorge L. Rodriguez grinux 45 cleo 01 37

Open. Science. Grid. org Summary of Services • Production Computational Services – Multiple Computing Elements, supports: • Analysis cluster with interactive logins • Dedicated Monte Carlo production • Production Data Handling & Storage Services – Storage Elements (RAID fileservers) – d. Cache installation: resilient, standard – Data Movement & Discovery services (CMS specific) • Phe. Dex, Pub. DB, Dataset Service • Miscellaneous and Development Services – Fabric infrastructure • Fabric management, Webserver, email lists cvs … – Development environments 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 38

Open. Science. Grid. org Plans • Evolve the current configuration – Continue to move towards SL 3 but retain legacy where needed – Complete “ROCKsification” of all elements – Add AMD 64 bit distribution(s) • Reorganization of the configuration – Sever development from production environment – Introduce ROCKS rolls • Separate major components into separate rolls • Greatly simplify support and distribution 9/18/2020 2 nd LCG Operations Workshop: Jorge L. Rodriguez 39