OSG Grid Operations Leigh Grundhoefer Indiana University leighg

OSG Grid Operations Leigh Grundhoefer Indiana University leighg at indiana dot edu

Agenda n n Grid Operations Development OSG Functional Service Cycle q q n Deployment Integration Provisioning Production OSG Operations Activities 3/10/2021 leighg at indiana dot edu 2

“Big Picture” Goals n The OSG Operations activity and Support Centers Group has been tasked with the role of preparing, provisioning and running the infrastructure used for the OSG production environment. n The operations activities duties to the OSG are to ensure that the production environment is usable for the current application base and to continue to evolve as a common service environment which is able to support multiple sciences with a applicationfriendly grid infrastructure. 3/10/2021 leighg at indiana dot edu 3

Grid Operations Security, Policy and Authentication Registration, Verification and Monitoring Integration Test Bed Provisioning Other Grids Knowledge base Production Releases And Updates of Commonly Answered questions Documentation of software Trouble reporting and ticketing Support Center coordinator 3/10/2021 leighg at indiana dot edu 4

Operations Environment n Organization and Definition of core elements q q n n n Grids Support Centers Virtual Organizations Resources Registration by Owners and Providers Common software services Verification and ongoing evaluation q q q Resources and services Support Center ticketing, ticketing response VO services 3/10/2021 leighg at indiana dot edu 5

Operations Environment (cont. ) n Information from monitoring q q q n n n Job slots and file transfers Published policies Accounting Coordination of OSG Help Desk Coordinator of support centers Definition and execution of Standard Operational Procedures (SOPs). 3/10/2021 leighg at indiana dot edu 6

Grid to Grid relationships n n n Understand create specialized help desk and trouble reporting schemas Understand create monitoring and accounting interfaces Create and deploy identity and authorization interfaces 3/10/2021 leighg at indiana dot edu 7

Agenda n n Grid Operations Development OSG Functional Service Cycle q q n Deployment Integration Provisioning Production OSG Operations Activities 3/10/2021 leighg at indiana dot edu 8

Where do you get this stuff? (Architecture and Requirements) Release Description Deployment Activity VO’s ITB 0. 3 Integration Test Bed Provisioning Candidate Blueprint (ARCH) Operations OSG 0. 4 Service Development (Sponsored Activities) Technical Groups 3/10/2021 leighg at indiana dot edu 9

OSG Integration Activity Readiness plan adopted Readiness plan Effort Resources Software & packaging Service deployment Middleware Interoperability Functionality & Scalability Tests feedback 3/10/2021 VO Application Software Installation Release Description Application validation Metrics & Certification leighg at indiana dot edu Release Candidate OSG Operations-Provisioning Activity OSG Deployment Activity OSG Integration Activity 10

Provisioning n Finalize all installation software and procedures q OSG based software packages Pre-compiled binaries n Source code (compiled during installation) n q n Post configuration scripts ( configure_osg. sh ) Create production “Grid Support” documentation for all software and defined procedures. 3/10/2021 leighg at indiana dot edu 11

Provisioning n n Translate OSG operations model into production operations activities Provide timelines for Resources and Support Centers Setup versioning using release procedures Install or upgrade grid wide services 3/10/2021 leighg at indiana dot edu 12

Production n Reports q q q n Meetings q q n Usage reports from OSG monitoring Operations reports to the OSG community Daily Sites Status reports to Support Centers Weekly Operations Activity Release-based provisioning Activity Weekly Support Centers Technical Group Weekly Documentation Activity Monitors for verification q ACDC Operations Dashboard 3/10/2021 leighg at indiana dot edu 13

Verification of Resouces n ACDC Operation Dashboard provides detailed cyclic testing of all resources q q q Tests based upon “site-verify” tool distributed with OSG common software, around 30 tests Tests results output available per test per site Five possible results n n n No Information Pass Fail Error Not Tested Resources are grouped in to three areas: Production, Pending or Offline 3/10/2021 leighg at indiana dot edu 14

Agenda n n Grid Operations Development OSG Functional Service Cycle q q n Deployment Integration Provisioning Production OSG Operations Activities 3/10/2021 leighg at indiana dot edu 15

GOC: A Communications Hub n Grid Operations Center q Leveraged Coverage from GRNOC n n OSG Trouble Ticket System q q n Trouble Ticket Exchange with Support Centers / Grids Weekly Issue tracking report Web Page Development and Documentation q q q n Abilene, NLR, Trans. PAC OSG Production web site www. opensciencegrid. org Commonly answered questions Knowledge Base Collaborative development information - osg. ivdgl. org OSG Registration Database 3/10/2021 leighg at indiana dot edu 16

GOC: A Service Center n Grid wide Information Services q q n Grid wide Monitoring Services q q n n Registration Database Catalog Production Software Caches OSG Knowledge Base Open. Science. Grid web site GLUE information providers ACDC Dashboards Grid. Cat Mona. Lisa archive Multiple OSG ITB resources Small but demonstrative OSG production resource 3/10/2021 leighg at indiana dot edu 17

GOC: Incident Response n Response has been defined by the Security Technical Group q q q n All incidents should be reported mail aliases which are monitored by the GOC. The GOC maintains a list of local site security contacts, derived from the registrations. The GOC designed and implemented a specialized mail service for secure correspondence A response team leader forms a group to access, contain, and report on each incident. 3/10/2021 leighg at indiana dot edu 18

GOC: Policy Procedures n n Grid Service Change or Upgrade Registrations Support Centers q Virtual Organizations q Resources q n Critical OSG Release Update 3/10/2021 leighg at indiana dot edu 19

Support Center Registration 3/10/2021 leighg at indiana dot edu 20

Virtual Organization Registration 3/10/2021 leighg at indiana dot edu 21

Resource/Service Registration 3/10/2021 leighg at indiana dot edu 22

Critical Release Update 3/10/2021 leighg at indiana dot edu 23

3/10/2021 leighg at indiana dot edu 24

OSG Community Support n n How do we support issues that fall outside the Support Model? Open Support Model q q Mailing Lists (OSG-General) Knowledge Base and Release Documentation Jabber chat room Weekly meetings 3/10/2021 leighg at indiana dot edu 25

3/10/2021 leighg at indiana dot edu 26

GOC: Conclusions n n Enables Users and Usage Creates a known and usable service environment Allows status, monitoring and accounting Helps users and applications “bridge the gap” between single use and grid based resource utilization 3/10/2021 leighg at indiana dot edu 27

Thank you! 3/10/2021 leighg at indiana dot edu 28