ATLAS Grid Information System Alessandro Di Girolamo CERN

ATLAS Grid Information System Alessandro Di Girolamo CERN IT - Support for Distributed Computing

Caveat § Information System is a BIG WOR(L)D word IS: Different people mean different things • wor. Ld IS: whatever you mean, it consists anyway of many different components… We do not discuss this today • § Today: the AGIS experience • How we created a framework Ø Useful Ø Used Ø Flexible and extensible • • Description of use cases Pros and cons 29 September 2015 Alessandro Di Girolamo IT-SDC 2

ATLAS Grid Information System § Collect, integrate and allow to define Sites (Resource Centers), their services information, the resources configuration and status • Static information (lifetime of the service) and semi-static (e. g. downtimes). Describe and expose the ATLAS topology § AGIS in few lines: § • • • DB backend: persistent storage and changes history web. UI and collectors REST API (POST and GET) 29 September 2015 Alessandro Di Girolamo IT-SDC 3

The AGIS project § Started “many” years ago (>5) • • § Project leadership (stable since ~5 years) • • § Strongly wanted by ATLAS Computing management Several groups involved Responsibility shared between CERN-IT/SDC and BINP (Novosibirsk) Alexey Anisenkov as Technical Coord, myself as Project Leader Several persons contributed over the years • Dubna; summer students; experts from CERN-IT/SDC … 29 September 2015 Alessandro Di Girolamo IT-SDC 4

AGIS collectors and web. UI § The list of sites supporting ATLAS is defined in AGIS Procedures for new sites joining ATLAS: • Within WLCG for pledged resources Ø ATLAS International Computing Board for Tier 3 s ( including unpledged HPC and Cloud. Resources) Ø § For each site AGIS get the list of services from GOCDB and OIM • § Service detailed information automatically collected or manually updated • § E. g. CE from BDII, Storages described manually by Site. Admins into AGIS web. UI • • • § Mainly AGIS uses them as Service Registry for Service Discovery Edit/Update specific service information Add new services (if needed, e. g. HTTPS storages, Object. Store…) Define and “attach” new ATLAS specific objects . • . 29 September 2015 Alessandro Di Girolamo IT-SDC 5

AGIS centric view § AGIS is a fundamental piece of ATLAS Computing • Under the hood for the non-expert Ø But does not mean less important • “Fight” each day (monthly? ) to avoid people/framework diverging Ø Continuous gathering of new requirements and development § Key point: the ADC components “speak the same language” through AGIS: • Panda, Rucio, monitoring Dashboards, pilot, Frontier-Squids. . . Ø You add e. g. one new storage you do not need to edit configs in N different frameworks, all within AGIS! • “Fast” complete integration of new technologies within ADC 29 September 2015 Alessandro Di Girolamo IT-SDC 6

AGIS e c r u o ed s e r us s“ ce p “ s ur v o res d e id ” y b by ” 29 September 2015 Alessandro Di Girolamo IT-SDC 7

AGIS § Collect integrate and allow the sites services information, resources configuration and status. E. g. : • • § CE host, type, jobmanager, port, queues (with details) Storage element host, type, protocols, path, etc Panda. Queue (aka Panda. Resource) config DDMEndpoint (aka RSE Rucio Storage Element) config New needs (e. g. Object. Store storage, HTCondor. CE)? • • Doable but not easy Several interactions are needed to understand clarify the requirements Experience tells that we are able to get things into AGIS in timely manner (weeks to get them in production) It is not just a click, no free lunch! 29 September 2015 Alessandro Di Girolamo IT-SDC 8

Feeding: Automatic & Manual § Automatic as much as possible! But not always possible • § Manual but with deep validation at insertion level! Clearly defining the possibilities for Site. Admins/Experts • Ø Too much freedom sometimes is no good How many new storage do we have per year? Very few! • Each change in a storage requires many changes in other frameworks Ø not something worth to automate Ø § REST API for both AGIS web. UI and for users • § Possibility to automate Common “Language” • AGIS is able to understand speak Glue 1. 3 and 2, and also able to digest other kind of information Ø Agreeing on a common language (at least meaning of keywords) is important 29 September 2015 Alessandro Di Girolamo IT-SDC 9

Distributed Data Management Endpoint diagram Service world Experiment world Storage http xrootd srm Protocols Storage. Area (Space. Tokens, paths …) 29 September 2015 DDM Endpoint Many more attributes: • Disk/Tape • path for HTTP, path for SRM, … • Which protocol for which activity, e. g. : • HTTP for logs and user download • grid. FTP for third party transfer • … • Physics Groups • Topology (Tier level, closeness) • … Alessandro Di Girolamo IT-SDC 10

Federation in AGIS: FAX storage redirector 29 September 2015 Alessandro Di Girolamo IT-SDC 11

Federation in AGIS: FAX storage redirector 29 September 2015 Alessandro Di Girolamo IT-SDC 12

Few thoughts § Federations for who? • different players interact with the federation at different level. E. g. Ø Panda for ATLAS could benefit of Federated sites, but maybe could also broker jobs better if it knows the internals Ø Data. Management system definitely would benefit in knowing the internals! Ø Users definitely benefit of a federation (both of data and metadata!) • § AGIS is able to provide such “views” Dynamic Storages • Can we enable/attach to a “federation” e. g. users’ laptop? 29 September 2015 Alessandro Di Girolamo IT-SDC 13

Federation: players view storage redirector 29 September 2015 Alessandro Di Girolamo IT-SDC 14

Federation: players view storage redirector 29 September 2015 Alessandro Di Girolamo IT-SDC 15

Summary § AGIS describe the topology and services of ATLAS Distributed Computing • • Concept: “Provided by” vs “Used by” Flexible: new objects/collectors can be added without major re-organization of the framework Ø But not enough to snap your fingers, need to be properly architected • § Clear goals, technical expertise to make it happen, support from the management are some of the key aspects which made AGIS useful and used Storages and Federations: • Need common strategy and solid agreements 29 September 2015 Alessandro Di Girolamo IT-SDC 16

Back. Up 29 September 2015 Alessandro Di Girolamo IT-SDC 17

. . § . • • . . Ø. . §. §. § 29 September 2015 Alessandro Di Girolamo IT-SDC 18