COMA some Event Index Overview Status Plans Elizabeth

COMA (& some Event. Index) Overview, Status, Plans Elizabeth Gallas, ADAM Workshop #1 January 20, 2021

COMA: “Conditions (& Configuration) Metadata for ATLAS” § TWiki: Atlas. Computing/Conditions. Metadata COMA / EIO System EIO: “Event. Index Oracle” (a subset of EI Hadoop event metadata: exploring relational storage) § EIOracle: Atlas. Computing/Event. Index. Oracle Two General Components: 1. Relational Database (Oracle) § python-based loading programs: load data into dedicated tables § § Sources: another slide later COMA: 91 tables + ~ 30 views; EIO 13 tables + 4 views § Conditions, Trigger, AMI (DB, DP experts, …), Tier-0, AFS, (TAG Catalog/DB) Reports and Browsers - php-based on TAGServices machines 2. § COMA Portal Page https: //atlas-tagservices. cern. ch/tagservices/Run. Browser/index. html COMA data is also available via py. AMI: TWiki Atlas. Computing/Coma. Py. Ami. Views Jan 2021 COMA / EIO 2

Conditions DBs Oracle Data Dictionaries Cool. Mon Py. COOL API Overview: COMA Data Sources AFS files COMA DB User Input (via AMI web interfaces) Frontier_ Track_Mod Run 2 SFO/ Tier-0 DB Run 1 AMI DBs Event Index Hadoop SQL Read & Copy Trigger DBs Jan 2021 TAG DB Catalog TAG DBs COMA / EIO SQL Read & Write Non-SQL access 3

COMA Loading Workflow <5 seconds ALL_Runs time Loaded by AMI/experts Every 1. 5 hours (5) Periods COMA Runs Filled by DPC/AMI Runs, LBs, Events <5 s All SMK Menus Event Index: Trigger decoding Every 3 hours (10) Once/day (2) In development Magnets Conditions DB Folder/tag metadata Fills Ready* SMK Menus Fills, Stable Beams Ready Flags Triggers, Streams <15 s <5 s 5001500 s Luminosity <10 s CTAP, CRAP <5 s Event Counts By Stream <10 s Subsystem Experts (AFS files) Filled by Cool. Tag. Coord/AMI <5 s By Run, LB <10 s Global. Tag: States <5 s PS Sets per Run Level Prescales Aggregate PS Conditions History Folder Classification Prescales <5 s EIOracle AMI Stuff <5 s GRL Metadata <5 s Trigger Lum Global. Tag/Datasets PS Corrected Lum Filled by AMI Jan 2021 COMA / EIO 1000 s to hours Trigger Counts* 4

COMA portal overview Popular for physics users Conditions DB experts Event-level services via EIOracle (built w/in COMA web) Jan 2021 COMA / EIO

COMA period menu Data period: • set of runs grouped for 'some' purpose (often grouping 'like' detector or machine conditions) • uniquely defined by project name (ex. data 17_13 Te. V) AND period name (ex. 'C' or 'C 2') • Choose periods by - Year, - Project, - Beam Energy/Type, - Period Name Jan 2021 COMA / EIO 6

Example COMA Period Description report Yellow links: show/hide members Jan 2021 COMA / EIO Table Header links: click for detailed description

Example COMA Period Runs report od R s n u iod R he Peri Per , ns in t -wise reports u R l l a Rows: : links to run ns Colum information, s, un et state d info n g basic r a m ry of elate summa luminosity r nd beam a Jan 2021 COMA / EIO 8

Example COMA Period Stream report R m a e riod, iod Str stream e r P e p e s h t t runs in d RAW even vents and for all regate (SFO e unts and g g n a o i t w a m o sho d infor vents, runs c e t a l e r ge and missin mpleteness) , s e m o volu index c I E , s e rang Per Jan 2021 COMA / EIO 9

Example COMA Period Trigger report orma x. # e tion (e er inf al trigg vents SBR) idu v y i d d n a i e led eam R B + detai in period, e l b a ed ring St u trigger d e g le ran presca What about GRLs ? (see the tutorial) Jan 2021 COMA / EIO 10

Usage: TAGServices AWStats Not much ‘verbal’ feedback § Just watch usage § § AWStats Report logging § § 2018 Next slide All 5 interfaces: Regular usage § Popularity of specific services varies over time May 2018 Jan 2021 COMA / EIO 11

COMA Report Logging Stats: Over 18 k Report Requests logged in 2018 55 different pages (reports) shown by name, time req’d, count This list does not include ‘error’ reports. . . another story § “Hit Requests” = Count each COMA Report is generated § § Jan 2021 § Report inputs values vary (i. e. there are thousands of possible trigger names) Not all services are logged “Avg. Time Seconds” = Average Time to generate reports COMA to / EIO (php start end time – query time is a subset)12 § § Reports are generated in usually less than 1 second Longer times needed for some Trigger-related reports spanning a large time range Logging: used to understand usage and focus report/query improvements (‘error’ log analysis)

§ Evolution: COMA evolved / grew a lot over time “organically” § Began as database to manage simulated luminosity before data taking and for MC Trigger Menu decoding for the TAG DB § § § Added menu decoding for real data (2010). . . And a few more conditions. . . Developed simple web interfaces to display the data stored Data Prep Coordinators. . . Influenced evolution § § History Jamie Boyd: requested to publish interfaces for general ATLAS use Beate Heinemann: advocated moving “Data Periods” (TWiki page) COMA. . . Comments from users / experts: § “fantastic! This already just saved me several hours of otherwise very tedious work. ” § “many thanks. I found a TWiki saying that maybe COMA will not be maintained on run 2 and now you are saving me. I am glad to see this working” § I just wanted to pass on a comment of our student. . . at the. . . group meeting today. He remarked that the COMA documentation. . . is really outstanding and helped him enormously to progress. I can only join this statement, COMA is really one of the best-documented work in ATLAS. And super useful of course! (Joerg Stelzer) Jan 2021 COMA / EIO 13

Typical use cases: Run Query • detailed information about data at the run-level • Critical: detector, operations tasks • nice luminosity plots • web interface and command line trigger (active, prescales), information by luminosity-block, summary about group of runs (what changed, in common), detector defects, machine conditions, etc. COMA • fast information about data in containers/periods/runs/lumi-blocks • overview of configuration/activation of streams and triggers • aggregation of various quantities across projects, periods, etc. • Command line via py. AMI quick overview of runs within periods of interest, period definitions, GRL-aware, overview of specific triggers, event counts by trigger AMI • catalogue framework for all data and MC datasets • web interface and command-line/API available the ATLAS m o fr e d li s A utorial induction / T ontext (COMA in c ols) with other to understand provenance of a dataset, investigate AMI-tag meanings, get MC metadata for analysis hn-atlas-physics. Metadata@cern. ch Summary https: //twiki. cern. ch/twiki/bin/view/Atlas. Computing/Software. Tutorial. Finding. Datasets Jan 2021 COMA / EIO 14

Plans using existing schema § Report/Brower Interface updates: § links: COOLR (à la Andrea Formica) § § Counts event on any combination of trigger conditions § § Add CREST metadata Add Conditions access from Athena logs § Incomplete development Combines GRL, trigger, luminosity, with EI GUID list selection Add new Oracle Views and Table functions (py. AMI) § § finally a real Conditions Browser ! New Development with extensions in scope Some links to AMI are deprecated, need to update Try browsing datasets from EI Run 3 (Hadoop / Phoenix) New service ideas: “Good data samples” § § § links: EI Hadoop “Trigger Counter” (Carlos Garcia) § § Plans pending requests from experts (CLI access to specific data) Preparation for Run 3 § Remove partitioning (may be temporary) § § Updates needed for changing input source format § § e. g. Trigger DB menus stored in JSON for Run 3. . . deal with key changes. . . Any changes to Conditions DB folders § Jan 2021 Limiting development due to Oracle version incompatibility e. g. Luminosity, Beams, Magnets, etc. COMA / EIO 15

§ COMA development contributors § Oxford students (qualification tasks) § Loading: § § Lydia Beresford: Run 2 (run workflow changes), GRLs Kathrine Pachal: Beam conditions and Luminosity Interfaces: § § Contributors, Publications Ryan Buckingham (2010): 1 st ‘Run Browser’, DQ (deprecated) Thanks also to the AMI Team and Andrea Formica ! Event. Index Oracle: developed with Gancho Dimitrov & Petya Vasileva § Main publications (@CHEP) § “Conditions and configuration metadata for the ATLAS experiment” (2012) “Utility of collecting metadata to manage a large scale conditions database in ATLAS” (2013) “A JEE RESTful service to access Conditions Data in ATLAS” (2015) “An Oracle-based Event Index for ATLAS” (2016) “Collecting conditions usage metadata to optimize current and future ATLAS software and processing” (2016) “Optimizing access to conditions data in ATLAS event data processing” (2018) § § § Jan 2021 COMA / EIO 16

Summary § COMA § Designed for people: provides quick access to Run-level metadata. . . for specific runs as well as queries across all ATLAS runs § § § Save expert time with many “look-up” tools, aggregation of information. Help new collaborators get an overview of ATLAS data & links to other systems Metadata collected daily from many sources. . . Plus data from experts: § Primary repository of “ATLAS Data Periods” § § Also stores global tag “state” designations § § Data Prep defines Periods in COMA via AMI entry interfaces COOL Tag Coordination: setting of “Current”, “Next” for BULK, ES, HLT Schema: § Incredibly stable Run 1 Run 2 § § Last 10 years: many extensions in scope. . . probably more to come. . . § Jan 2021 Metadata is relatively stable even as underlying storage changes Generally adding new data (either columns or new tables) COMA / EIO 17

Jan 2021 COMA / EIO 18

The command line • AMI (https: //ami. in 2 p 3. fr/py. AMI/) and COMA (TWiki: Coma. Py. Ami. Views) setup: pyami (more later), valid VOMS proxy, Athena release > setup. ATLAS > lsetup pyami 'asetup 20. 1. 4. 5, here' > voms-proxy-init -voms atlas • run. Query TWiki Atl. Run. Query sample commands (after ATLAS setup): > Atl. Run. Query. py --help > Atl. Run. Query. py 'f r 2012. G-H / sh all' Jan 2021 COMA / EIO