ATLAS COMA Database for COnditions Metadat A Elizabeth

ATLAS COMA Database: for COnditions Metadat. A Elizabeth Gallas - Oxford ATLAS-UK Distributed Computing Tutorial Edinburgh, UK – March 21 -22, 2011 Mar 2011 Elizabeth Gallas - COMA

Outline § What is COMA § § Tour of COMA Interfaces (Run. Browser package): § § Principles Components Recent changes COMA Data Period Reports (AKA: no old name) COMA Reports (AKA: run. Browser. Report) COMA Browser (AKA: run. Browser) Summary and Conclusions Mar 2011 Elizabeth Gallas - COMA 2

What is COMA ? COMA: short name for “COnditions Metadat. A”: Originally built to support dynamic queries of TAG DB Now, also a standalone system with interfaces. Part of general effort to consolidate/relate ATLAS physics metadata and relate it to dataset/processing metadata. § COMA Database Content: § Must provide information needed to decode TAG attributes § Include information for both Online and MC Runs § § TAGs for Online/MC have the same attributes (no MC truth) Catalogue for Online/MC reflects similar processing workflows Overall system must handle gracefully missing information Upload: select Conditions for Runs of ‘analysis interest’ § Note: NOT all Runs § § Runs with recorded events which are reconstructed, producing TAGs* Note: NOT all Conditions § Only upload Conditions in LOCKED COOL tags (w/cross checks) Refine/Correct/Derive conditions to form more effective criteria COMA and AMI Reports and Interfaces § Relay information to users and systems § § Mar 2011 Elizabeth Gallas - COMA 3

Components of the COMA Project: Storage: COMA Database Tables § A collection of selected and derived Run/LB-wise Conditions Source: mostly Conditions DB, Trigger DB plus other sources Interfaces: Run. Browser Package (sorry: now in naming transition) § General principle is to provides unique interfaces for users with reports complementary to, or distinct from, other interfaces: https: //atlas-tagservices. cern. ch/tagservices/Run. Browser/index. html Topic 3 New Name COMA_Browser 2 1 COMA_Report COMA_Period. Report NOTE: AMI / ELSSI also has links to and reports/uses COMA information Mar 2011 Elizabeth Gallas - COMA 4

COMA: Recent Changes § COMA moved to main ATLAS Offline database § § Supported as a critical service Moved into COMA : ATLAS Data Periods § As of last week ! Mar 2011 Elizabeth Gallas - COMA 5

Topic 1 § A Data Period is a set of ATLAS Runs grouped for a purpose § § Defined by Data Preparation Coordinators Used in ATLAS data processing, assessment, and selection … Each Period uniquely defined with a combination of § Project name (i. e. ‘data 10_7 Te. V’) § Period name (i. e. ‘C 1’, ‘C 2’, ‘C’, ‘All. Year’ …) Before 2011, Data Periods were § § Introduction: ATLAS Data Periods Described on TWiki page § https: //twiki. cern. ch/twiki/bin/view/Atlas. Protected/Data. Periods Stored in a file based system § Edited by hand by Data Prep Coordination (experts) § Structure evolved over last year with experience This experience valuable to decide/define long term solution New for 2011: Data Periods stored in the COMA DB § Thanks: Beate (Data. Prep Coordinator), AMI team, DB experts. Mar 2011 Elizabeth Gallas - COMA 6

Data Periods: Links to Reports and Services The links/info below can be found on the revised TWiki page: https: //twiki. cern. ch/twiki/bin/view/Atlas. Protected/Data. Periods § Interactive USERS COMA Data Period Documentation Interface § https: //atlas-tagservices. cern. ch/RBR/r. BR_Period_Report. php § § Comments: hn-atlas-physics. Metadata@cern. ch. Next slide Programmatic USERS For systems needing period info: run. Query, beamspot, Data Quality, …, “Data Period Services” provided via py. AMI: § § http: //ami. in 2 p 3. fr/opencms/AMI/www/Client/Data. Periods_py. AMI. pdf § Comments: AMI / Tag_Collector Team. Data Preparation EXPERTS: Entry Interface: § https: //ami. in 2 p 3. fr/AMI/servlet/net. hep. atlas. Database. Bookkeeping. AMI. Servlet. Command? link. Id=1479 § Mar 2011 Comments: AMI / Tag_Collector Team. Elizabeth Gallas - COMA 7

Period Documentation Menu Purpose: Generate Period documentation for chosen input criteria The report will include a description of all Periods By Year E. G. all ‘ 2010’ By Project e. g. ‘data 10_7 Te. V’ By specific Period or Group Click on the project and then your Period of interest Wildcards can be entered in this optional section, then click on Submit button https: //atlas-tagservices. cern. ch/RBR/r. BR_Period_Report. php Elizabeth Gallas - COMA 8 Mar 2011

Example Report: All 2010 Data Period Descriptions Input criteria: Shown in header -/+ highlighted links: These sections expand to show period members Members of data 10_7 Te. V. Vd. M are Vd. M 1, Vd. M 2, Vd. M 3 Mar 2011 Links to COMA and run. Query multi-Run Reports for that Period Elizabeth Gallas - COMA 9

COMA Reports: General Principles Topic 2 COMA Report (aka run. Browser. Report) Purpose: General Report interface for COMA information § § § Display what COMA knows about Runs § Useful for experts/users Provide external links to more detailed information (in other systems) § run. Query, AMI, Trigger, Data Quality …reports using COOLCherry. Py Provide reports distinctive from other systems § Avoid duplication of effort § Display COMA ‘derived’ information § § Not available in other systems COMA Report php uses GET methods: § Inputs are not complex, user can modify URL to get report they want, put links into TWiki or other documentation https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. php? runs=161562 § Currently: AMI, ELSSI, COMA Browser, COMA Period Reports generate links to COMA Reports for selections in those systems. Mar 2011 Elizabeth Gallas - COMA 10

COMA Report: Index page INDEX: https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. html GUIDE: https: //atlas-tagservices. cern. ch/RBR/r. BR_Documentation. html “Index” page: § Enter any input quantity, § IE: a Run number § IE: a Chain Name § Click on submit (or return key) Links at bottom of every page: RBR reports use “GET” methods: Modify the URL to change input RBR reports contain links No need to use index page for every query Links lead you to other COMA reports other external reports AMI, run. Query, Trigger. DB… Mar 2011 Elizabeth Gallas - COMA 11

COMA Report(s) Example Links COMA single-Run Report (Run 161562): https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. php? runs=161562 § General Run information, AMI tags, DQ, Prescale Evolution, Triggers § Trigger tables show derived Run-integrated prescales, passthrough COMA Trigger Super Master Key Report (Key 877): https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. php? smk=877 § Runs using this SMK, physics/commissioning chains, Level 1 items COMA Chain Report (Chain EF_g 11_etcut): https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. php? cn=EF_g 11_etcut § § Summary: SMK, Runs, Signature appearances of this chain Runs with this chain ‘active’ COMA Chain Wildcard Report (chain name matching string EF_g 1%): https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. php? cn=EF_g 1% § Summary: SMK, Runs with this chain: Configured, Active COMA Prescale Report (Run 162882 Prescales for chain EF_g 11_etcut): https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. php? runs=162882&cn=EF_g 11_etcut § Derived chain activity over the Run and LB-wise prescales Mar 2011 Elizabeth Gallas - COMA 12

Contains: basic information for the runs matching the input criteria Input criteria: printed in the header COMA Multi-Run Report: Click on a specific Run link: COMA Single Run report Links: Contact email, COMA Report doc Links in Table column headers: Short description of column Click on SMK link: COMA Trigger Super Master Key Report Hover any link: Indicates what will happen https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. php? pn=Vd. M Mar 2011 Elizabeth Gallas - COMA 13

COMA Single Run Overview General Run info: § § Project, LB range, Date … Events, HLT Release … Links to COMA reports Links to external reports Yellow sections expand (6): § § § Data Periods of this Run AMI tags for TAG production DQ LBSUMM assessments § § § Active Streams Prescale Evolution § § Open this section in next slide… Mar 2011 Elizabeth Gallas - COMA COOL tagged/locked PS Key changes Trigger section § § All HLT signatures All LVL 1 items 14

COMA Single Run -/+Trigger Section See examples here of COMA “derived” info Open this section in next slide… Mar 2011 Elizabeth Gallas - COMA 15

COMA Run Report (167844): Trigger Section HLT Summary: § total, active, disabled § physics, comm signatures Show/Hide the grey rows grey disabled signatures Run-wise Agg. PS Range, Flags(PT, PS, RR) Choose minbias (j. Query. Column. Filters) RBR Chain Report: All Runs w/ this chain/item Mar 2011 Elizabeth Gallas - COMA RBR Prescale Report: LB-wise PS info for this Run/Signature 16

RBR Chain Report (chain EF_e 10_loose) Configuration Summary: Shows where this element is configured: § Super Master Key(s), § Runs (Summary), § Signature Appearances Activation Summary: Shows the Runs in which this element is “active” separated by: § “Physics” EF-L 2 -L 1 signatures § § Open this section in next slide… Mar 2011 Via prescale Via pass through Via rerun “Commissioning” signatures § § § Via prescale Via pass through Via rerun Elizabeth Gallas - COMA 17

RBR Chain Report (chain EF_e 10_loose) Expand Run-wise Activation … “Physics” EF-L 2 -L 1 signatures Runs in Data Periods Active via Prescale Table Shows: § § § Mar 2011 Period, Link: Run, SMK Reports Level bit assignments Link to: Chain/Item Reports (3) Range of Aggregate Prescale while chain is active via prescale in Run Links: COMA Prescale Report (3) Elizabeth Gallas - COMA 18

RBR Chain Wildcard Report (input: “EF_g 1%”) Purpose: See all the names matching a pattern or Find exact name from part of the name Report: Displays chain/item names matching the input string … text size proportional to occurrence in SMK § In Period Runs and in All Runs Mar 2011 Elizabeth Gallas - COMA 19

RBR Prescale Report (Run 167844, chain EF_e 5_tight) Run-wise Summary: § § § Includes link to COMA reports: § Run, Master Key, § Chain, Prescale (each level) Aggregate PS Range: 250 -1000 Flags: § Partially active via PS § Pass Through at L 2 § Never Rerun LB Range Summary table: § § LB-wise Prescale Evolution Link: Trigger. DB prescale reports Prescales: § L 1, L 2, EF and Aggregate Chain Inactive from § LB 1 -78 and 271 -287 (LB ranges in COMA always inclusive) § § Mar 2011 L 2 pass through N=315 § from LB 79 -270 EF Pass Through: Never Elizabeth Gallas - COMA 20

COMA Browser General Principles Topic 3 COMA Browser (AKA run. Browser) = interface for Run. LB selection using COMA § Purpose: Make conditions metadata available as selection criteria in advance of analysis … Envisioned as the Run-level browser for ELSSI … current implementation makes it also available stand-alone. § Intermediate results may be what the user is looking for I. E. show me the Runs taken on this date, during a Data Period, or w/this DQ tag. § Final output (clicking on “Finish” button): LB level criteria is applied at the final “Finish” stage. Output: A report showing the Run/LBs passing final criteria Output: An xml file (Good. Run. List) which can be used by ELSSI etc. § § § run. Browser IS NOT run. Query (browser to all online Runs in COOL) Enables not only Run selection by conditions criteria but also displays the possible values of remaining criteria and its relationship to other criteria Criteria can be imposed in any order … § § Where appropriate: § § § Mar 2011 some choices open selection to deeper criteria Allows radio, checkbox, or text (command line) entry of criteria Allow list and/or ranges of values, wildcards, case insensitivity. . . Incorporate features to customize rows displayed and other tricks to improve performance Elizabeth Gallas - COMA 21

COMA Browser Overview § § § Note: “Under Construction”! Each section expands/collapses showing the available values Blue links: § § Choose ANY criteria, ANY order, click SUBMIT § § pop up documentation run. Browser now tells you what is left … iterate until you are happy Selection Sections see next slide § Click on FINISH button when you are finished making selections Reports Run/LB passing Reports Run/LB failing (Unofficial) GRL xml § Mar 2011 Elizabeth Gallas - COMA 22

Selection Criteria Sections 1. Purpose / Instructions 2. Selection Summary 3. Starts out empty (stand-alone) Starts with Collection (ELSSI) Selection Criteria J. Uploaded Runs Temporal Selection Data Periods Data Source (data or MC) Run Type Project Name (Filename. Tag) DAQ Configuration Run Number Magnets (*) Stable Beams (*) K. Ready Status Flag (*) L. Trigger Master Key A. B. C. D. E. F. G. H. I. • M. Mar 2011 LVL 1/HLT PS Key Data Quality Elizabeth Gallas - COMA 23

COMA Browser Finish Button Output: § (Unofficial) GRL xml § § w/link to send to ELSSI A humanly readable report… § § § Table of Run LB Ranges List of criteria Report of why particular LB ranges failed DQ criteria -- This is a test release of "run. Browser” But the data content and functionality is still in development. – This version should give people an idea of what the system will be able to do… Ideas and feedback is very welcome! Mar 2011 Official GRL is described in another session of the tutorial. Elizabeth Gallas - COMA 24

COMA: Documentation and Links § COMA TWiki https: //twiki. cern. ch/twiki/bin/view/Atlas/Conditions. Metadata § TAG Services portal (contains links to ELSSI, Run. Browser): https: //atlas-tagservices. cern. ch/ § Run. Browser Portal: https: //atlas-tagservices. cern. ch/tagservices/Run. Browser/index. html § § § COMA Data Period Documentation Interface COMA Reports (run. Browser. Report) COMA Browser (run. Browser) Mar 2011 Elizabeth Gallas - COMA 25

COMA Summary and Future § The COMA project § § Enables/Provides unique interfaces for users with reports complementary to other interfaces: § Run. Query, Trigger DB, … interfaces provide much more detail in those areas Plans for more data in COMA: https: //twiki. cern. ch/twiki/bin/viewauth/Atlas/Conditions. Metadata. Plans § Data Periods is now incorporated into the COMA DB § § Report and Service interfaces available Futures ideas for Data Period extensions: § § Extend data content into other areas: § Data Containers, Official Good. Run. Lists, … Reports: Add luminosity, event counts, …by period Mar 2011 Elizabeth Gallas - COMA 26

COMA Conclusions § This is an evolving system … information in the system is growing based on information available and use cases § § § We want to insure the Metadata is § § § complete enough to satisfy use cases while reflecting accurately its limitations Interfaces are being constructed to use selection syntax, criteria, and communication in common use in ATLAS § § Adding more dimensions to the Conditions data § With suitable relationships to facilitate queries Making that criteria available in dynamic useable interfaces This facilitates cross checks with other systems Continuous process: talking with various experts to ensure § data integrity, completeness, compatibility w/other systems … Very positive feedback so far … more always welcome … Mar 2011 Elizabeth Gallas - COMA 27

Backup Mar 2011 Elizabeth Gallas - COMA 28

Some COMA “derived” prescale quantities “Aggregate Prescale”: LB Range-wise: TRIG_AGG_PS § AGG_PS = L 1 * L 2 * EF prescales, § § when all trigger levels are active (prescale >= 0). AGG_PS = -1 § For each • Signature • LB range in ps evolution AGG_PS PT_FLAG when any level PS <0 Run-wise: RUN_AGG_PS. MIN_PS, MAX_PS MIN_PS = MIN (AGG_PS), MAX_PS = MAX(AGG_PS) Of all AGG_PS >=1 MIN_PS = MAX_PS = -1, When all AGG_PS < 1 over the Run. For each • Signature • Run MIN_PS MAX_PS PS_FLAG PT_FLAG RR_FLAGS: Prescale, Pass. Through, Rerun (3 modes chains can be ‘active’): § PS_FLAG (at the Run level) = -1 / 0 / 1 chain active via prescale for none / part / all of Run. § PT_FLAG (at LB Range and Run level) = 0 / 1 / 2 / 3 no PT / PT at L 2 and EF / PT at L 2 / PT at EF. § RR_FLAG (at the Run level) = -1 / 0 / 1 indicates if the chain Never / Sometimes / Always can be RERUN during the Run (online). Mar 2011 Elizabeth Gallas - COMA 29

COMA trigger notes § All these notes (and more details) can be found in COMA Documentation § COMA divides signatures in two mutually exclusive categories: § “Physics” complete EF-L 2 -L 1 signatures with distinct name, counter, prescale § § “Commissioning” irregular/incomplete signatures (missing one/more of above) § § Does not imply all are written to “physics” streams ! Includes signatures with L 1 -OR’d conditions (never used for physics analysis) COMA derived Aggregate Prescales and PS_FLAG: § Do not take into account “Stream Prescales” § § Never applied to “Physics” streams Do not take into account effective Level 1 prescale in chains using L 1 -OR’s § These are not physics triggers anyway THEREFORE: Aggregate prescales for triggers writing to nonphysics streams may be underestimates § § Mar 2011 But this does not apply to chains used for physics So not highest priority now, may incorporate this later in stream reports Elizabeth Gallas - COMA 30

COMA Trigger info use in ELSSI: “Event Level Selection Service Interface” § The ATLAS TAG DB browser TAG DB (and TAG files) contain event-wise trigger decisions § § At all levels of the trigger At EF level, contains the ‘physics’ and ‘raw’ decision Workflow: ELSSI user selects the equivalent of one/more Runs 1. ELSSI looks to COMA tables to find § § chain/item name/number mapping For the “Physics Trigger” tab: § § The relationship between trigger levels Displays the “active” EF-L 2 -L 1 chains in the selected runs Also, there is a “Level 1 Trigger tab” showing all items 2. ELSSI: gives link to run. Browser. Report for selected Runs 3. ELSSI: can also launch run. Browser, not covered today § What does this look like in ELSSI ? ? ? Next slides … which will lead into the run. Browser. Report(s) … Mar 2011 Elizabeth Gallas - COMA 31

ELSSI “Trigger” tab, “Physics Chains” subtab Mar 2011 Elizabeth Gallas - COMA 32

ELSSI Link to COMA Reports ELSSI contains a link to the COMA Run Report: 1. User chooses the Collection Name (Period selection also available) 2. ELSSI shows Runs in the Collection 3. ELSSI displays link to pop-up run. Browser. Report Mar 2011 Elizabeth Gallas - COMA 33

Oracle Database: TAG DB and associated metadata tables § TAG DB Event-wise metadata tables § Stores Event-wise attributes: electron (Et, eta, phi …) § muon (Et, eta, phi …) … and references to RAW, ESD, AOD files § ORACLE DB 1. TAG DB: Event-wise metadata Source: Official data processing chain RAW ESD AOD TAGfiles TAGDB § § LB Event … Data Catalogue tables § Stores information on file and dataset processing and location § § Run Project name AMI tag (what processing occurred)… 2. DATA Catalogue Sources: AMI, Tier 0 … ‘COMA’ (COnditions Metadat. A) tables § Conditions of data taking § § § Trigger and DAQ conditions Beam conditions Data Quality … 3. COMA Sources: Conditions DB, files (log, xml, txt), email… § Mar 2011 Elizabeth Gallas - COMA RUNS 34

SOURCES COMA Schema Overview RUN_TYPES RUN_MAGNETS* MASKS RUN_FILLS* STREAM_LBN_EVENTS PH 2 L_OBS PD_OBS STREAM_EVENTS SHORTNAMES P 2 R_OBS PGRL_OBS PERIOD_DEFS FILLS* TRANSITIONS EXPLAINS PERIOD_GRLS PERIOD_PHI_TO_PLOS PERIOD_P 1_TO_RUNS EVENTS LOAD STATUS Master_Keys RUNS LBN_TIMES RUN_LUMS LBN_LUMS Run_Agg_PS COOL_CONNECTIONS MK_Triggers LUM_DETS VIRTUAL_FLAGS MKT_Physics Trig_Agg_PS Prescale_Keys VFLAG_DETS MKT_Others PS_Sets DETSTATUS TRIG_STR STREAM Mar 2011 LVL 1_PSK HLT_PSK LVL 1_PS HLT_PS Elizabeth Gallas - COMA DETECTORS CODES GROUPS 35

COMA History § History of Conditions Metadata … evolution § § § 2007: Develop Luminosity prototype & ELSSI prototype Providing Run wise criteria to ELSSI (MC and online) Split (developed first DTD for Good. Run. List XML): § § § Mar 2011 ELSSI – Event-wise selection browser Run. Browser – Run/LB-wise selection browser COMA Reports emerge COMA becomes storage for ATLAS Data Periods COMA DB and associated reports … growing and evolving. Elizabeth Gallas - COMA 36

Example demonstrates General Principles: § There is no prescribed order of selection or mandatory selections § Expand section of interest, make selection: § Iterate any number of times § available radio/checkbox or use the textbox to type a list or range of values Click Submit I chose Project “data 09_900 Ge. V”, then § I see there are 99 Runs left § I see their run and date range § I see the criteria has appeared in the selection summary � § § I see ALL the other sections have changed to reflect this criteria ! § Mar 2011 I could remove it with button click Look at the remaining 99 runs. . (next slide) Elizabeth Gallas - COMA 37