AMI and its place in ATLAS Metadata Solveig

AMI and its place in ATLAS Metadata Solveig Albrand Jerome Fulachier Fabian Lambert 9/10/2020 S. A. 1

AMI Part I 1. Introduction to AMI 2. Data sources 3. Known problems S. A. 2

AMI Brief Intoduction to AMI ATLAS Metadata Interface A generic catalogue application. Used by several ATLAS applications. Application specific layer Datasets SQ NICOS Tag. Collector AMIWhen you make a Generic functions catalogue in AMI you immediately get a web Connections browser interface, a python CLI + generic query and data ORACLE my. SQL manipulation functions. S. A. 3

AMI Interfaces ami. in 2 p 3. fr py. AMI data. Puller • Web interface : • Browsing + all server commands (add, clone, update) if authorised. • Personal pages with bookmarks • py. AMI : • A SOAP web service + examples of use in the ATLAS release. All server commands accessible. • Data Puller Tasks (see next slide) S. A. 4

AMI Data Puller Tasks • Run in a separate instance of AMI. • Each task runs in its own thread. • Task. Threads managed by a “master” thread and database tables. – Can modify frequency of tasks, and mutual exclusion dynamically. • Email alerts • Watch dog mechanism. S. A. 5

AMI Features • Uses "MQL" from GLite. Users do not need to know the relational structure of the tables. • A powerful graphic query builder for ad-hoc queries. • Multi-threaded queries and "natural" partioning. • Supports some schema evolution. • Indirect DB connections. RDBMS & database site transparent for user. • Efficient connection pool and transaction management. • Exploits prepared statements. • Manages its own user database, and autorisation. (users roles commands catalogues). VOMS proxy upload possible. • Possibility to browse third party databases if they have a simple relational structure. (Primary key/ Foreign key) S. A. 6

AMI OVERVIEW PAGE AMI catalogues OFFICIAL datasets (data_*, mc_*, and SOME physics group datasets. No USER datasets as yet. By default : - • No searching in "archived" catalogues. • Datasets known to be bad are hidden. S. A. 7

AMI Search Results S. A. 8

AMI is a MEDIATOR interface • • “a software module that exploits encoded knowledge about some sets or subsets of data to create information for a higher layer of applications”. (Gio Wiederhold, Mediators in the architecture of future information systems, IEEE Computer Magazine, Vol 25, No 3, p 38 -49 March 1993) To put it another way, a mediator is an application which puts some domain expertise between the user and a group of data sources, so that information coming from these different sources can be aggregated. S. A. 9

AMI Sources of AMI data • Real data : From the Tier 0 : DAQ data, and first reconstruction is registered in AMI < 5 minutes after it is declared “ready” (both datasets and files). • Monte Carlo and reprocessing. – From the Task Request DB : Tasks, dataset names, MC and reprocessing configuration tags ("AMI tags") – From the production DB : Finished tasks – files and metadata. • From physicists with AMI writer role. – M. C. GEN datasets – MC Dataset number info, physics group owner, … – Corrected cross sections and comments. (coordinated by Borut K and Claire Gwenlan. ) – Tier 0 configuration tags (AMI tags for real data. ) – … N. B. I have neglected Tag Collector here and in what follows – for clarity 9/10/2020 S. A. 10

AMI Tier 0 "AMITags" are written in AMI before use Ready Tier 0 Done This AMI task reads datasets, files and some metadata from Tier 0 using a semaphore mechanism. AMI Simple and efficient – AMI only get datasets when they are completed. S. A. 11

AMI AKTR (Task Request) AKTR Read AMITags AMI 1. Reads new tasks, once they are no longer pending, if they are not aborted before submission. 2. Reads provenance, 3. Reads updates to a bad status. 4. Reads MC and reprocessing AMITags. S. A. 12 5. N. B. Does NOT read finished/done status here.

AMI Prod. DB AMI 1. Follows RUNNING status of tasks. 2. When FINISHED reads metadata. xml for all jobs of the task. S. A. 13

AMI DQ 2 nomenclature DQ 2 Info Data deleted AMI S. A. 14

AMI PROBLEMS • Problems we've been dealing with for such a long time that it's almost not worth mentioning them, as I fear that in some cases nothing will ever get done. "Minor Problems" • Mechanism problems. Major problems we really SHOULD do something about • Content problems. I am not competent to say anything about these, so I won't. S. A. 15

AMI Minor Problems • Tier 0 : No update mechanism. (Have not needed one up to now. ) • AKTR : – Provenance very acrobatic. (c. f. AMI Review 2006). Input Dataset column in AKTR DB very messy. – No "dataset container" "task number" link • General : No finite state description as far as I know. We had to guess. It is sometimes confusing for users. Dataset states need some work from the collaboration. S. A. 16

AMI Important Problems - 1 • Validation of metadata. xml – Validation of metadata output is not part of release validation, and it should be. – Is it entirely reliable even when it works? There have been many examples of one job out of many which fails to produce metadata. Why are these jobs still declared "done"? The result is that a wrong number of files/events is reported. (https: //savannah. cern. ch/bugs/? 70591 It's because we run with "--ignoreerror=True". Pavel suggested how to fix it. Alvin agreed. Will anything get done? ) S. A. 17

AMI Important problems - 2 • Relation between TRANSFORMS (e. g. https: //twiki. cern. ch/twiki/bin/view/Atl as/Reco. Trf ) and METADATA. – An improved grammar for metadata. xml (task aware so more compact) is available, but has never been put into production. – Manpower - Who is going to replace Alvin Tan? S. A. 18

AMI Important problems - 3 • Lack of coherence between AMI and DQ 2 for dataset container contents (and status). – At the moment AMI guesses that a finished TID dataset will be placed in its container by the production system when the task is declared FINISHED, using information from AKTR/Prod. DB) This is sometimes the case, but not always. What is the logic? Is it documented? – Solution Stop guessing and get the info from DQ 2. Have not done this up to now, but currently working on improving AMI DQ 2 information exchange using ACTIVE MQ server. Need confirmation that this can go into production. • Need a policy for file status, but that can come later when we have more Active MQ experience. S. A. 19

AMI Requirements • Our requirements come from many sources. – – – – Data Preparation coordination + physics groups. Reprocessing Coordination Metadata coordination MC generator group Fast Reconstruction/Tier 0 DA tools (Ganga) ADC (DQ 2) User suggestions. • Coming soon : – better treatment of Physics Containers, – better support for physics groups, – User datasets ? ? ? • Reported until recently to DB, and "close contact" with Data preparation (AMI review recommendations). Now in ADC dev (as well as? instead of ? ) S. A. 20

AMI Part II The ATLAS metadata context S. A. 21

AMI Other Metadata Applications in ATLAS • There are many metadata applications in ATLAS and many granularities of data/metadata. • Metadata Task Force Report 2006/2007 S. A. 22

AMI Metadata use (MTF report) • To determine the status of a process such as calibration or analysis (e. g. stage of processing). • To provide statistics on some process without having to read in large amounts of data (e. g. run completed gracefully or terminated abnormally). • To provide early indications of problems and clear indications of their source (e. g. detector status summary). • To allow the cross referencing of data in different data stores or different formats (conditions, events, TDAQ), (POOL, n-tuple), … • To locate other types of data. • To encapsulate the state of a process so that it can be repeated or easily modified (e. g. versions of software and configuration used for a process). S. A. 23

AMI Granularity of metadata (MTF Report) Event ; 1 File ; 102 – 104 Luminosity Block ; 104 Run ; 105 -106 Datastream per run; 104 – 106 Dataset ; 105 – 109 "A means to navigate between these granularities of scale will be needed to support queries where the user indicates a requirement on a given granularity but where the data needed is stored in a different granularity. " S. A. 24

AMI Metadata Flow (MTF Report) • From the detector to analysis • From the MC production to analysis • Through different granularities S. A. 25

There are many metadata applications in ATLAS AMI • Developed independently. • Different positions in the metadata flow. • Different DB structures and interface choices. • Links are ad hoc – these has never been any concerted design or effort to develop an overall metadata structure (with the notable exception of dataset nomenclature). S. A. 26

AMI It is too complicated to try and classify all of them by granularity. Application event file LB Run Stream Dataset And not very useful! S. A. 27

AMI User Data Discovery Intefaces • The AIM of AMI is to show physicists the DATASETS which are available for analysis, and to allow SEARCHING on PHYSICS criteria. • There are two other important ATLAS mediator applications for Data Discovery at other granularities. – Run. Query (Runs) – ELSSI/TAG DB (Events) • Granularity Clouds ? (c. f. Grid architecture) S. A. 28

AMI Overview of Data Discovery in ATLAS "Navigation Parameters" run. Number Dataset event. Number dataset. Name AMI ELSSI Run. Query Run S. A. Event 29

AMI Inside the Dataset cloud AMI DQ 2 AKTR PRODDB PANDA S. A. Tier 0 30

AMI Links from (and to) AMI SVN/NICOS Run Query Run Summary COMA Twiki AMI ELSII Geometry GANGA p. AThena Trigger Config DB Prod. DB S. A. Panda 31

AMI Links from Run Query (taken from RQ tutorial) The actual call of Atl. Run. Query after parsing Which criterion are applied Total number of runs / events selected and execution time Table with runs selected with show results (run, links, LB, time, #events always shown) Links to other run information (DS, BS, AMI, DQ, E-log) S. A. Summary (where useful) 32

AMI TAG DB Structure & Processes COOL COMA Run Fill Progs Trigger DB DQ 2 Conditions Metadata DB (MC and Prod) Repro Data Loader AMI TAG Upload Tier-0 10/09/2020 TAG DB Application and Service Knowledge DB POOL Relational Collections S. A. Slide from Florbela Viegas, SW week July 2010 33 33

AMI Some "Blue Sky" Questions • Remove the "ad-hoc" mechanisms for communication? – Generalise Active MQ communication between applications? • Move to an ATLAS wide query grammar for data discovery? – "Ontology" (http: //en. wikipedia. org/wiki/Ontology_%28 computer_science%29) • A common vocabulary for sharing information about a domain. It includes definitions of the basic concepts in the domain, and the relations among them. Each actor would then implement its own adaptor to translate to and from the common language S. A. 34

AMI Each application implements an adaptor to a common language. Result User Query AMI METADATA PORTAL Result Aggregation Metadata Dictionary Query Builder ATLAS METADATA ONTOLOGY Translations done by each application S. A. The Good Run List is almost a prototype for this! 35

AMI ? • A very interesting long term project for a real computer scientist, but do we really need it? • Implies a common desire to achieve such an interface. S. A. 36

AMI Conclusions. • It is important that each mediator interface is well anchored in its granularity cloud. • It is equally important to keep good coordination between them. • The physicist user is the boss! Make it easy for him/her. AMI ELSSI Run. Query S. A. 37

AMI Extra slides S. A. 38

AMI Key for interface diagrams ami. in 2 p 3. fr py. AMI Data Puller Tasks Key AMI links to trigger. DB AMI reads from prod. DB AMI AMI writes to Tier 0 AMI S. A. 39

AMI Granularity MTF selection Event 1 2872903475 1 File 102 – 104 631715 4548 Luminosity Block 104 No easy way of counting Run 105 -106 5006 573892 stream per run 104 -106 126 ? Dataset 105 – 109 9304 308781 S. A. 40

AMI Container incoherence example • Task 73194, marked "finished" in AKTR 2010 -04 -22 11: 03: 54. 0 (actually was probably finished on 2009 -07 -16 09: 44: 24) • All containers are empty. – data 09_cos. 00121238. physics_RPCw. Beam. recon. TAG_COMM. r 733/ – data 09_cos. 00121238. physics_RPCw. Beam. recon. NTUP_MUONCALIB. r 733/ – data 09_cos. 00121238. physics_RPCw. Beam. recon. ESD. r 733/ – data 09_cos. 00121238. physics_RPCw. Beam. recon. AOD. r 733/ – data 09_cos. 00121238. physics_RPCw. Beam. recon. log. r 733/ – data 09_cos. 00121238. physics_RPCw. Beam. recon. HIST. r 733/ • But the TID datasets exist in DQ 2 (i. e. the data is not deleted) • Example : data 09_cos. 00121238. physics_RPCw. Beam. recon. TAG_COMM. r 733_ tid 073194 (3827 files) S. A. 41