ATLAS Metadata Interfaces AMI and ATLAS metadata catalogues

ATLAS Metadata Interfaces (AMI) and ATLAS metadata catalogues Solveig ALBRAND Jerome FULACHIER LPSC Grenoble 3/12/2021 1

Brief History of AMI • Spring 2000 : Electronic notebook for ATLAS LAr testbeam. • 2001: DC 0 bookkeeping (and Tag Collector ) • 2002 - 2003 : DC 1 bookkeeping (and Ex. Tra) • 2004 : CTB and DC 2 bookkeeping. Tag Collector II Need to support many DB applications reuse as much software as possible 3/12/2021 CHEP 04 (Interlaken) 2

Some principles • · The software should be independent of the particular DBMS used. • · It should be possible to manage database schema evolution. • · The interfaces should be as generic as possible so that software is reused. • · The software should not depend on a particular operating system. • · The system should support geographic distribution. 3/12/2021 CHEP 04 (Interlaken) 3

Consequences • Use JAVA. (multi OS, JDBC, Web Services, Reflection, Many packages publicly available) • Use RDBMS + SQL. Start with my. SQL. • Use multi-layered design, and self describing databases (“AMI Compliant”). • Make most of it generic, but also have application specific sw. 3/12/2021 CHEP 04 (Interlaken) 4

Schematic View of the Architecture DB specific plugins 3/12/2021 CHEP 04 (Interlaken) 5

AMI Client Server model There are 3 possibilities for the client: • A Web Services client (SOAP). • From a browser (HTTP) using the AMI web search page, or from a command line. • By installing the AMI core software on the client side. 3/12/2021 CHEP 04 (Interlaken) 6

AMI Client Server Model II 3/12/2021 CHEP 04 (Interlaken) 7

Main Applications of AMI • ATLAS Production bookkeeping (physics metadata) - see later. • Tag Collector: a tool for release management– testimonials from former Release Coordinators – “The best thing since sliced bread…” – “The only piece of ATLAS sw which works. . ” – “The centre of ATLAS offline computing…” At least one of these statements is untrue – but visit the poster![133] • Ex. Tra – nothing to do with ATLAS – but uses AMI. Analysis of network traffic by interrogation of routers. (http: //lpsc. in 2 p 3. fr/informatique/reseau/extra/) 3/12/2021 CHEP 04 (Interlaken) 8

The EXTRA Interface 3/12/2021 CHEP 04 (Interlaken) 9

Bookkeeping Applications • ATLAS data challenges and Combined Test Beam • Different schema, sometimes evolving rapidly. • Access by the unique Logical Dataset Name. Threaded cross schema queries implemented. • Insert/update/query commands mostly generic. • Different sub projects need different parameters for searching. They are configurable in AMI • Web interfaces can have different links (e. g. MAGDA URL for DC 1 files, Don Quijote WS for DC 2 files), and actions can be associated with records. Use JAVA reflection. Generic approach has served us well! 3/12/2021 CHEP 04 (Interlaken) 10

Generic Pros and Cons • Rapid development • Schema evolution • Eases multi RDBMS support • Many applications supported • Slower • Possibly miss out on DB optimization • Coding more complicated • Interfaces too rich? : - ( : -) 3/12/2021 CHEP 04 (Interlaken) 11

3/12/2021 CHEP 04 (Interlaken) 12

3/12/2021 CHEP 04 (Interlaken) 13

3/12/2021 CHEP 04 (Interlaken) 14

3/12/2021 CHEP 04 (Interlaken) 15

So what about performance? • Cannot compare easily AMI with a simple SQL query, because there is greater functionality – but can measure its performance, and see if it meets requirements. (What ARE requirements? ) • Needed to do this anyway because of Web Services – everybody says SOAP is slow. And they are right! 3/12/2021 CHEP 04 (Interlaken) 16

SOAP Overhead Time SOAP HTTPS CMD Message size 3/12/2021 CHEP 04 (Interlaken) 17

SOAP Overhead (detail) Time SOAP CMD HTTPS Message size 3/12/2021 CHEP 04 (Interlaken) 18

We Like Web Services! • Even though they are slow there are many advantages. Ours is very simple. • Makes bug fixes, and extensions very easy as no new clients need to be distributed. • The web service was used by the CTB team for input using a command line interface and a C++ program to transfer data from Cond DB (JAVA client) • A python client is available – used by Ganga team. Also C++, PHP and RUBY. • We are waiting for ARDA/EGEE/OGSA/GODOT to come up with another WS implementation. Until then we’ll stick with SOAP (and try to optimise AMI code) 3/12/2021 CHEP 04 (Interlaken) 19

COMING SOON • ATLAS Distributed Analysis has sophisticated requirements for cataloguing. We expect to have a first implementation very rapidly. We will implement the g. Lite metadata catalogue interface. • ORACLE implemementation before end 2004. (and perhaps Postgres? ) • New (shifter driven? ) interface design. • Application level user authorization. (Based on mapping users to roles) • First model of distribution during 2005. Hope to “piggy-back” on D 3 3/12/2021 CHEP 04 (Interlaken) 20

For more information • The AMI Web Portal http: //ATLASbkk 1. in 2 p 3. fr: 8180/AMI Contains links to the generic web search interface, all documents, AMI wiki, etc. . Also links to Tag Collector. 3/12/2021 CHEP 04 (Interlaken) 21

Thanks for your attention ! The role of the audience Presentations involve both a speaker and the audience. People in the audience play a role in how well a presentation goes. People in the audience have an obligation to: • Listen politely • Make occasional eye contact with speaker • Take notes or jot down interesting facts • Control negative facial expressions • Control bored body language • Do not put your head down on the desk or tilt your head back to sleep • Control the impulse to constantly check watch • Expect a Question & Answer period to be part of the presentation • Participate in Question & Answer period – either by listening or by posing a question. • Prepare to remain attentive throughout the Q&A – speakers will dismiss their audience • Remain seated until the speaker is finished 3/12/2021 CHEP 04 (Interlaken) 22