The History and Future of ATLAS Data Management

The History and Future of ATLAS Data Management Architecture D. Malon, S. Eckmann, A. Vaniachine (ANL), J. Hrivnac, A. Schaffer (LAL), D. Adams (BNL) CHEP’ 03 San Diego, California 24 March 2003

Outline T Persistent principles T Pre-POOL: The ATLAS event store architecture T Hybrid event stores T ATLAS and POOL T Non-event data in ATLAS T ATLAS data management and grids T ATLAS data management and other emerging technologies David M. Malon, ANL CHEP'03, San Diego 24 March 2003 2

Long-standing principles T Transient/persistent separation—not by any means unique to ATLAS— means (in ATLAS, anyway): q Physics code does not depend on storage technology used for input or output data q Physics code does not “know” about persistent data model q Selection of storage technology, explicit and implicit, involves only job options specified at run time T Commitment to use common LHC-wide solutions wherever possible, since at least the time of the ATLAS Computing Technical Proposal q Once this implied Objectivity/DB (RD 45) q Now this implies LCG POOL David M. Malon, ANL CHEP'03, San Diego 24 March 2003 3

Pre-POOL: The ATLAS event store architecture T Event collection is fundamental: processing model is “read one or more collections, write one or more collections” q Note: collections are persistent realizations of something more general T Model allows navigational access, in principle, to all upstream data q Output “event headers” retain sufficient information to reach any data reachable from input event headers T Architecture supports strategies for “sharing” data, so that writing events to multiple streams, for example, does not require (but may allow) replication of component event data David M. Malon, ANL CHEP'03, San Diego 24 March 2003 4

Pre-POOL: The ATLAS event store architecture - II T Data selection model is, in relational terms, “SELECT … FROM … WHERE …” q SELECT which components of event data? q FROM which event collection(s)? q WHERE qualifying events satisfy specified conditions q Designed to allow server-side and client-side selection implementation T Architecture also describes a placement service, though our implementations were rather rudimentary q Mechanism for control of physical clustering of events (e. g. , by stream), of event components (e. g. , by “type”), and for handling file/database allocation and management q Interface could satisfied by a within-job service, or by a common service shared by many jobs (e. g. , in a reconstruction farm) T “Extract and transform” paradigm for selection/distribution David M. Malon, ANL CHEP'03, San Diego 24 March 2003 5

Hybrid event stores T There were (and are) many questions about where genuine database functionality may be required, or useful, in an event store T STAR, for example, had already successfully demonstrated a “hybrid” approach to event stores: q File-based streaming layer for event data q Relational database to manage the files T A hybrid prototype (Athena. ROOT) was deployed in ATLAS in parallel with the ATLAS baseline (Objectivity-based) T Transient/persistent separation strategy supports peaceful coexistence; physicists’ codes remain unchanged …all of this was input to the LCG requirements technical assessment group that led to initiation of POOL… David M. Malon, ANL CHEP'03, San Diego 24 March 2003 6

ATLAS and POOL T ATLAS is fully committed to using LCG persistence software as its baseline, and to contributing to its direction and its development T This means POOL: see earlier talks for POOL details T What POOL provides is closer to a framework than to an architecture q …though architectural assumptions, both implicit and explicit, go into decisions about POOL components and their designs q The influence is bidirectional q Because POOL is still in its infancy, we do not fully understand its implications for ATLAS data management architecture T Not a criticism: even if POOL delivered all the functionality of Objectivity/DB (or of Oracle 9 i), LHC experiments would still need to define their data management architectures David M. Malon, ANL CHEP'03, San Diego 24 March 2003 7

Non-event data in ATLAS T While event and conditions stores may be logically distinct and separately managed at some levels, there is no reason they should not: q employ common storage technologies (POOL/ROOT, for example) q register their files in common catalogs T ATLAS currently has a variety of non-event data in relational databases (e. g. , the “primary numbers” that parameterize ATLAS detector geometry) q Today this entails ATLAS-specific approaches, but in principle, a POOL My. SQL(ODBC? ) Storage Manager implementation could be used, as could the LCG SEAL dictionary for data definition T For interfaces unique to time-varying data, e. g. , access based on timestamps and intervals of validity, ATLAS again hopes to employ common LHC-wide solutions where possible David M. Malon, ANL CHEP'03, San Diego 24 March 2003 8

Architectural principles for time-varying data T Separate the interval-of-validity (IOV) database infrastructure from conditions data storage. T Should be possible to generate and store conditions data in any supported technology (POOL ROOT, My. SQL, plain ASCII files, …) without worrying about the interval-of-validity infrastructure q Generation of data, and assignment of intervals of validity, versions, and tags, may be widely separated in time, and done by different people T Later register the data in the IOV database, assigning an interval of validity, tag, version, … T Transient IOV service will consult IOV database to get pointer to correct version of data, then invoke standard Athena conversion services to put conditions objects in the transient store David M. Malon, ANL CHEP'03, San Diego 24 March 2003 9

Transient Conditions Store 4. Build transient conditions object s rvice e s n rsio nve rd co a d n a st ce via ren refe 3. De Conditions data 2. Ref to data 1. Timestamp, tag, version IOV Database David M. Malon, ANL CHEP'03, San Diego 24 March 2003 10

Nonstandard data? T Cannot expect that all useful non-event data are produced by ATLASstandard tools T How do such data enter ATLAS institutional memory? q Is it as simple as registration in an appropriate file catalog (for files, anyway) managed by the ATLAS virtual organization? T Is there a minimal interface such data must satisfy? q ATLAS “dataset” notions are relevant here q Ability to externalize a pointer to an object in this technology? T What is required in order for an application in the ATLAS control framework (Athena) to access such data? q Provision of an LHCb/Gaudi-style conversion service? T LCG SEAL project may influence our answers to these questions David M. Malon, ANL CHEP'03, San Diego 24 March 2003 11

ATLAS data management and grids T With the current state of grid tools, grid data management has meant, primarily, file replica cataloging and transfer, with a few higher-level services T ATLAS has prototyped and used a range of grid replica catalogs (GDMP, EDG, Globus (pre-RLS), RLS, …), grid file transport tools, grid credentials T Principal tool for production purposes is MAGDA (ATLAS-developed) q MAGDA designed so that its components can be replaced by grid-standard ones as they become sufficiently functional and mature z This has already happened with transport machinery; will happen with replica catalog component as RLS implementations improve David M. Malon, ANL CHEP'03, San Diego 24 March 2003 12

Databases on grids T Need more detailed thinking on how to deal with database-resident data on grids q Do Resource Brokers know about these? q Connections between grid replica tracking and management and databaseprovided replication/synchronization, especially when databases are updateable T Have looked a bit at EDG Spitfire T Could, in some cases, transfer underlying database files via replication tools, and register them (a la GDMP with Objectivity/DB databases) q Have done some prototyping with My. SQL embedded servers T On-demand access over the net poses some challenges T Grid/web service interfaces (OGSA) should help David M. Malon, ANL CHEP'03, San Diego 24 March 2003 13

Recipe management, provenance, and “virtual data” T Every experiment maintains, via software repositories and managed releases, official versions of recipes used to produce data T Everyone logs the recipes (job options, scripts) used for official collaboration data production in a bookkeeping database q ATLAS does this in its AMI bookkeeping/metadatabase T Virtual data prototyping has been done in several ATLAS contexts q Parameterized recipe templates (transformations), with actual parameters supplied and managed by a database in DC 0/1 (derivations) z See Nevski/Vaniachine poster q Similar approach in At. Com (DC 1) q Gri. Phy. N project’s Chimera virtual data catalog and Pegasus planner, used in SUSY data challenge, and (currently) for reconstruction stage of Data Challenge 1 David M. Malon, ANL CHEP'03, San Diego 24 March 2003 14

Provenance T “Easy” part of provenance is at the “job-as-transformation” level: q What job created this file? q What job(s) created the files that were input to that job? q …and so on… T But provenance can be almost fractal in its complexity: q An event collection has a provenance, but provenance of individual events therein may be widely varying q Within each of those events, provenance of event components varies q Calibration data used to produce event component data have a provenance q Values passed as parameters to algorithms have a provenance q … David M. Malon, ANL CHEP'03, San Diego 24 March 2003 15

Some provenance challenges T CHALLENGES: q Genuinely browsable, queryable transformation/derivation catalogs, with sensible notions of similarity and equivalence q Integration of object-level history tracking, algorithm version stamping, …. , (currently experiment-specific), with emerging provenance management tools David M. Malon, ANL CHEP'03, San Diego 24 March 2003 16

The metadata muddle T Ill-defined: One man’s metadata is another man’s data T Essential, though: a multi-petabyte event store will not be navigable without a reasonable metadata infrastructure T Physicist should query a physics metadatabase to discover what data are available and select data of interest T Metadata infrastructure should map physics selections to, e. g. , lists of logical files, so that resource brokers can determine where to run the job, what data need to be transferred, and so on T Logical files have associated metadata as well T Some metadata about provenance is principally bookkeeping, but some is as useful as physics properties to physicists trying to select data of interest David M. Malon, ANL CHEP'03, San Diego 24 March 2003 17

Metadata integration? T Current ATLAS data challenge work distinguishes 1. ) physics metadata, 2. ) metadata about logical and physical files, 3. ) recipe/provenance metadata, 4. ) permanent production bookkeeping, and 5. ) transient production log data, as a starting point T It is not too hard to build an integrated system when the components are all under the experiment’s control, but when replica metadata management is coming from one project, provenance metadata management from another, physics, perhaps, from the experiment itself, bookkeeping from (perhaps) still another source, a system that supports queries across layers is a CHALLENGE T …in ATLAS, we still do not quite have a system that lets a physicist choose a physics sample as input, and emits EDG JDL, for example, and this is just a small step David M. Malon, ANL CHEP'03, San Diego 24 March 2003 18

Beyond persistence T Persistence—saving and restoring object states—is a minimalist view: it is necessary, but is it sufficient? T Should the object model (in transient memory) of the writer determine the view that clients can extract from an event repository? T “Schema evolution” does not suffice: q Schema evolution recognizes that, though I write objects {A, B, C, D} today, the class definitions of A, B, C, and D may change tomorrow q It fails to recognize that my object model may use entirely different classes {E, F, G} in place of {A, B, C, D} next year T Simple persistence fails to acknowledge that readers may not want the same objects that writers used, and that not all readers share a single view David M. Malon, ANL CHEP'03, San Diego 24 March 2003 19

Beyond persistence: a trivial example T Can a reader build a “simple” track (AOD track) from a “full track” data object (the saved state of an object of a different class), without creating an intermediate “full track” on the client side? T In a relational database, I can write a 500 -column table but read and transfer to clients the data from only three of the columns T Simplified view: Need an infrastructure that can answer the question, “Can I build an object of type B from the data pointed to by this persistent reference (perhaps the saved state of an object of type A)? ” David M. Malon, ANL CHEP'03, San Diego 24 March 2003 20

LCG Object Dictionary: Usage (diagram thanks to Pere Mato). adl. xml . h Population ROOTCINT GCC-XML CINT generated code Dict generating code in (1) CINT Dict ROOT I/O LCG to CINT Dict gateway Conversion Streamer ADL/GOD LCG Dictionary (2) out Other Clients: (python, GUI, etc. ) Reflection Requires elaboration David M. Malon, ANL CHEP'03, San Diego 24 March 2003 21

Elaboration? T Selection of persistent representation and streamer generation can be separated q More than one persistent representation may be supported q Custom streamers q Separation of transient object dictionaries and persistent layout dictionaries T On input, what one reads need not dictate what one builds in transient memory Not “Ah! This is the state of a B; I’ll create a transient B!” q Rather, “Can I locate (or possibly create) a recipe to build a B from these data? ” Conversion Streamer CINT Dict LCG to CINT Dict gateway q ROOT I/O David M. Malon, ANL CHEP'03, San Diego 24 March 2003 22

Other emerging ideas T Current U. S. ITR proposal is promoting knowledge management in support of dynamic workspaces T One interesting aspect of this proposal is in the area of ontologies q An old term in philosophy (cf. Kant), a well-known concept in the (textual) information retrieval literature, and a hot topic for semantic web folks q Can be useful when different groups define their own metadata, using similar terms with similar meanings, but not identical terms with identical meanings q Could also be useful in defining what is meant, for example, by “Calorimeter data, ” without simply enumerating the qualifying classes David M. Malon, ANL CHEP'03, San Diego 24 March 2003 23