CHEP 2000 Padova February 7 11 2000 Distributing
CHEP 2000 Padova, February 7 -11, 2000 Distributing Data around the Ba. Bar collaboration’s Objectivity Federations Emanuele Leonardi - I. N. F. N. Roma Simon Patton - L. B. N. L. on behalf of the Ba. Bar Data Distribution group 9/2/2000 E. Leonardi, S. Patton CHEP 2000
Objectivity Federations (FD) z On-site z Off-site y DAQ: IR 2 Boot y Prompt Reco: Prod. Boot y Analysis: Anal. Boot y Reprocessing: Reco. Boot y MC Production: SP 2 Prod. Boot y MC Analysis: SP 2 Anal. Boot y Ba. Bar Regional Centers (CCIN 2 P 3, INFN, RAL) copy (part of) Anal. Boot and SP 2 Anal. Boot y MC Production Centers write data to SP 2 Prod. Boot • How do we transfer data among these FD’s? • How do we ensure consistency in the data? 9/2/2000 E. Leonardi, S. Patton - CHEP 2000 2
Data Transfer z A simple file transfer is not possible y FD must be “told” about the presence of new data y FD structure must be updated z Ad hoc tools (Bdb. Dist. Tools) were developed y Based on Ba. Bar database structure and organization y Assume total correspondence between FD’s DB Id’s (see later) y By-pass standard Objectivity DB administration tools y Only bulk data transfer for the moment 9/2/2000 E. Leonardi, S. Patton - CHEP 2000 3
Data Export z Select data formats to export (AOD, ESD, …) z Check for modified/new DB files z Export these files (by-pass oocopydb: too slow) y Lock full DB y Copy file to stage-out area y Unlock DB z Get DB ID’s corresponding to exported files z Create TDF (Transfer Description File) with all info needed to re-attach the DB y DB file name, DB Id, Size, . . . 9/2/2000 E. Leonardi, S. Patton - CHEP 2000 4
Data Import z Copy all exported files (network or tape) to stage-in area z Use TDF to identify each file z Copy files to correct position in FD structure y Depends on local FD lay-out z Attach files to FD y ooattach makes a consistency check but it is VERY SLOW, so. . . y create empty DB file with correct name and position y ooattach the empty file (VERY FAST) y copy the real DB over the empty one 9/2/2000 E. Leonardi, S. Patton - CHEP 2000 5
DBID Allocation z A “FD family” is a group of FD’s sharing part of the data y IR 2 Boot, Prod. Boot, Reco. Boot, Anal. Boot, “off-site”-Anal. Boot y Sp 2 Prod. Boot, SP 2 Anal. Boot, “off-site”-SP 2 FD’s z To avoid overwriting of shared DB’s, each FD contains: y A “Global DB” area xall DB’s which may be shared with other FD’s in the same family y A “Local DB” area xall DB’s which are unique to that FD. z Each FD has write access to the Local area and to a preallocated portion of the Global area 9/2/2000 E. Leonardi, S. Patton - CHEP 2000 6
DB Id’s allocation schema Prod. Boot INFN-Anal. Boot 9/2/2000 Prod. Boot Area INFN Anal. Boot Area Global DB DB Id’s Local DB Anal. Boot Reco Data Off-line Processing Part of Reco Data Production Skims User Skims E. Leonardi, S. Patton - CHEP 2000 Off-site User Processing 7
Conclusions z Use of multiple Objectivity FD’s in different sites requires ad hoc tools to move data around. z Objectivity Administration tools are usually too slow: Ba. Bar developed its own set of utilities to handle data distribution. These tools assume a complete correspondence between FD’s structures. z A careful management of DB Id’s is mandatory in order to avoid DB overwriting during data distribution. 9/2/2000 E. Leonardi, S. Patton - CHEP 2000 8
- Slides: 8