STOREGLTE A Data Model Toolkit for the Atlas

STOREGLTE A Data Model Toolkit for the Atlas Software Architecture Paolo Calafiura, C. Leggett, H. Ma, D. Quarrie, S. Rajagopalan CHEP 2003, Mar 24 // session category 7 Store. Gate, a Data Model for the Atlas. . .

Event Data Model Analysis H H HHH DD D D D T T T “The Event” Tracking Hit Reconstruction CHEP 2003, March 24 // session category 7 Store. Gate, a Data Model Toolkit… 2

The Blackboard Pattern operates on Knowledge Source execute Data Store read write wipe activates Control Framework loop next. Source adapted from the ”Siemens” Patterns book CHEP 2003, March 24 // session category 7 Store. Gate, a Data Model Toolkit… 3

Data Store Pros and Cons Tracking Da ta Write Track. Coll Control Flow Separates stable data from evolving algorithms l Algorithms can be independently added, updated or removed from a running system l Event Store Read Track. Coll Analysis CHEP 2003, March 24 // session category 7 Flo w Track Collection Algorithms need to locate data in the store l Control Framework must be configured correctly to reach the desired solution l Store. Gate, a Data Model Toolkit… 4

What is Store. Gate? l. A type-centric Data Store ¡ Store. Gate. Svc: Data Store client API (Gaudi service) ¡ Data Object virtual proxy with lazy instantiation l. A set of Data Object design tools ¡ Handle for “automatic update” of time-varying data ¡ Persistable inter-object references ¡ Polymorphic STL-like Collections with defined element ownership ¡ Memory pool for optimized memory allocation CHEP 2003, March 24 // session category 7 Store. Gate, a Data Model Toolkit… 5

Store. Gate. Svc: Type-centric Store. Gate. Svc retrieve(type, key) record(type, key) Store. Map Type Templated read/write Proxy. Map All types in store Key All instances of a given type Data. Proxy Virtual proxy Data Object CHEP 2003, March 24 // session category 7 Store. Gate, a Data Model Toolkit… 6

Identifying Data Objects My Tracking Write “My. Tracks” Event Store Read “My. Tracks” My Analysis CHEP 2003, March 24 // session category 7 pro l Data ve n an ce “My. Tracks” Track. Collection “Your. Tracks” Track. Collection Read “Your. Tracks” object type l User defined key (well-known name) l Default data object l Use Data Object History (provenance) ¡ Reduce user-defined keys ¡ Not yet implemented Your Analysis Store. Gate, a Data Model Toolkit… 7

Store. Gate. Svc API To record (write) a data object Status. Code sc = p_sg. Svc record(p. Trk. Coll, Trk. Coll. Key); To retrieve (read) a data object p_sg. Svc retrieve(p. Trk. Coll[, Trk. Coll. Key]); p_sg. Svc retrieve(begin. TC, end. TC); where : ¡ ¡ ¡ p. Trk. Coll is a (const) pointer to a Track. Collection Trk. Coll. Key is an identifier for a data object. It can be a simple string like: “My. Track. Collection” begin. TC, end. TC are iterators over all Trk. Coll instances in the store CHEP 2003, March 24 // session category 7 Store. Gate, a Data Model Toolkit… 8

Control Object Access and Creation IConverter l Data. Proxy ¡ ¡ Virtual Proxy that collaborates with Gaudi (one day, POOL) transientpersistent conversion mechanism In the future may use history info to reconstruct data object on-demand Lazy data object instantiation (on client access) Support/enforce const-correctness l Store. Gate. Svc allows to “freeze” a data object ready to be used or written out CHEP 2003, March 24 // session category 7 Store. Gate, a Data Model Toolkit… create. Obj create. Rep Data. Proxy access. Data creates Data. Object 9

Data. Handle for Time-Varying Data l Purpose: ¡ Hide store dynamics from client code Handle is bound to Data. Proxy during initialization l Automatically updated (again using lazy instantiation) when Data Object enters a new validity range l l Use cases: ¡ alignment data ¡ calibration objects ¡ detector description information ¡ anything which has an associated validity range CHEP 2003, March 24 // session category 7 Store. Gate, a Data Model Toolkit… 10

Asynchronous update of Calibration Data Alg Handle Storegate Proxy PPSvc IOVDb. Svc Pers. Svc Cond. DB Nova. Cnv initialize reg. Handle(handle, key) bind. Handle(handle) reg. Proxy(proxy) reg. Fcn(this, &Fcn, handle, key) T* access. Data() Data. Obj* update. Address(TA) IOA get. Data(key) IOV, string execute set. Range(cl. ID, key, IOV) set. Address(IOA) build. IOA(string) IOA build. IOA(string) partial IOA create. Obj(IOA) Data. Obj* T* asyn c reset() CHEP 2003, March 24 // session category 7 reset() callback(db. Key) Store. Gate, a Data Model Toolkit… 11

Work with User Types l What is a Data Object? An object recorded into Store. Gate (SG) ¡ Only constraint is to be an “Assignable” ¡ No base class for SG data objects l SG uses generic programming to work with user types adapting to the interface they provide (“export”) l Same strategy used for SG collections and inter-object relationships CHEP 2003, March 24 // session category 7 Store. Gate, a Data Model Toolkit… 12

Inter-Object Relationships Need Persistable Ptr that Store. Gate. Svc ¡ is <<Data. Object>> Hit. Collection registry unknown <<Element>> Hit CHEP 2003, March 24 // session category 7 Track technology-independent ¡ does not require Hit to derive from any base class ¡ supports deferred access, creating the pointed-at object upon dereferencing ¡ allows to point at SG primary objects (Data Objects) l SG secondary objs (elements of a Data Object) l Store. Gate, a Data Model Toolkit… 13

Data. Link and Element. Link Storage. Policy Indexing. Policy get. Data. Obj locate. Element Policy-based design (parameterized inheritance) l Element. Link is a composition of Host policies Storage. Policy Data. Link l Default Indexing get. Data. Obj Policy generated according to host container type l CHEP 2003, March 24 // session category 7 Store. Gate, a Data Model Toolkit… Host Storage. Policy Indexing. Policy Element. Link get. Element Host Generate. Indexing Policy 14

The Need for Speed l Atlas L 2 and EF are using Athena & SG ¡ l random access record raw-data object access Speed-up > x 10 ¡ ¡ l share Algorithms and Services with offline Performance Studies ¡ l Fixed number of digits/event (1 M) Optimized containers Key hashing random access perfect hash A lot of work CHEP 2003, March 24 // session category 7 Store. Gate, a Data Model Toolkit… 15

Back to the Drawing Board Missed reqs and implementation surprises l Writing concrete and reading abstract ¡ “Symlinks” (Concrete DObj) (ABC) l l Templates and physical design ¡ Use PIMPL idiom to hide template implementation l l Proper solution when reflection available Keep libraries and executables small Templates and compilers ¡ Undetected compilation errors l ¡ unit tests also improve compilation coverage Unreadable compiler msgs (100+ lines) l CHEP 2003, March 24 // session category 7 boost concept-checking and compiler assertions Store. Gate, a Data Model Toolkit… 16

Status and Outlook l SG blackboard is a mature product ¡ ¡ Used extensively by Atlas developers offline and online Performance and memory footprint meet requirements Data Object tools (links, containers, Data. Handle) complete but newer: will need some fine tuning l Need tools to exploit Athena History Objects l to Identify Data Objects ¡ (perhaps) to steer Reconstruction-on-demand ¡ l Integration with LCG New data link storage policy based on POOL Refs ¡ Use SEAL Dictionary for Inheritance info ¡ CHEP 2003, March 24 // session category 7 Store. Gate, a Data Model Toolkit… 17

18

Store. Gate. Svc in Athena/Gaudi Application Manager Message Service Job. Options Service Store. Gate. Svc Detector Store Scripting Service Data Files Persistency Service Data Files T T T Event Store Other Services CHEP 2003, March 24 // session category 7 H H HHH Persistency Service DDD D D D Algorithm Particle Prop. Service Auditors Converter Event Loop. Mgr Sequencer Histogram Service Store. Gate, a Data Model Toolkit… 19

IOV Class Example class my. IOVClass { public: virtual Status. Code Call. Back. Fcn(list<string> &db. Keys); private: const Data. Handle<Cal. Align> m_align 1; const Data. Handle<Cal. Align> m_align 2; }; my. IOVAlg 2* that; my. IOVClass: : initialize() { p_IOVSvc->reg. Handle(m_align 1, d. Bkey 1); p_IOVSvc->reg. Fcn(&my. IOVClass: : Call. Back. Fcn, this, m_align 2, d. Bkey 2); } p_IOVSvc->reg. Fcn(&my. IOVClass: : Call. Back. Fcn, this, &my. IOVClass 2: : Other. Call. Back. Fcn, that); CHEP 2003, March 24 // session category 7 Store. Gate, a Data Model Toolkit… 20