Don Quijote Data Management for the ATLAS Automatic

Don Quijote Data Management for the ATLAS Automatic Production System 10/05/2004 Miguel Branco – CERN ATC Don Quijote - Status & Plans miguel. branco@cern. ch

Overview q Don Quijote o New Focus q Functionalities o POOL q Architecture q Current Status o o Nordu. Grid US Grid 3(+) LCG-2 Integration with ATLAS prodsys q Future plans 10/05/2004 Don Quijote - Status & Plans 2

Don Quijote q Data Management for the ATLAS Automatic Production System q Allow transparent registration and movement of replicas between all grid “flavors” used by ATLAS o US Grid o Nordugrid o LCG o (support for legacy systems might be introduced soon) q Avoid creating yet another catalog o which grid middleware wouldn't recognize (e. g Resource Brokers) o use existing catalogs and data management tools o find common features between tools and catalogs o bridge them and provide a unified interface q Accessible as a service o lightweight clients 10/05/2004 Don Quijote - Status & Plans 3

Don Quijote – new focus q Provide a single tool to end-users to manage data files o Integrates all tools that users would have to know about into a single one. E. g. : Ø Ø Ø FCpublish, FCregister, … (POOL File Catalogs) edg-rm, edg-rmc, edg-lrc, … (EDG) globus-rls-cli, globus-url-copy, … (Globus) ldapsearch, … (querying information system) rfdir, rfcp, … (common use of Castor) q Acts as a POOL-aware Replica Manager q Eases security requirements for end-users o Temporarily! 10/05/2004 Don Quijote - Status & Plans 4

Functionalities q q q q search | full. Search | search. Hosts ( lpn ) add[Restricted] ( lpn, url [, guid, fsize, md 5 sum ] ) add. Temporary[Restricted] ( lpn, url, nrhours [, guid, fsize, md 5 sum ] ) keep. Until ( url, nrhours ) Replica Catalogs Manipulation make. Permanent ( url ) remove. Replica ( url ) remove ( lpn ) rename ( old lpn, new lpn ) q stage. Out( url ) q get. To. Destination ( src SE, lpn , dest ) q put. To. SE ( src turl, lpn, dest SE [, guid, md 5 sum] ) File Movement LPN = Logical Collection Name + Logical File Name (unique) 10/05/2004 Don Quijote - Status & Plans 5

Functionalities - POOL q Integrates file movement with POOL XML File Catalogs o o Uses DQ + POOL FC command line tools Python scripts q Use-cases: o Get local copy of file and generate or update corresponding Pool. File. Catalog. xml Ø o (to provide input data and input POOL XML catalog for a job) Copy and register a local copy of a file to a grid flavor given UUID in the local Pool. File. Catalog. xml Ø (to register output data from a job) 10/05/2004 Don Quijote - Status & Plans 6

Architecture q Python Client o C++ client library o Configuration file indicating endpoint of each server q Servers o Per grid-flavor o GSI and insecure o Configuration file User interface tool written in Python Servers and client library written in C++ 10/05/2004 Don Quijote - Status & Plans 7

Changes on Server-side q Why was server-side code rewritten? o Partly because of CMS experience Ø Ø Ø o Persistent connections were necessary Connection pooling mechanism Each request could not instantiate a connection to the grid catalog – too slow! Partly from our initial experience Ø Flexible security mechanism • Either provide a single certificate for all, or delegate credentials q Initial version: o A command line tool for each grid flavor with the same syntax and same “output” o Clarens server was forking out a process that executed the request by calling the command line tool o This proved to be inefficient and too restrictive – e. g. could not maintain persistent connections across multiple requests! q Therefore, o Server code was built by extending the command line tools – each tool is now a daemon 10/05/2004 Don Quijote - Status & Plans 8

Current Status q Current structure: Dq. Core Dq. Pool. Rls C++ Client Module Dq. Globus. Rls Dq. Lcg. Replica. Access Dq. Classic. Replica. Access Dq. Lcg. Info. Service Dq. Vdt. Info. Service dms. py Python Module C++ Python wrapper (user interface) Dq. Ng. Info. Service Dq. Lcg. Pool. File. Catalog Dq. Fake. Pool. File. Catalog Dq. Factory Dq. Config. File Dq. Interface Dq. Monitor Dq. UI Dq. Server. Lcg, Dq. Server. Ng, Dq. Server. Vdt 10/05/2004 Don Quijote - Status & Plans 9

Nordu. Grid q Globus RLS 2. x q Only Classic Storage Elements (Grid. FTP servers) q Information System Dq. Core o o Connects to LDAP Special attributes in the RLS Dq. Globus. Rls Dq. Classic. Replica. Access Dq. Ng. Info. Service Dq. Fake. Pool. File. Catalog Dq. Factory Dq. Config. File Dq. Interface Dq. Monitor Dq. UI Dq. Server. Ng 10/05/2004 Don Quijote - Status & Plans 10

LCG-2 q EDG/LCG RLS (v 2. 2) q GFAL support: o o o SRM/Castor support SRM/d. Cache support Classic Storage Element support Dq. Core Dq. Pool. Rls Dq. Lcg. Replica. Access q Information System: o LDAP-based (MDS) q Native POOL Support o Dq. Lcg. Info. Service Using POOL-1. 6. 5 Dq. Config. File Dq. Lcg. Pool. File. Catalog Dq. Factory Dq. Interface Dq. Monitor Dq. UI Dq. Server. Lcg 10/05/2004 Don Quijote - Status & Plans 11

US Grid 3(+) q Globus RLS 2. x q DQ supports at the moment only Classic Storage Elements (Grid. FTP servers) q No “information system” interface o DQ creates a “dummy” information system which consists of a local configuration file Dq. Config. File Dq. Core Dq. Globus. Rls Dq. Classic. Replica. Access Dq. Vdt. Info. Service Dq. Fake. Pool. File. Catalog Dq. Factory Dq. Interface Dq. Monitor Dq. UI Dq. Server. Vdt 10/05/2004 Don Quijote - Status & Plans 12

Integration with ATLAS prodsys q Executors are using their “native” grid tools to do file registration o o But are adding extra-metadata attributes required by DQ This allows integration with DQ q Windmill is using DQ o o o To locate replicas of files Renaming of logical files to their final names (after validation) This week: move files across grids so that each executor finds at least a replica of all files required by the jobs 10/05/2004 Don Quijote - Status & Plans 13

Future plans q Better integration with POOL o Must come from end-users experience q Better end-user documentation and support o For now, focus has been only on the Automatic Production System q Get “best” replica (not high priority) o o within a grid between grids q Monitoring Still being discussed… q Reliable transfer service o Using My. SQL database to manage transfers and automatic retries o 10/05/2004 Don Quijote - Status & Plans 14

Future plans q Release command line tools appropriate for end-users o Request has been made to provide such tools for the Combined Test Beam effort q Provide servers as Pacman-caches q Much to improve o o Reliability Easy installation of client tool for users outside “grid” Ø Ø Get local copies of files to non-grid machine ? wrap in Pacman the minimal Globus Grid. FTP libraries q As true interoperability comes, Don Quijote goes… o Common information schema & similar catalogs o Common interface to storage resource “managers” 10/05/2004 Don Quijote - Status & Plans 15