Database Deployment and Operations US ATLAS DDM Operations

Database Deployment and Operations US ATLAS DDM Operations and MC Production Workshop BNL, Brookhaven, NY September 28 -29, 2006 Alexandre Vaniachine (Argonne)

Outline n WLCG 3 D Project and ATLAS Database Deployment and Operations n Synergistic activities: – Project GSIMF – Project SHIELDS n Secure database deployment in production n Database release deployment n Conclusions Alexandre Vaniachine US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept. 28 -29, 2006 2

WLCG Distributed Deployment of Databases (3 D) Project n To set-up database services and facilities for relational data transfers as part of the WLCG infrastructure the WLCG Distributed Deployment of Databases (3 D) project is coordinating database deployment and operations activity between LHC experiments and WLCG tier sites [http: //lcg 3 d. cern. ch] n The 3 D project is deploying distributed database services and developing advanced technologies allowing LCG applications to access the required data n 3 D full production service: – Oct ‘ 06 - onwards Alexandre Vaniachine US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept. 28 -29, 2006 3

Database Deployment & Operations: Activity n As ATLAS Computing moves from a development-centric organization to an operations-centric one a new “Database Deployment & Operations” activity is now defined: n The activity consists in the development and deployment (in collaboration with the WLCG 3 D project) of the tools that allow the worldwide distribution and installation of databases and related datasets, as well as the actual operation of this system on ATLAS multi-grid infrastructure http: //twiki. cern. ch/twiki/bin/view/Atlas/Database. Operations Alexandre Vaniachine US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept. 28 -29, 2006 5

Database Deployment & Operations: Domains n Distributed Deployment - Grigori Rybkine (U. K. ) – LCG Deployment: Mireia Dosil (Spain) and Suijian Zhou (Taiwan) – OSG Deployment: Yuri Smirnov (U. S. ) – Nordu. Grid Deployment: Frederik Orellana (Switzerland) – Legacy Sites (batch): John Idarraga (Canada) n Distributed Operations (ATLAS subset of WLCG 3 D operations) – Tier 1 operations: Gancho Dimitrov, Florbela Viegas (CERN) – Tier 2 operations: Stefan Stonjek (U. K. ) – Distributed calibration centers: Manuela Cirilli (U. S. ) n Development - Jerome Fulachier (France) – Monitoring, SHIELDS, Fro. NTier/Squid, Dynamic Deployment, … n Documentation & User Support - vacant – Data access, File Transfer (for users), Best Practices, . . . n Further information is at http: //twiki. cern. ch/twiki/bin/view/Atlas/Database. Operations Alexandre Vaniachine US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept. 28 -29, 2006 6

ATLAS Software & Computing Workshop, Sept. 11 -15 n The first Database Deployment and Operations session was held during the ATLAS Software Workshop this week http: //indico. cern. ch/conference. Display. py? conf. Id=a 057208#2006 -09 -12 n The very first session was quite full – Which is a good sign for the newly defined activity Alexandre Vaniachine US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept. 28 -29, 2006 7

US ATLAS Efforts n Tier 1 activities – see next two presentations: – Iris Wu and Yuri Smirnov n ATLAS Tier 2 Database Center Services – Marco Mambelli / Rob Gardner – http: //twiki. mwt 2. org/bin/view/Data. Services/Web. Home n ATLAS Database Grid Authentication – Wensheng Deng – http: //www. hep. anl. gov/atlas/ACG/SHIELDS n ATLAS Database User Support – Jerry Gieraltowski – http: //twiki. cern. ch/twiki/bin/view/Atlas/Release. Validation n ATLAS Fro. NTier evaluation (now led by David Front - LCG support) – Richard Hansen (Argonne) – Carlos Gamboa (Brookhaven) – http: //twiki. cern. ch/twiki/bin/view/Atlas. Fro. NTier. Evaluation n Focus on COOL Performance Studies (CMS is not using COOL) Alexandre Vaniachine US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept. 28 -29, 2006 8

A Side Note: ATLAS Fro. NTier Issues with COOL n Based on general Fro. NTier performance from the CMS experience prospects are good to achieve the performance required for ATLAS – ATLAS does not store large amounts of data in COOL: • The bulky payload data are stored outside of COOL - in files n It is proposed that these ATLAS COOL POOL ROOT files will be managed by the ATLAS DDM – Currently they are included in ATLAS Database Release • Which is not a scalable solution n Also to be addressed in Fro. NTier evaluation n Squid cache consistency with COOL – Current issues with COOL tagging interface • May result in stale cache condition • Potentially a showstopper – if there is no COOL manpower to address this issue Alexandre Vaniachine US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept. 28 -29, 2006 9

Leveraging Synergistic US Activities n Database deployment support from Open Science Grid – Fro. NTier will become a part of OSG release – Grid-enabled My. SQL client is already a part of OSG release n Building technical collaboration with two small–scale projects – Tech-X Corp. project GSIMF: • Grid Software Installation Management Framework http: //collaborate. txcorp. com/collaborate/distributed-technologies/grid-software-installation-management-framework-gsimf • Argonne is providing specs to Tech-X so that their software product could be used by individual OSG users for ATLAS Database Release deployment – Argonne project SHIELDS: • securing databases on the Grid Alexandre Vaniachine US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept. 28 -29, 2006 10

Project SHIELDS Alexandre Vaniachine US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept. 28 -29, 2006 11

What SHIELDS is doing: End-to-End Globus Security Solution for Database Access n A drawback of a middleware approach is vulnerability of the actual database access at the back-end n To eliminate this vulnerability SHIELDS delivers end-to-end Globus security Client Globus-secure communications Client Middleware End-to-end transport-level Globus security Vulnerable back-end communications Database Alexandre Vaniachine US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept. 28 -29, 2006 13

Building Upon the Foundation of DASH n A collaborative DOE HEP SBIR project was completed by April 2006 – A partnership of Argonne HEP Division and PIOCON Technologies: • A database consulting business with a track record of Database and Grid integration projects n Project DASH studied the feasibility of the grid-enabled My. SQL database development n Building upon the foundation of previous studies project SHIELDS speeded-up the development cycle and provided the first release – http: //www. hep. anl. gov/atlas/ACG/SHIELDS/mysql-shields-1. 0 -alpha-pc-linux-gnu-i 686. tar. gz Alexandre Vaniachine US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept. 28 -29, 2006 16

Elaborate Globus Security Infrastructure Embedded SHIELDS Release 1. 0 -alpha ldd bin/mysqld linux-gate. so. 1 => (0 x 00626000) librt. so. 1 => /librt. so. 1 (0 x 001 ff 000) libz. so. 1 => /usr/local/osg/Mona. Lisa/Service/VDTFarm/pgsql/libz. so. 1 (0 x 00 e 47000) libpthread. so. 0 => /libpthread. so. 0 (0 x 00 fd 3000) libglobus_gsi_proxy_core_gcc 32 dbgpthr. so. 0 => /usr/local/osg/globus/libglobus_gsi_proxy_core_gcc 32 dbgpthr. so. 0 (0 x 00 ecb 000) libglobus_gsi_credential_gcc 32 dbgpthr. so. 0 => /usr/local/osg/globus/libglobus_gsi_credential_gcc 32 dbgpthr. so. 0 (0 x 00111000) libglobus_gsi_callback_gcc 32 dbgpthr. so. 0 => /usr/local/osg/globus/libglobus_gsi_callback_gcc 32 dbgpthr. so. 0 (0 x 00 c 37000) libglobus_oldgaa_gcc 32 dbgpthr. so. 0 => /usr/local/osg/globus/libglobus_oldgaa_gcc 32 dbgpthr. so. 0 (0 x 00120000) libglobus_gsi_sysconfig_gcc 32 dbgpthr. so. 0 => /usr/local/osg/globus/libglobus_gsi_sysconfig_gcc 32 dbgpthr. so. 0 (0 x 00 d 3 f 000) libglobus_gsi_cert_utils_gcc 32 dbgpthr. so. 0 => /usr/local/osg/globus/libglobus_gsi_cert_utils_gcc 32 dbgpthr. so. 0 (0 x 00 a 9 f 000) libglobus_openssl_gcc 32 dbgpthr. so. 0 => /usr/local/osg/globus/libglobus_openssl_gcc 32 dbgpthr. so. 0 (0 x 00823000) libglobus_openssl_error_gcc 32 dbgpthr. so. 0 => /usr/local/osg/globus/libglobus_openssl_error_gcc 32 dbgpthr. so. 0 (0 x 009 cb 000) libglobus_proxy_ssl_gcc 32 dbgpthr. so. 0 => /usr/local/osg/globus/libglobus_proxy_ssl_gcc 32 dbgpthr. so. 0 (0 x 0012 a 000) libglobus_common_gcc 32 dbgpthr. so. 0 => /usr/local/osg/globus/libglobus_common_gcc 32 dbgpthr. so. 0 (0 x 0012 f 000) libssl_gcc 32 dbgpthr. so. 0 => /usr/local/osg/globus/libssl_gcc 32 dbgpthr. so. 0 (0 x 0016 a 000) libcrypto_gcc 32 dbgpthr. so. 0 => /usr/local/osg/globus/libcrypto_gcc 32 dbgpthr. so. 0 (0 x 00212000) libltdl_gcc 32 dbgpthr. so. 3 => /usr/local/osg/globus/libltdl_gcc 32 dbgpthr. so. 3 (0 x 00 c 47000) libdl. so. 2 => /libdl. so. 2 (0 x 008 f 7000) libcrypt. so. 1 => /libcrypt. so. 1 (0 x 006 a 3000) libnsl. so. 1 => /libnsl. so. 1 (0 x 00 c 82000) libstdc++. so. 6 => /usr/libstdc++. so. 6 (0 x 00336000) libm. so. 6 => /libm. so. 6 (0 x 0041 f 000) libgcc_s. so. 1 => /libgcc_s. so. 1 (0 x 001 a 0000) libc. so. 6 => /libc. so. 6 (0 x 00445000) /lib/ld-linux. so. 2 (0 x 00627000) Alexandre Vaniachine US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept. 28 -29, 2006 17

Secure Database Deployment in Production n The deployment of SHIELDS software product and its integration into the existing distributed data management system in production requires additional efforts, e. g. , the – connection protocol string proposed by Wensheng Deng: mysqlx 509: //hbf 7 e 4 e 5 a: x 509 up_u 2914@adbpro. usatlas. bnl. gov: 3306/test_lrc n A dedicated meeting on that and other technical issues of grid-enabled databases took place at Brookhaven during this workshop n In particular we discussed plans for deployment in production at – US ATLAS Tier 1 Computing Center at Brookhaven Lab – As well as deployment of latest versions for testing at • US ATLAS Tier 2 Computing Center at the University of Chicago n Can we accomplish the deployment soon? Alexandre Vaniachine US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept. 28 -29, 2006 20

Database Release Deployment Technologies n Common LHC software used by ATLAS for data is technology neutral – that is why various technologies can be used for actual data storage: – POOL ROOT files – Relational Databases • Server-based: – Oracle – My. SQL • File-based: SQLite n Database-resident data replication technologies: – For server-resident data • Oracle streams • Fro. NTier – For file-based data • ATLAS Distributed Data Management technology (dataset subscription) • ATLAS Software Installation technology Alexandre Vaniachine US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept. 28 -29, 2006 22

Validation and Decoupling VALIDATION n Software installation is not a simple data transfer (also compilation) – Thus, the Software Installation technology includes integrated release validation procedures n In contrast, the validation of the data transfer is much simpler – e. g. the checksum n Database Release deployment does not need complex validation after the data transfer DECOUPLING n The event data and the software release are decoupled – Any data can be processed by any software release – A similar decoupling for the Database Release data is required n A detailed discussion thread in the Computing Operations mailing list showed that ATLAS software installation technology will not support decoupling of the Database Release from the Software Release – Note, that on the grid the software and the event data are installed at each site in different SE areas Alexandre Vaniachine US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept. 28 -29, 2006 26

Technical Choices n For now, the two candidate technologies are the – ATLAS Software Release Installation technology • Push Model – ATLAS DQ 2 dataset subscription technology • Pull Model n During the Recent ATLAS Software Workshop we tried to reach conclusion on the database release deployment strategy – I recommended to implement proven DQ 2 dataset subscription technology Alexandre Vaniachine US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept. 28 -29, 2006 27

Database Replicas Deployment: Implementation n Two proven ATLAS technologies – SW Release installation and the DDM DQ 2 dataset subscription - are available for implementation: – Integration with the Software Release installation technology • Possible if DB Release and SW Release are tightly coupled – i. e. each DB Release results in new SW Release installation – not sustainable in the long term • Attempts to decouple DB Release and SW Release installations revealed technical ‘showstoppers’ in the current production system – Integration with the DDM DQ 2 dataset subscription mechanism • Implemented now thanks to Martin, Vakho, Pavel and Grigori n To collect experience before making long term deployment decisions the deployment technology will be tested in Release 12. 0. 3 production Alexandre Vaniachine US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept. 28 -29, 2006 28

Conclusions n New ATLAS Database Deployment and Operation Activity has been defined – US ATLAS contributes efforts to the Activity n To secure production operations the grid-enabled database has to be deployed n Before making long term deployment decisions – Fro. NTier cache consistency must be validated for COOL – The new database deployment technology will be tested in Release 12. 0. 3 production Alexandre Vaniachine US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept. 28 -29, 2006 29