LHC Computing Comprehensive Review Frdric Hemmer CERNIT Middleware

LHC Computing Comprehensive Review • Frédéric Hemmer – CERN/IT Middleware Status • November 14, 2005

Outline • Features, Status and short term plans • WMS • FTS • R-GMA • VOMS • Storage Elements • Catalogs • Others • Middleware convergence • Summary LCG Comprehensive Review – Middleware Status 2

Workload Management (new) • Features • Status • • • Bulk job submission (Web Services) Jobs with Shared Sandboxes Interfaces to DLI & Storage Index Condor-C GSI enabled Support of R-GMA as resource information (as well as BDII & CEMon) CEMon also in VDT DGAS Accounting system Deployed on Pre-Production Service and several sites in Italy Compatibility with LCG 2. x demonstrated • • • RB can still use Condor-G for job submission Interoperability tested in the “mixed” Certification testbed Short Term evolution • Support for “CE reputability ranking“ • • Maintains recent statistics of job failures at CE’s Feeds back to WMS to aid planning • To be used e. g. to put a job on hold, waiting for e. g. the staging of the input data • Support for hold and resume in BLAH • GPbox • • XACML-based policy maintainer, parser and enforcer. Can be used for authorisation checks at various levels • Pilot job which "prepare" the execution environment and then get and execute the actual user job • • • Long term job information storage Useful for debugging, post-mortem analysis, comparison of job executions in different environments Useful for statistical analysis • • Support for execution of all DAG nodes within a single CE - chosen by user or by the WMS matchmaker Support for file peeking to access files during the execution of the job Initial integration with G-Pbox - considering simple Auth. Z policies Initial support for pilot job • Job provenance LCG Comprehensive Review – Middleware Status 3

File Transfer Service • Features • • • Status • • • Provides Point-to-Point Reliable file transfer Services Used against Castor, Castor 2, d. Cache and DPM SRM’s FTS evolved over summer to include FTS clients distributed since LCG 2. 5. 0 (Jun’ 05) Deployed on SC 3 sites and some Pre-Production Service sites Short term plans • SRM v 2 support • SRM lookup from BDII • • cache to services. xml Rather than using services. xml Staging support for FTS • Delegation / My. Proxy • • • Basic SRM v 2 usage SRM. get / SRM. put / SRM. copy. No extra features. • • • Support for My. SQL and Oracle Multi-VO support Grid. FTP and SRM copy support My. Proxy server as a CLI argument Many small changes/optimizations revealed by SC 3 usage MSS specific staging plugins - make sure staging can be done without occupying a transfer slot Current mechanism, using available LCG-2 production services. Many “small” changes required by SC 3 experience • Including better statistics, reporting and management tools Detailed work plan maintained at https: //uimon. cern. ch/twiki/bin/view/EGEE/DMFts. Work. Plan FTS Workshop on November 16, 2005: http: //agenda. cern. ch/full. Agenda. php? ida=a 056842 LCG Comprehensive Review – Middleware Status 4

R-GMA • Essentially bug fixes & consolidation • Merging LCG & g. Lite code base • Secure version • Used in production as monitoring data aggregator • Service Discovery • An interface has been defined and implemented for 3 back-ends • R-GMA • BDII • Configuration File • Command Line tool for easy query and conversion between back- ends • Used WMS and Data Management clients LCG Comprehensive Review – Middleware Status 5

VOMS • Features • Manages VO Membership • Provides support for Groups and Roles • Status • Available from LCG 2. x • Deployed on the production and pre-production • Included in VDT • VOMS and VOMS Admin • Support for Oracle and My. SQL backends • VOMS ADMIN (Oracle) still problematic • Support issues clarified • Support for many other clients than SLC 3 LCG Comprehensive Review – Middleware Status 6

Storage Elements • Castor, Castor 2 • CERN, CNAF, PIC • See separate talk • d. Cache • Distributed with LCG 2. x • Many sites • See separate talk • DPM • Features • Management of disk space • Management of name space (including ACLs) • Control interfaces: socket, SRM v 1. 0 (no srm. Copy method yet), SRM v 2. 1 without global space reservation • Data access protocols: secure RFIO, Gsi. FTP (Globus 2. 4) • Status • Deployed at 16 sites • UK, Russia, France, Italy, Taiwan, Romania, Canada • Short Term plans • VOMS-enabled DPM • Support for srm. Copy • DPM Grid. FTP 2 (from Globus 4) LCG Comprehensive Review – Middleware Status 7

Catalogs • LFC • Features • • • Status • Local LFC (12 sites) • Global LFC (11 sites) – Spain, CERN, France, Italy, Germany, Russia, Netherlands, UK – CERN, Portugal, France, Hungary, Germany, China, Netherlands, Belgium, Greece Short Term Plans • • Hierarchical namespace Oracle and My. SQL versions available Secure Services with ACL’s Interfaced with lcg_util Python/C/GFAL Interfaces Exposing DLI (for matchmaking) Read-only replicas VOMS support ready but not deployed yet Faster “list” operations in LFC (CMS and LHCb) Sessions and transactions Fi. Re. Man catalog • Features • • Hierarchical namespace Oracle and My. SQL versions available Secure services, using VOMS groups, ACL support for DNs Full set of Command Line tools Symbolic link support Bulk operations Exposing Service. Index and DLI (for matchmaking) Status • • – Separate catalog available as a keystore for data encryption (‘Hydra’) Deployed on Pre-production Service Used by DILIGENT and Biomedical applications LCG Comprehensive Review – Middleware Status 8

Other Data Management Services • g. Lite I/O • Posix-like I/O to grid files • Castor, d. Cache and DPM supported • Added a remove method to be able to delete files • Configuration through Service-discovery • Improved error reporting • Used for the Bio. Medical Demo at EGEE 4 th conference • Encryption and DICOM SRM • AMGA generic Meta. Data Catalog • ARDA contribution • Result of Middleware & ARDA prototyping together with EGEE PTF assessment • Used for the Bio. Medical Demo at EGEE 4 th conference LCG Comprehensive Review – Middleware Status 9

Credits and g. Lite convergence • Current Middleware comes from many sources • Including CERN, Condor, Globus, INFN, RAL and many others … • Integrated through distributions • g. Lite, LCG and VDT • EGEE, VDT providing support through their partners • Forum for collaboration • EGEE Middleware Design Team – Gathers people from EGEE, Globus and VDT – Gateway to OSG thanks to its US representation • The LCG and g. Lite middleware will converge to a single distribution called g. Lite • Being discussed with Deployment people • EGEE-II startup timeframe LCG Comprehensive Review – Middleware Status 10

Summary • Basic middleware components are being deployed • Tested and being certified on the Pre-Production Service • Evaluated and tested by ARDA, Task Forces and other projects • Some subsystems are used on the Production infrastructure • FTS, VOMS and R-GMA • Many tests are now concentrating on performance • Developers reacting relatively quickly to users needs • Used by High Energy Physics and other projects • A plan for converging LCG & g. Lite distributions is being discussed LCG Comprehensive Review – Middleware Status 11