Riccardo Zappi Luca Magnoni INFNCNAF Bologna Page Bologna

Riccardo Zappi / Luca Magnoni INFN-CNAF, Bologna Page Bologna, 21 Marzo 2006

Summary 1. What SRM is 2. What Sto. RM is 3. Sto. RM status 4. Sto. RM next release 5. Tests done on Sto. RM 6. Conclusion 2 INFN Storage Workshop Bologna, 21 Marzo 2006

What SRM is • SRMs are middleware components that manage shared storage resources on the grid and provide management functionalities like: – Uniform access to heterogeneous types of storage – File pinning – Disk space allocation and advanced disk space reservation – Protocol negotiation – Life time management of files – Management of security • SRMs implements a SRM interface specified by GSM-WG within GGF. (v. 1. 1, v. 2. 1. 1 and v. 3. 0) 3 INFN Storage Workshop Bologna, 21 Marzo 2006

Managent Access & Data Access to SE 4 INFN Storage Workshop Bologna, 21 Marzo 2006

How SRM-service should be used? • GFAL – a low-level IO interface to Grid Storage – Storage Element Abstraction ( SRM v 1) • lcg_util – Provides both direct C library calls and CLI tools – Is a thin wrapper on top of GFAL SRM v 1 • FTS – The FTS receive requests either through their Web Service interface directly from the clients or indirectly through the Data Scheduler. – The transfers managed by the FTS are all asynchronous, i. e. the client submits a transfer ’job’, which may contain a list of files (with proper source and destination qualifiers) to be transferred. – Current version use SRM v 1. 5 INFN Storage Workshop Bologna, 21 Marzo 2006

What Sto. RM is Page Bologna, 21 Marzo 2006

What Sto. RM is • Sto. RM is a Storage Resource Manager. • It is exposed as web service. – Web service description (wsdl) of Sto. RM is compliant with SRM specification version 2. 1. 1 (WS-I compliant). • Sto. RM uses Cluster File System with ACL support. • Current Sto. RM version is built on top of GPFS • Sto. RM allows direct access (native posix call) – Sto. RM works without server I/O too. – Aggregation is performed by underlying file system (better than Sto. RM could do). 7 INFN Storage Workshop Bologna, 21 Marzo 2006

Sto. RM is different … I metadati relativi ai file e allo storage (come il namespace, space management, ecc. . ) sono gestiti completamente a livello applicativo, replicando le funzionalità tipiche dei file system (performance? Scalabilita? ) Gli accessi allo storage avvengono con modalità Posix-like. L’interfaccia esposta NON è del File System, ma è un wrapping fatto attraverso qualche libreria. . (dcap, rfio, ecc. ) 8 I metadati relativi ai file e allo storage, (quando possibile), sono gestiti completamente dal file system sottostante. Si evita così di replicare le stesse funzionalità del file system a livello applicativo (performance! Scalabilita!) Gli accessi allo storage avvengono attraverso chiamate POSIX esposte dal file system sottostante. Nel caso di Parallel File System, si possono sfruttare in pieno le performance offerte!! INFN Storage Workshop Bologna, 21 Marzo 2006

Sto. RM architecture • Front end (FE) has responsibilities of : – – – – • Data Base : – Store SRM request and status – Store application data • Back end (BE) has responsibilities of : – – – 9 expose a web service interface manage connection with authorized clients store asynchronous request into data base. retrieve asynchronous request status. co-operate with backend directly for synchronous call. co-operate with external authorization service to enforce security policy on service. manage user authentication accomplish all synchronous (active) action. get asynchronous request from data base. accomplish all asynchronous action. bind with underlying file system. enforce authorization policy on files manage SRM file and space metadata. INFN Storage Workshop Bologna, 21 Marzo 2006

Sto. RM Security Aspects 1. User perform srm. Prepare. To. Get 2. Sto. RM verifies if the principal holds a valid proxy certificate and delegates the external policy decision point to validate the request. 3. Sto. RM then queries the Authorization Sources to verify if the user can perform the specified operation on the SURL 4. Sto. RM queries LCMAPS to obtain local user account corresponding to the grid identity of the requestor 5. Physical file name derives by SURL and user attributes (Virtual organization name space) 6. The file system wrapper enforces permissions by setting a new ACL on the physical file. 7. The user job can be executed into the worker node 8. The application can perform a standard POSIX call to access the file into/from the storage system. 10 INFN Storage Workshop Bologna, 21 Marzo 2006

Sto. RM “Command Line” clients • Current release provides Command Line Clients. • These Clients are intended for Test use only. • FTS, lcg-util and GFAL are not usable with SRM v. 2. 1 yet. • Interoperability Clients are going to be used with Sto. RM. 11 INFN Storage Workshop Bologna, 21 Marzo 2006

Sto. RM : Status 1) Current release 2) Next release Page Bologna, 21 Marzo 2006

Sto. RM status – Release • • • First release [v 1. 0. 0] – End of November 2005 : base version Christmas release [v 1. 0. 1] – Xmas 2005 : patched version Second release [v 1. 0. 2] – End of January 2006 (CHEP version) : directory functions. Current release provides these functionalities: - Data transfer : srm. Copy, srm. Pt. G, srm. Pt. P, srm. Status<XXX> - Space Management : srm. Reserve. Space, srm. Get. Space. Metadata - Directory : srm. Ls, srm. Rm, srm. Mk. Dir, srm. Rmdir. Documentation (installation and configuration) available. Web Site of project : User management to better support 13 INFN Storage Workshop Bologna, 21 Marzo 2006

Sto. RM status – Today • Third release [v 1. 1. 0] • pre INFN storage workshop • Build completed ( Friday 17 March. . ) • Important improvements implemented New clients – Interoperability (WSDL WS-I compliant, SRM v. 2. 1. 1_modified) – Inline help improved – Parameter syntax checking Front End improved – WSDL WS-I compliant – Larger space reservation (2^64 byte) – Better explanation in case of failure Back End improved – Space Reservation : multiple utilization – “thread-safe” LCMAPS usage – performance tuning 14 INFN Storage Workshop Bologna, 21 Marzo 2006

Sto. RM status – fine Aprile/Maggio • Fourth release [v 1. 2. 0] • Scheduled for May (check web site for news). • Important improvements designed: Client improvement – – More parameters available Client-API on space management for Advance Reservation service. – – – VOMS – GPBOX authorization service integration Checking on malformed parameters Information Service Integration – – – Configuration file based on XML format (hierarchic and more expressive) Authorization source integration (LFC and/or ad-hoc solution) New functionalities : service monitoring, … – New documentation (installation and configuration) in different media (wiki). Front End improvement Back End improvement Web site and documentation 15 INFN Storage Workshop Bologna, 21 Marzo 2006

Sto. RM : TEST Page Bologna, 21 Marzo 2006

Sto. RM TESTING – current version • Third release [v 1. 1. 0] • Site A : Tier 1 (22 TB) • Site B : Bari (2 TB) • Site C : ICTP-Trieste (30 GB) • Site D : CNAF-Cert-SE (50 GB) • Tests • Functionality tests : Sito C e D • Stress tests : Sito A • Transfer tests : Siti A B 17 INFN Storage Workshop Bologna, 21 Marzo 2006

Test results Data transfer T 1 to/from Bari via srm. Copy v. 2. 1. 1 50 parallel srm. Copy with • From SURL at CNAF • To SURL at BARI • 1 GB file size, everyone Bandwidth saturated. Only 100 Mb/s 18 INFN Storage Workshop Bologna, 21 Marzo 2006

Sto. RM : Releasing process Group dedicated to Page • Developing • Documentation • Testing and integration • Prototyping Bologna, 21 Marzo 2006

Releasing process 20 INFN Storage Workshop Bologna, 21 Marzo 2006

Conclusions • Sto. RM service development involves many aspects (Development, Integration, Testing, Prototyping, Dissemination). Important progress done on each service component. • Sto. RM will be in production version by June! • We need help on Testing and Prototyping. • Help on integration is welcome. 21 INFN Storage Workshop Bologna, 21 Marzo 2006

Sto. RM Develop Team • Alberto Forti (CNAF), from 01 April • Luca Magnoni (CNAF), 100% • Riccardo Zappi (CNAF), 100% • Ezio Corso (ICTP), 100% • Antonio Messina (ICTP), 50% • Riccardo Murri (ICTP), 50% Project managed by Antonia Ghiselli. 22 INFN Storage Workshop Bologna, 21 Marzo 2006

Questions and Suggestions ? ! 23 INFN Storage Workshop Bologna, 21 Marzo 2006