Enabling Grids for Escienc E Storage Element Model

Enabling Grids for E-scienc. E Storage Element Model and Proposal for Glue 1. 3 Flavia Donno, Maarten Litmaath CERN, 6 November 2006 www. eu-egee. org INFSO-RI-508833

Outline Enabling Grids for E-scienc. E • Summary of WLCG view of Storage Element Model • Proposed Glue Schema for Glue v 1. 3 • Proposal • Status of higher level middleware (GFAL, lcg-utils, FTS) • Discussion INFSO-RI-508833 SRM v 2. 2 Meeting, CERN - 6 November 2006 2

Summary of WLCG view of Storage Element Model Enabling Grids for E-scienc. E • A Storage Element (SE) is a Grid service that allows Grid users to store and manage files together with the space assigned to them. • The SE Implementation is the software system used to manage the storage devices and servers. Examples of this are: CASTOR, d. Cache, DPM, Sto. RM, etc. • An SE exposes Total Sizes: an Online for space on disks and Sizes size for space on tape or slow devices. • An SE can have multiple Storage Areas. INFSO-RI-508833 SRM v 2. 2 Meeting, CERN - 6 November 2006 3

Summary of WLCG view : the Storage Component Enabling Grids for E-scienc. E • A Storage Component (SC) identifies a specific storage with certain properties: § Retention Policies: CUSTODIAL or REPLICA or OUTPUT § Access Latency: NEARLINE or ONLINE (or OFFLINE) § Access Protocols (examples: rfio, dcap, file, etc. ) • A multivalued retention policy and a single access latency are associated with an SC. – The concept of a Storage Component was introduced to describe the type of storage that is used to offer a certain quality of storage. For instance a Storage Component is a tape set or a pool of filesystems. • Two Storage Components cannot overlap. • An SC exposes Sizes and possibly quotas. – a Total Space: is the nominal capacity of the Storage Component subsystem (tape, dvd, disk, etc. ) – a Reserved Space: is the size of space reserved but not yet used – a Used Space: is the size occupied by files that are not candidates for garbage collection INFSO-RI-508833 SRM v 2. 2 Meeting, CERN - 6 November 2006 4

Summary of WLCG view : the Storage Component Enabling Grids for E-scienc. E INFSO-RI-508833 SRM v 2. 2 Meeting, CERN - 6 November 2006 5

Summary of WLCG view : the Storage Area Enabling Grids for E-scienc. E • A Storage Area (SA) is a view on a portion of the total space: – It is created by the System or VO Administrators – It can span different kinds of storage devices within a Storage Element – It exposes a single retention policy and a single access latency (which the underlying storage devices together can support) – In case of WLCG it implements a Storage Class instance. • An SA may be dedicated or shared between certain Vos/groups/roles – For WLCG the default SA is typically shared • For WLCG the SA implements a Storage Class instance: – It is identified by a Space Description (that can differ per VO). INFSO-RI-508833 SRM v 2. 2 Meeting, CERN - 6 November 2006 6

Summary of WLCG view : the Storage Area Enabling Grids for E-scienc. E INFSO-RI-508833 SRM v 2. 2 Meeting, CERN - 6 November 2006 7

SE Model from WLCG perspective Enabling Grids for E-scienc. E INFSO-RI-508833 SRM v 2. 2 Meeting, CERN - 6 November 2006 8

SE Model from WLCG perspective Enabling Grids for E-scienc. E INFSO-RI-508833 SRM v 2. 2 Meeting, CERN - 6 November 2006 9

SE Model from WLCG perspective Enabling Grids for E-scienc. E INFSO-RI-508833 SRM v 2. 2 Meeting, CERN - 6 November 2006 10

Proposed WLCG Glue v. 1. 3 Schema Enabling Grids for E-scienc. E J. Jensen v. 0. 8 INFSO-RI-508833 SRM v 2. 2 Meeting, CERN - 6 November 2006 11

Enabling Grids for E-scienc. E INFSO-RI-508833 SE Glue schema v 1. 2 SRM v 2. 2 Meeting, CERN - 6 November 2006 12

Enabling Grids for E-scienc. E INFSO-RI-508833 SE Glue schema v 1. 2 SRM v 2. 2 Meeting, CERN - 6 November 2006 13

Enabling Grids for E-scienc. E INFSO-RI-508833 SE Glue schema v 1. 2 SRM v 2. 2 Meeting, CERN - 6 November 2006 14

Report from Glue Working Group Enabling Grids for E-scienc. E • • VOSAAssociation rejected because of the many-to-many association. SA->SC association rejected because of the many-to-many association. SC and Access. Type too complicated for v 1. 3. Implementation and Names of Protocols should stay enum. Old sizes should stay. For the proposed ones, users should use Get. Space. Metadata SRM call. OK for proposed Glue. SE class changes. Extending SA to include VOSAAssociation: Access. Control. Base: cms. FQAN Access. Control. Base: atlas. FQAN Space. Token. Description: atlas? atlas. Disk 1 Tape 1 Space. Token. Description: cms? cms. Disk 1 Tape 1 Path: atlas? /castor/ads. rl. ac. uk/grid/data/atlas/ Why ? Instead of : Path: cms? /castor/ads. rl. ac. uk/grid/data/cms/ Expiration. Mode not needed for WLCG Quality: custodial-online-never. Expire Can we assume Permanent? Quality: custodial-nearline-never. Expire Volatile is default Quality: replica-online-release. When. Expired ? ? ? VO specific ? Quality: replica-online-warn. When. Expired INFSO-RI-508833 SRM v 2. 2 Meeting, CERN - 6 November 2006 15

Enabling Grids for E-scienc. E INFSO-RI-508833 New proposal SRM v 2. 2 Meeting, CERN - 6 November 2006 16

Enabling Grids for E-scienc. E Proposal • 11 th November 2006 proposed date for reaching agreement. Proposal: • Use v 1. 2 with addition of the proposed attributes for Glue. SE. • Glue. Foreign. Key as Glue. Service to Glue. SE association (check examples to see why it is needed). • Glue. Service. Data Key-Value pairs: to be used for VOSAAssociation in Glue. SA (Glue. SAData? ) INFSO-RI-508833 SRM v 2. 2 Meeting, CERN - 6 November 2006 17

Higher level middleware: GFAL, lcg-utils, FTS, SAM Enabling Grids for E-scienc. E • Under test at the moment an implementation providing SRM v 2. 2 support • Plain Glue v 1. 2 SE schema used. Some wanted functionalities are missing • Use cases: – Find SE type of host (classic or SRM ? ) § Glue. SEName: CERN-PROD-LHCB: disk (classic) § Glue. SEName: CERN-PROD-LHCB: srm_v 1 § Glue. SEName: CERN-PROD-LHCB: srm_v 2 ? ? – Find SE endpoint for host § Glue. Service. URI: httpg: //castorgridsc. cern. ch: 8443/srm/managerv 1 § Glue. Service. Endpoint: httpg: //castorgridsc. cern. ch: 8443/srm/managerv 1 – Find SE port for host (for classic SE) § Glue. SEPort: 2811 – Find SA root for VO on host (for direct access from a WN) § Glue. SARoot: lhcb – Find SA path for VO on host (needed to find Grid. FTP path for VO) § Glue. SAPath: /storage/atlas INFSO-RI-508833 SRM v 2. 2 Meeting, CERN - 6 November 2006 18

Higher level middleware: GFAL, lcg-utils, FTS, SAM Enabling Grids for E-scienc. E • Use cases: – Find SE access protocol for host § Glue. SEAccess. Protocol. Type: rfio § Glue. SEAccess. Protocol. Port: 5001 – Find CE access point for host (for “file” protocol and classic SE) § Glue. CESEBind. CEAccesspoint: /castor/cern. ch/grid/ • Use cases not addressed in v 1. 2 – Find a CE with a close SE that supports ATLAS_RAW – Find a CE that has a close SE with an SA accessible by a given VO FQAN that supports “replica” as a retention policy. – Find an SE close to a specific CE that supports ATLAS_ESD and the protocol gsidcap on WAN – Find the total size of all SAs with User Description ATLAS_RAW – Find an SE accessible by a specific VO/FQAN – Find out status of an SE service – Find out status of an SA INFSO-RI-508833 SRM v 2. 2 Meeting, CERN - 6 November 2006 19