The EU Data Grid Data Management The European

The EU Data. Grid Data Management The European Data. Grid Project Team http: //www. eu-datagrid. org Data. Grid is a project funded by the European Union Grid Tutorial 9/25/2020 – n° 1

Overview u Data Management Issues u Main Components n Replica Manager n Replica Location Service n Replica Metadata Catalog n Replica Optimization Service Grid Tutorial - 9/25/2020 – Data Management Services - n° 2

File Management Motivation Site A Site B Storage Element A Storage Element B File Transfer File A File X File B File Y File A File C File B File D Grid Tutorial - 9/25/2020 – Data Management Services - n° 3

File Management Motivation Replica Catalog: Map Logical to Site files Site A Site B Storage Element A Storage Element B File Transfer File A File X File B File Y File A File C File B File D Grid Tutorial - 9/25/2020 – Data Management Services - n° 4

File Management Motivation Replica Catalog: Replica Selection: Map Logical to Site files Get ‘best’ file Site A Site B Storage Element A Storage Element B File Transfer File A File X File B File Y File A File C File B File D Grid Tutorial - 9/25/2020 – Data Management Services - n° 5

File Management Motivation Replica Catalog: Replica Selection: Map Logical to Site files Get ‘best’ file Pre- Post-processing: Prepare Site A files for transfer Validate files after transfer Site B Storage Element A Storage Element B File Transfer File A File X File B File Y File A File C File B File D Grid Tutorial - 9/25/2020 – Data Management Services - n° 6

File Management Motivation Replica Catalog: Replica Selection: Map Logical to Site files Get ‘best’ file Pre- Post-processing: Replication Automation: Prepare Site A files for transfer Validate files after transfer Data Source subscription Site B Storage Element A Storage Element B File Transfer File A File X File B File Y File A File C File B File D Grid Tutorial - 9/25/2020 – Data Management Services - n° 7

File Management Motivation Replica Catalog: Replica Selection: Map Logical to Site files Get ‘best’ file Pre- Post-processing: Replication Automation: Prepare Site A files for transfer Validate files after transfer Data Source subscription Site B Load balancing: Replicate based on usage Storage Element A Storage Element B File Transfer File A File X File B File Y File A File C File B File D Grid Tutorial - 9/25/2020 – Data Management Services - n° 8

Replica Manager: ‘atomic’ replication operation File Management single client interface orchestrator Replica Catalog: Replica Selection: Map Logical to Site files Get ‘best’ file Pre- Post-processing: Replication Automation: Prepare Site A files for transfer Validate files after transfer Data Source subscription Site B Load balancing: Replicate based on usage Storage Element A Storage Element B File Transfer File A File X File B File Y File A File C File B File D Grid Tutorial - 9/25/2020 – Data Management Services - n° 9

Replica Manager: ‘atomic’ replication operation File Management single client interface orchestrator Replica Catalog: Replica Selection: Map Logical to Site files Get ‘best’ file Pre- Post-processing: Replication Automation: Prepare Site A files for transfer Validate files after transfer Metadata: LFN metadata Storage Element A Transaction information Access patterns File A File X File B File Y Data Source subscription Site B Load balancing: Replicate based on usage Storage Element B File Transfer File A File C File B File D Grid Tutorial - 9/25/2020 – Data Management Services - n° 10

Replica Manager: ‘atomic’ replication operation File Management single client interface orchestrator Replica Catalog: Replica Selection: Map Logical to Site files Get ‘best’ file Pre- Post-processing: Replication Automation: Prepare Site A files for transfer Validate files after transfer Metadata: LFN metadata Storage Element A Transaction information Access patterns File A File X File B File Y Data Source subscription Site B Load balancing: Replicate based on usage Storage Element B File Transfer File A File C File B File D Grid Tutorial - 9/25/2020 – Data Management Services - n° 11

Data Management Tools u Tools for n Locating data n Copying data n Managing and replicating data n Meta Data management u On EDG Testbed you have RM RLS n Replica Location Service (RLS) n Replica Metadata Service (RMC) n Repica Optimisation Service (ROS) n Replica Manager (RM) RMC ROS Grid Tutorial - 9/25/2020 – Data Management Services - n° 12

Replication Services: Basic Functionality Each file has a unique Grid ID. Locations corresponding to the GUID are kept in the Replica Location Service. Users may assign aliases to the GUIDs. These are kept in the Replica Metadata Catalog. Files have replicas stored at many Grid sites on Storage Elements. Replica Metadata Catalog Replica Manager Storage Element Replica Location Service The Replica Manager provides atomicity for file operations, assuring consistency of SE and catalog contents. Grid Tutorial - 9/25/2020 – Data Management Services - n° 13

Higher Level Replication Services Hooks for user-defined pre- and postprocessing for replication operations are available. The Replica Manager may call on the Replica Optimization service to find the best replica among many based on network and SE monitoring. Replica Manager Replica Metadata Catalog Replica Location Service Replica Optimization Service Storage Element SE Monitor Network Monitor Grid Tutorial - 9/25/2020 – Data Management Services - n° 14

Interactions with other Grid components User Interface or Worker Node Virtual Organization Membership Service Resource Broker Information Service Replica Metadata Catalog Replica Manager Replica Location Service Replica Optimization Service Storage Element Applications and users interface to data through the Replica Manager either SE Network Monitor directly or through the Resource Broker. Management calls should never go directly to the SRM. Grid Tutorial - 9/25/2020 – Data Management Services - n° 15

Storage Resource Management (1) Data are stored on disk pool servers or Mass Storage Systems n n storage resource management needs to take into account n Transparent access to files (migration to/from disk pool) n File pinning n Space reservation n File status notification n Life time management SRM (Storage Resource Manager) takes care of all these details SRM is a Grid Service that takes care of local storage interaction and provides a Grid interaface to outside world n n In EDG we originally used the term Storage Elemement n now we use the term SRM to refer to the new service Grid Tutorial - 9/25/2020 – Data Management Services - n° 16

Storage Resource Management (2) n Original SRM design specification: LBL, JNL, FNAL, CERN n. Support for local policy n. Each n n. Disk storage resource can be managed independently Internal priorities are not sacrificed by data movement between Grid agents and tape resources are presented as a single element n. Temporary n locking/pinning Files can be read from disk caches rather than from tape n. Reservation n. File on demand advance reservation n Space can be reserved for registering a new file n Plan the storage system usage status and estimates for planning n Provides info on file status n Provide estimates on space availability/usage EDG provides one implementation as part of the current software release n Grid Tutorial - 9/25/2020 – Data Management Services - n° 17

Simplified Interaction Replica Manager - SRM Replica Catalog 1 2 6 Replica Manager client 3 6 1. 2. 3. 4. 5. 6. SRM 4 5 Storage The Client asks a catalog to provide the location of a file The catalog responds with the name of an SRM The client asks the SRM for the file The SRM asks the storage system to provide the file The storage system sends the file to the client through the SRM or directly Grid Tutorial - 9/25/2020 – Data Management Services - n° 18

Naming Conventions n Logical File Name (LFN) s n Site URL (SURL) (or Physical File Name (PFN)) s n An alias created by a user to refer to some item of data e. g. “lfn: cms/20030203/run 2/track 1” The location of an actual piece of data on a storage system e. g. “srm//: pcrd 24. cern. ch/flatfiles/cms/output 10_1” Globally Unique Identifier (GUID) s A non-human readable unique identifier for an item of data e. g. “guid: f 81 d 4 fae-7 dec-11 d 0 -a 765 -00 a 0 c 91 e 6 bf 6” Logical File Name 1 Logical File Name 2 Logical File Name n Physical File SURL 1 GUID Physical File SURL n Grid Tutorial - 9/25/2020 – Data Management Services - n° 19

Replica Metadata Catalog (RMC) vs. Replica Location Service (RLS) u RMC: n RM Stores LFN-GUID mappings RLS RMC u RLS: n ROS Stores GUID-SURL mappings Logical File Name 1 Logical File Name 2 Logical File Name n RMC Physical File SURL 1 GUID Physical File SURL n RLS Grid Tutorial - 9/25/2020 – Data Management Services - n° 20

Replica Location Service (RLS) u The Replica Location Service is a system that maintains and provides access to information about the physical location of copies of data files. u It is a distributed service that stores mappings between globally unique identifiers of datafiles and the physical identifiers of all existing replicas of these datafiles. u Design is a joint collaboration between Globus and EDG-WP 2 RM RLS RMC ROS Grid Tutorial - 9/25/2020 – Data Management Services - n° 21

Replica Location Service RLS u Local Catalogs hold the actual name mappings u Remote Indexes redirect inquiries to LRCs actually having the file u LRCs are configured to send index updates to any number of RLIs u Indexes are Bloom Filters Grid Tutorial - 9/25/2020 – Data Management Services - n° 22

RLS Components (1) u Local Replica Catalog (LRC) n Stores GUID to SURL (PFN) mappings for a single SRM n Stores attributes on SURL (PFN), s n Maintains local state independently of other RLS components s n complete local record of all replicas on a single SRM Many Local Replica Catalogs in a Grid s n e. g file size, creator can be configured, for instance one LRC per site or one LRC per SRM Fairly permanent fixture s LRC coupled to the SRM, SRM removal is infrequent s new LRCs added with addition of new SRMs to a site RLI RM RLS RMC LRC ROS Grid Tutorial - 9/25/2020 – Data Management Services - n° 23

RLS Components (2) u Replica Location Index (RLI) n Stores GUID to LRC mappings n Distributed index over Local Replica Catalogs in a Grid n Receives periodic soft state updates from LRCs n s information has an associated expiration time s LRCs configured to send updates to RLIs (push) Maintains collective state s n Can be installed anywhere s n inconsistencies due to the soft state update mechanism not inherently associated with anything Uses Bloom Filter Indexes RLI RM RLS RMC LRC ROS Grid Tutorial - 9/25/2020 – Data Management Services - n° 24

LRC Implementation u. LRC data stored in a Relational Database n Runs with either Oracle 9 i or My. SQL u. Catalogs implemented in Java and hosted in a J 2 EE application server n Tomcat 4 or Oracle 9 i. AS for application server Java and C++ APIs exposed to clients through Apache Axis (Java) and g. Soap (C++) n u. Catalog APIs exposed using WSDL u. Vendor neutral approach taken to allow different deployment options Grid Tutorial - 9/25/2020 – Data Management Services - n° 25

RLI Implementation u Updates implemented as a push from the LRCs n RLIs less permanent than LRCs u Bloom filter updates only implemented n O(106) (or more ) entries in an LRC n May contain false positives, but no false negatives s rate depends on the configutation of the bloom filter u Bloom filters stored to disk u Impractical to send full LRC lists to multiple RLIs n fine for tests n not scalable in a production environment u Implemented in Java as a web service as for the LRCs Grid Tutorial - 9/25/2020 – Data Management Services - n° 26

User Interfaces for Data Management u Users are mainly referred to use the interface of the Replica Manager client: n Management commands Catalog commands RLS n Optimization commands RMC n File Transfer commands ROS n RM u The services RLS, RMC and ROS provide additional user interfaces n Mainly for additional catalog operations (in case of RLS, RMC) n Additional server administration commands s s Should mainly be used by administrators Can also be used the check the availability of a service Grid Tutorial - 9/25/2020 – Data Management Services - n° 27

The Replica Manager Interface – Management Commands u copy. And. Register. File n u u args: source/lfn, dest, protocol, streams Replicate a file between grid-aware stores and register the replica in the Replica Catalog as an atomic operation. delete. File n u Copy a file into grid-aware storage and register the copy in the Replica Catalog as an atomic operation. replicate. File n args: source, dest, lfn, protocol, streams args: source/se. Host, all Delete a file from storage and unregister it. Example edg-rm --vo=tutor copy. And. Register. File file: /home/bob/analysis/data 5. dat -d lxshare 0384. cern. ch Grid Tutorial - 9/25/2020 – Data Management Services - n° 28

The Replica Manager Interface – Catalog Commands (1) u register. File n u args: lfn/surl/guid List all replicas of a file. register. GUID args: surl, guid n u Unregister a file from the Replica Catalog. list. Replicas n u Register a file in the Replica Catalog that is already stored on a Storage Element. unregister. File args: source, guid n u args: source, lfn Register an SURL with a known GUID in the Replica Catalog. list. GUID n args: lfn/surl Print the GUID associated with an LFN or SURL. Grid Tutorial - 9/25/2020 – Data Management Services - n° 29

The Replica Manager Interface – Catalog Commands (2) u add. Alias n u u args: guid, lfn Remove an alias LFN from a known GUID. print. Info() n u Add a new alias to GUID mapping remove. Alias n args: guid, lfn Print the information needed by the Replica Manager to screen or to a file. get. Version() n Get the versions of the replica manager client. Grid Tutorial - 9/25/2020 – Data Management Services - n° 30

The Replica Manager Interface – Optimization Commands u list. Best. File n u u Return the 'best' replica for a given logical file identifier. get. Best. File n args: lfn/guid, se. Host, protocol, streams Return the storage file name (SFN) of the best file in terms of network latencies. get. Access. Cost n args: lfn/guid, se. Host args: lfn/guid[], ce[], protocol[] Calculates the expected cost of accessing all the files specified by logical. Name from each Computing Element host specified by ce. Hosts. Grid Tutorial - 9/25/2020 – Data Management Services - n° 31

The Replica Manager Interface – File Transfer Commands u copy. File n u Copy a file to a non-grid destination. list. Directory n args: soure, dest args: dir List the directory contents on an SRM or a Grid. FTP server. Grid Tutorial - 9/25/2020 – Data Management Services - n° 32

Replica Management Use Case edg-rm copy. And. Register. File -l lfn: higgs CERN LYON edg-rm list. Replicas -l lfn: higgs edg-rm replicate. File -l lfn: higgs NIKHEF edg-rm list. Best. File -l lfn: higgs CERN edg-rm get. Access. Cost -l lfn: higgs CERN NIKHEF LYON edg-rm get. Best. File -l lfn: higgs CERN edg-rm delete. File -l lfn: higgs LYON edg-rm list. Best. File -l lfn: higgs CERN Grid Tutorial - 9/25/2020 – Data Management Services - n° 33

Metadata Management and Security Project Spitfire u 'Simple' Grid Persistency n Grid Metadata n Application Metadata n Unified Grid enabled front end to relational databases. u Metadata Replication and Consistency u Publish information on the metadata service Secure Grid Services u u Grid authentication, authorization and access control mechanisms enabled in Spitfire Modular design, reusable by other Grid Services Grid Tutorial - 9/25/2020 – Data Management Services - n° 34

Spitfire Architecture u Web/Grid Services Paradigm n SOAP interfaces n JDBC interface to RDBMS Plugability and extensibility AP u Atomic RDBMS is always consistent No local replication of data authentication and fine-grained authorization based on the user's certicate Connecting Layer SOAP SO u u P A SO AP u browser access and dedicated client access through a uniform, secure interface SO u SO A P Global Spitfire Layer SO AP Oracle. Layer DB 2 Layer PGLayer My. Layer Oracle DB 2 Post. Gres My. SQL Local Spitfire Layer Grid Tutorial - 9/25/2020 – Data Management Services - n° 35

Conclusions and Further Information u The second generation Data Management services have been designed and implemented based on the Web Service paradigm u Flexible, extensible service framework u Deployment choices : robust, highly available commercial products supported (eg. Oracle) as well as open-source (My. SQL, Tomcat) u First experiences with these services show that their performance meets the expectations u Further information / documentation: n www. cern. ch/edg-wp 2 Grid Tutorial - 9/25/2020 – Data Management Services - n° 36