SAM Replica Catalog Roadmap of Talk EDG May

SAM Replica Catalog Roadmap of Talk • • • EDG May 12 -16, 2003 Lee Lueking Fermilab Computing Division CEPA Department 12 -16 May, 2003 • • SAM Data Management Overview EDG - SAM Cross Reference SAM Features and Use Case Examples EDG – SAM Command Reference Summary Lee Lueking, EDG Int. Proj. Conf. 1

An Overview of SAM Data Management d 0 db. fnal. gov/sam 12 -16 May, 2003 Lee Lueking, EDG Int. Proj. Conf. 2

Compute Resources (CPU + Memory) Managing Resources in SAM Fair-share Resource allocation Data and Compute User groups Co-allocation Project= Local Batch DS on Station Consumer(s) SAM Global Optimizer SAM Station Servers SAM Datasets metadata Definitions (DS) Cache Management Data Resources (Storage + Network) Batch scheduler 12 -16 May, 2003 SAM Meta-data SAM servers Lee Lueking, EDG Int. Proj. Conf. Batch + SAM 3

SAM Station Components The SAM Station Producers/ Project Managers Temp Disk File Storage Clients File Storage Server Data flow Control 12 -16 May, 2003 /Consumers Cache Disk Station & Cache Manager File Stager(s) Lee Lueking, EDG Int. Proj. Conf. MSS or Other Station eworkers 4

SAM as a Distributed System CORBA Name Server Database Server(s) (Central Database) Station 1 Servers Global Resource Manager(s) Station 3 Servers Station 2 Servers Arrows indicate Control and Data Flow 12 -16 May, 2003 Mass Storage System(s) Lee Lueking, EDG Int. Proj. Conf. Log server Station n Servers Shared Globally Local To Site Shared Locally 5

EDG and SAM Terminology Preliminary – to generate discussion 12 -16 May, 2003 Lee Lueking, EDG Int. Proj. Conf. 6

Naming Conventions EDG Acronym EDG Name SAM Name or comment SFN Storage File Name. UUID Universally Unique IDentifier Date and time info GUID Grid Unique IDentifier File names must be unique LFN Logical File Name Closest concept is dataset, or a collection of files referred to by logical name. TURL Transport URL Location is stored as 1. host, station, or MSS with full unix path, or 2. url for network attached files (RFIO, d. CAP) 12 -16 May, 2003 Lee Lueking, EDG Int. Proj. Conf. 7

Data Management EDG Acronym EDG Name SAM Name or Comment DMS Data Management Services SAM provides data management and adapters to storage systems. RMS Replica Management Services Provided through SAM Stations in conjunction with SAM DB and Global Optimizer RFT Reliable File Transfer SAM Stager. Uses retries and CRC to assure reliable transfer SRM Storage Resource Manager SAM Station Cache management. Part of SAM station servers. Discussing migrating to the protocol referred to as “SRM” from LBNL. 12 -16 May, 2003 Lee Lueking, EDG Int. Proj. Conf. 8

Replica Management EDG Acronym EDG Name SAM Name or Comment ERM EDG Replica Manager SAM CORBA IDL’s, SAM user interface, CLI and WEB RLS Replica Location Service Through SAM DB server LRC Local Replica Catalog File Locations table in Central SAM Database RLI Replica Location Index Central Database RMC Replica Metadata Catalog Data_files and other tables in SAM Database ROS Replica Optimization Service SAM Optimizer RSH Replica Storage Handler SAM Station 12 -16 May, 2003 Lee Lueking, EDG Int. Proj. Conf. 9

SAM Function and Use Cases 12 -16 May, 2003 Lee Lueking, EDG Int. Proj. Conf. 10

Storing and Accessing SAM Data and Meta-Data ü Sam store – Description of metadata, – Auto destination – Station data forwarding ü The SAM Schema – tracking file lineage – The concept of “dimensions” ü SAM data Access – Using file metadata to create logical sets of files – Accessing files through projects on SAM stations ü SAM Station file replication and cache management – Station configurations with and without SAM stagers on workers 12 -16 May, 2003 Lee Lueking, EDG Int. Proj. Conf. 11

Storing Data sam store –desc=Description. File. py ü Description files – Contain physics and file metadata. – Written as Python scripts – They are required to store data. – Latest version of description file uses name–value pairs for more flexibility in adding parameters for data and MC files ü Auto-destination – A map which relates information in the description file to physical storage location ü File forwarding – Data is forwarded from source station to designated physical storage location 12 -16 May, 2003 Example Description File from import_classes import * # # Generated by run. MCwin # my_d 0 gstar = App. Family( "simulator", "p 07. 00. 05 a", "d 0 gstar" ) class My. Process(Proc. Family): group="higgs" origin_location="FNAL" origin_facility="d 0 mino" produced_for="Qizhong Li" phase="group-phase 1" def __init__(self, stream, param_file, produced_by): self. stream=stream self. param_file=param_file self. produced_by=produced_by class Simulator(My. Process): appfamily=my_d 0 gstar channel = Channel("bbh", "bbbb") minbi = Min. Bias("none", "0. 0") d 0 g_fil=Simulator(stream="notstreamed", param_file="d 0 gstar_test 185201919. params", produced_by="Avto Kharchilava") d 0 g_file_import =Simulated. File("d 0 g. pythia_bbh_bbbb 1. dat", d 0 g_fil, 65123, Events(1, 500), "07/03/2001 17: 44", "07/04/2001 05: 23", "pythia_bbh_bbbb 1. dat", 1, 1, channel) Lee Lueking, EDG Int. Proj. Conf. 12

SAM Simplified Database Schema Run Conditions MC Request & Info Run Data Tier Physical Data Stream Luminosity Calibration Trigger DB Alignment Events ID Event Number Trigger L 1 Trigger L 2 Trigger L 3 Off-line Filter Thumbnail 12 -16 May, 2003 Event-File Catalog ID Name Format Size # Events Trigger Configuration Project File Storage Locations • SAM schema has over 100 tables • There are several other related tablespaces also available Files Volume Station Config. & Cache info Group and User information Lee Lueking, EDG Int. Proj. Conf. Creation & Processing Info 13

Tracking File Lineage ü Application name and version information (Pkg) ü Parent or parents information ü File splitting and merging. Phase Boundaries Pkg 1 Dataset 1 12 -16 May, 2003 DS 2 Pkg 1 Pkg 2 DS 3 Processing Line Pkg 3 Dataset 4 Pkg 3 Lee Lueking, EDG Int. Proj. Conf. 14

Challenge: Transform the complex SAM schema into a form that is user friendly, and avoids badly formed user SQL queries. Solution: Transform the schema to look like one giant table. Data. File Dimension Name file Run Event Date Trigger Apo App vsn … file 1 file 2 file 3 file 4 file 5 filen 12 -16 May, 2003 Lee Lueking, EDG Int. Proj. Conf. 15

Accessing Data: Defining Datasets ü There are dozens of dimensions available and they are easily defined. – APPL_NAME, APPL_NAME_ANALYZED, CONSUMED_DATE, CONSUMED_STATUS, CONSUMER_GROUP, CONSUMER_ID, CREATE_DATE, DATASET_DEF_ID, DATASET_DEF_NAME, DATASET_ID, DATASET_VERSION, DATA_FILE_LOCATION_STATUS, DATA_TIER, DATA_TIER_ANALYZED, DELIVERED_STATUS, EVENT_NUMBER, FAMILY, FAMILY_ANALYZED, FILE_NAME, FILE_PARTITION, FILE_STATUS, FULL_PATH, LOGICAL_DATASTREAM_NAME, PARAM_TYPE, RUN_ID, RUN_NUMBER, RUN_QUALITY, VERSION_ANALYZED, WORK_GRP_NAME , etc. ü __SET__ : Special dimension allowing you to include an existing dataset definition. ü Constraint operators: =, !=, >, < >=, <=, like, not like, in, not in, between, is null, is not null ü Sets operators: and, or, minus, (union, intersection to be added) ü syntax: --dim="[(]name [con. Oper] value [set. Oper name [con. Oper] value][)]. . . " ü Command line examples: – sam define dataset --defname=dataset_definition_name --group=work_group_name -dim="(run_number 100930 data_tier digitized) minus physical_datastream_name electron+jet" – sam create dataset --defname=dataset_definition_name 12 -16 May, 2003 Lee Lueking, EDG Int. Proj. Conf. 16

12 -16 May, 2003 Lee Lueking, EDG Int. Proj. Conf. 17

12 -16 May, 2003 Lee Lueking, EDG Int. Proj. Conf. 18

SAM User API ü Lightweight python interface to the sam command suite allowing multiple sam tasks to be performed and the results manipulated according to the users desire. ü For example: import Sam. User. Api sam = Sam. User. Api() provides an object which has all the needed sam functionality. ü So starting up sam file delivery tasks and querying the delivery status of each file and building lists of files which had problems and need to be retried. ü Allows simple, dynamic control and tailoring of file delivery on the fly based on what is happening with a job. ü For example, submitting processing jobs as files become available to optimise resource usage. Eg, if only a few files are available at a time then only a few jobs are started, but if more files arrive, then more jobs can be started. 12 -16 May, 2003 Lee Lueking, EDG Int. Proj. Conf. 19

Monte Carlo Request System ü User defines required data in terms of a set of metadata keyword/values which define the physics details of the requested MC sample. ü This is then stored in SAM and when the request is processed, this physics data is extracted, and augmented with further 'processing mechanics' information and converted into executable jobs which are tailored to the resource they are executed on. ü The resulting data is stored in SAM with the physics metadata augmented by the details of the workflow and data provenance. ü Essentially it provides a metadata materialization service (a. k. a. virtual data system). 12 -16 May, 2003 Lee Lueking, EDG Int. Proj. Conf. 20

EDG and SAM Commands Preliminary – to generate discussion 12 -16 May, 2003 Lee Lueking, EDG Int. Proj. Conf. 21

Storage Management Commands EDG Command Action SAM equivalent and Comment copy. And. Register. File (cp) Store and register Sam store replicate. File (rep) Replicate a file Station cache operation delete. File (d. El) Remove file and unregister Rm file and “sam undeclare”, Not allowed for files with existing links 12 -16 May, 2003 Lee Lueking, EDG Int. Proj. Conf. 22

Catalog Commands EDG Command Action SAM equivalent and comment register. File (rf) Register file in catalog Sam declare register. GUID (rg) Register file with known Sam add location GUID in catalog unregister. File (uf) Unregister file from catalog Sam undeclare, Not allowed for files with existing links list. Replicas (lr) List replicas Sam get file location list. GUID (lg) List GUID of LFN or SFN Sam translate constraints (possibly) add. Alias Add an LFN alias to existing GUID Sam create dataset 12 -16 May, 2003 Lee Lueking, EDG Int. Proj. Conf. 23

Catalog and File Transfer Commands EDG Command Action SAM equivalent get. Best. File (gbf) Replicate a file from best source Done by station in global routing list. Best. File (lbf) List replica with smallest access cost Internal to station get. Access. Cost (ac) List access costs for all Internal to station replicas copy. File (cp) Copy a file to local destination 12 -16 May, 2003 Done via project definition and project manager Lee Lueking, EDG Int. Proj. Conf. 24

Additional SAM Commands (of possible interest) ü Some are tied to storage management, and not strictly the file metadata or file replica catalog. ü Many other administrative commands for controlling station, auto-destination map, and monitoring. 12 -16 May, 2003 SAM Object Possible Actions via Commands File Declare, store, dump, erase, get metadata, insert crc, mark content status File physical locations Add, erase, mark status Dataset definitions create Dataset Create (made from DS definition) Projects Get next file, create project, create consumer Mc request Create, get details, modify details, modify status, Lee Lueking, EDG Int. Proj. Conf. 25

Summary ü SAM is distributed, end-to-end Data Management and Handling tool providing the ability to store, and access data and associated metadata information. ü The SAM Database Schema provides many capabilities to maintain physics and processing related information about the data. ü There are many commonalities between the EDG and SAM concepts and the commands for management and access can be readily mapped. ü At this meeting I hope we can plant the seeds needed to achieve the common interfaces which will allow the EDG wp 2 and SAM to provide replica services for both EDG and SAM-Grid. 12 -16 May, 2003 Lee Lueking, EDG Int. Proj. Conf. 26

Thank You 12 -16 May, 2003 Lee Lueking, EDG Int. Proj. Conf. 27

SAM Station: Dzero Distributed Cache Reconstruction Farm • Network • Each Stager Node accesses Enstore (MSS) directly • Worker nodes get data from stagers. • Intra-station data transfers are “cheap” • Job Dispatch • Fermi Batch System • A job runs on many nodes. • Goal is to distribute files evenly among workers SAM manages replicas within a cluster too Enstore Mass Storage High Speed Switch Worker 1 2 3 SAM Stager 12 -16 May, 2003 SAM Stager Lee Lueking, EDG Int. Proj. Conf. SAM Master Node Station “D 0 bbin” Servers SAM Stager 10 Worker N SAM Stager 28

SAM Station: Shared Cache Configuration w/ PN (used at Grid. Ka and U. Michigan NPACI) Gateway Node • Network • Gateway node has acces to the intrenet • Worker nodes are on VPN • Job Dispatch • PBS or other local Batch System • Appropriate adapter for SAM • Software and Data Access • Common disk server is NFS mounted to Gateway and Worker nodes Local Naming Service Calibration DB Servers SAM Station Servers SAM Stagers WA N May be optional RAID Server Virtual Private Network Worker 1 2 3 12 -16 May, 2003 Firewall Lee Lueking, EDG Int. Proj. Conf. Worker N 29

Data to and from Remote Sites Data Forwarding and Routing Station Configuration • Replica location • Prefer • Avoid • Forwarding • File stores can be forwarded through other stations • Routing • Routes for file transfers are configurable 12 -16 May, 2003 SAM Station 1 SAM Station 2 Remote SAM Station MSS SAM Station 3 SAM Station 4 Remote SAM Station Extra-domain transfers use bbftp or Grid. FTP (parallel transfer protocols) Lee Lueking, EDG Int. Proj. Conf. 30