The EDG Middleware User Interface Flavia Donno for

The EDG Middleware User Interface Flavia Donno for IT and WPs flavia. donno @pi. infn. it

Outline u The u Structure of a Virtual Organization n Authorization via LDAP servers n The MDS for VOs n The Replica Catalog Overview of the EDG services n The WP 1 services. An overview of the Job Submission. n GDMP (Grid Data Mirroring Package) - WP 2/PPDG n Spitfire - WP 2 n WP 3/R-GMA, GRM/PROVE n WP 7/Network Monitoring Tools. Flavia Donno The EDG Middleware User Interface ° 2

The Structure of a Virtual Organization u Authorization via LDAP servers User 1 User 2. . . Group 1 Group 2. . . LDAP-CA LDAP-VO Grid-mapfile /C=FR/O=CNRS/OU=CPPM/C… /C=IT/O=INFN/L=Torino/CN=. . . /C=FR/O=CNRS/OU=CC-LYON. . . ##### GROUP: group URL [lcl_user] group ldap: //ldapgrid. cnaf. infn. it/cn=group 1, dc=testbed 2, dc=org tb 2 group ldap: //ldapgrid. cnaf. infn. it/cn=group 3, dc=testbed 6, dc=org group ldaps: //ldapgrid. cnaf. infn. it/cn=group 2, dc=testbed 4, dc=org tb 4 ##### ACL: deny|allow pattern_to_match deny *L=Parma* allow *INFN* allow *CESNET* allow *dutchgrid* ##### DEFAULT LOCAL USER default_lcluser testbed 1 ##### GRID-MAPFILE-LOCAL gmf_local. . /etc/grid-mapfile-local The System Administrator has complete control over the grid-mapfile Flavia Donno The EDG Middleware User Interface ° 3

The Structure of a Virtual Organization u The MDS (Information System) for VO n It contains info related to resources of the GRID u The n s Computing Elements s Storage Elements s Network Monitoring s Site Info Static and dynamic VO Replica Catalogue for Data Info It contains the mapping between the Data Logical File Names and Physical Locations. No support for Collections. Flavia Donno The EDG Middleware User Interface ° 4

The WP 1 services. An overview of the Job Submission. u User Interface (UI) n n Command-line for job management operations (job submission, job removal, job status monitoring, logging info visualization) Job Description Language (JDL) used when the job is submitted, to specify its characteristics and requirements s n u Condor Class. Ads based RB & LB Client Resource Broker (RB) n n n Responsible to choose the “best” CE’s where to submit jobs Matchmaking between requirements specified in the JDL expression and available resources (published in the MDS + Replica Catalog) It uses Logging Client. Flavia Donno The EDG Middleware User Interface ° 5

The WP 1 services. An overview of the Job Submission. u u u Job Submission Service (JSS) n Job management operations and responsible to keep tracks of submitted jobs n Wrapper of Condor-G n It runs together with the RB. It uses Logging Client. Information Index (II) n Used by the RB as first filter in the resource discovery process n Application of Globus GIIS Logging and Bookkeeping Services (L&B) n Bookkeeping: short term (volatile) data about currently active jobs n Logging: long term (persistent) info about jobs and workload management system Flavia Donno The EDG Middleware User Interface ° 6

Brokerinfo Flavia Donno The EDG Middleware User Interface ° 7

WP 1 PM 9 Integration WP 2/Replica Catalogue II Input Sandbox UI JDL RB Input Sandbox Job Submit Event Output Sandbox L&B SE Brokerinfo JSS Output Sandbox Job Status CE Flavia Donno The EDG Middleware User Interface ° 8

The WP 1 services. An overview of the Job Submission. dg-job-submit jobad 6. jdl -o jobs_list -n elisabetta. ronchieri@cnaf. infn. it # # ----- Job Description File ----# Executable = "WP 1 test. C"; Std. Input = "sim. dat"; Std. Output = "sim. out"; Std. Error = "sim. err"; Input. Sandbox = {"/home/wp 1/Hands. On-0409/WP 1 test. C", "/home/wp 1/Hands. On-0409/file*”, "/home/wp 1/DATA/*"}; Output. Sandbox = {"sim. err", "test. out", "sim. out"}; Rank = other. Average. SI 00; Requirements = (other. Op. Sys == "Linux RH 6. 1" || other. Op. Sys == "Linux RH 6. 2”) && (other. Run. Time. Environmnet == “CMS 3. 2”); Input. Data = "LF: test 10096 -0009"; Replica. Catalog = "ldap: //sunlab 2 g. cnaf. infn. it: 2010/rc=WP 2 INFN Test Replica Catalog, dc=sunlab 2 g, dc=cnaf, dc=infn, dc=it"; Data. Access. Protocol = "gridftp"; Flavia Donno The EDG Middleware User Interface ° 9

An overview of WP 1 testbed. RC Submitting machine (UI, RBclient, LBclient) RBServer, JSS, LB UI lx 01 Imperial College sunlab 2 g II CE SE Testbed 001 (CERN) INFN-CNAF grid 001 f grid 004 f CE CE Testbed 005 (CERN) SE INFN-Padova LFS - Condor CESNET-Prague PBS Testbed 002 (CERN) SE INFN-Milano PBS Flavia Donno The EDG Middleware User Interface ° 10

An overview of the Job Submission. The II % ldapsearch -v -LLL -p 2170 -h grid 001 f. cnaf. infn. it -b "o=grid" "objectclass=*" ldap_init( grid 001 f. cnaf. infn. it, 2170 ) filter pattern: objectclass=* returning: ALL filter is: (objectclass=*) dn: ce=bbq. mi. infn. it: 2119/jobmanager-pbs-dque, dc=mi, dc=infn, dc=it, o=Grid object. Class: Globus. Top object. Class: Computing. Element CEId: bbq. mi. infn. it: 2119/jobmanager-pbs-dque Globus. Resource. Contact. String: bbq. mi. infn. it: 2119/jobmanager-pbs gramversion: 1. 71 architecture: Intel Op. Sys: Linux RH 6. 2 Min. Physical. Memory: 128 Min. Local. Disk. Space: 100 Total. CPUs: 4 Free. CPUs: 2 Num. SMPs: 1 Min. SPUProcessors: 2 Max. SPUProcessors: 2 Total. Jobs: 15 Running. Jobs: 4 Idle. Jobs: 11. . . Static Dynamic Flavia Donno The EDG Middleware User Interface ° 11

An overview of the Job Submission. The II Authorized. User: /C=IT/O=INFN/L=Padova/CN=Massimo Sgaravatto/Email=massimo. sgaravatto@pd. infn. it Authorized. User: /C=IT/O=INFN/L=Milano/CN=Silvia Resconi/Email=Silvia. Resconi@mi. infn. it Authorized. User: /O=Grid/O=UKHEP/OU=hep. ph. ic. ac. uk/CN=Fabrizio Pacini Authorized. User: /C=IT/O=INFN/L=Catania/CN=salvatore cavalieri/Email=salvatore. cavalieri@ct. infn. it Run. Time. Environment: CMS 3. 2 Run. Time. Environment: EO 4. 2 dn: closese=testbed 005. cern. ch, ce=bbq. mi. infn. it: 2119/jobmanager-pbs-dque, dc =mi, dc=infn, dc=it, o=Grid object. Class: Globus. Top object. Class: Close. Storage. Element CEId: bbq. mi. infn. it: 2119/jobmanager-pbs-dque Close. SE: testbed 005. cern. ch Mount. Point: /diskmi dn: se=testbed 005. cern. ch, dc=mi, dc=infn, dc=it, o=Grid object. Class: Globus. Top object. Class: Storage. Element SEId: testbed 005. cern. ch Close. CE: bbq. mi. infn. it: 2119/jobmanager-pbs-dque SEtypearchitecture: disk SEsize: 30 SEResource. Contact. String: testbed 005. cern. ch SEmaxfilesize: 4 SEmaxdata: 2 SEmaxnumfile: 30 Flavia Donno The EDG Middleware User Interface ° 12

An overview of the Job Submission. The II dn: seprotocol=gridftp, se=testbed 005. cern. ch, dc=mi, dc=infn, dc=it, o=Grid object. Class: Globus. Top object. Class: Storage. Element. Protocol SEId: testbed 005. cern. ch SEprotocol: gridftp Port: 4000 dn: seprotocol=file, se=testbed 005. cern. ch, dc=mi, dc=infn, dc=it, o=Grid object. Class: Globus. Top object. Class: Storage. Element. Protocol SEId: testbed 005. cern. ch SEprotocol: file Flavia Donno The EDG Middleware User Interface ° 13

An overview of the Job Submission. The RC dn: dc=sunlab 2 g, dc=cnaf, dc=infn, dc=it objectclass: Globus. Organization dn: lc=test 0, rc=WP 2 INFN Test Replica Catalog, dc=sunlab 2 g, dc=cnaf, dc=infn, dc=it objectclass: top objectclass: Globus. Top objectclass: Globus. Replica. Logical. Collection filename: test 10097 -0001 filename: test 10096 -0009 filename: test 10097 -0003 filename: test 10097 -0002 filename: test 10096 -0011 filename: test 10096 -0010 filename: file 1 -28 sep filename: file 2 -28 sep filename: file 3 -28 sep filename: run 2/test_file 1 filename: run 2/test_file 2 Flavia Donno The EDG Middleware User Interface ° 14

An overview of the Job Submission. The RC dn: rf=test 10096 -0009, fin=LFROOT, lc=test 0, rc=WP 2 INFN Test Replica Catalog, dc=sunlab 2 g, dc=cnaf, dc=infn, dc=it objectclass: top objectclass: Globus. Top objectclass: Globus. Replica. Logical. File size: 100 dn: rf=test 10096 -0010, fin=LFROOT, lc=test 0, rc=WP 2 INFN Test Replica Catalog, dc=sunlab 2 g, dc=cnaf, dc=infn, dc=it objectclass: top objectclass: Globus. Top objectclass: Globus. Replica. Logical. File size: 100 Flavia Donno The EDG Middleware User Interface ° 15

An overview of the Job Submission. The RC dn: re=testbed 001. cern. ch, lc=test 0, rc=WP 2 INFN Test Replica Catalog, dc=sunlab 2 g, dc=cnaf, dc=infn, dc=it objectclass: top objectclass: Globus. Top objectclass: Globus. Replica. Info uc: testbed 001. cern. ch path: /home/andreado/ffiles filename: test 10096 -0009 filename: test 10096 -0010 dn: re=testbed 002. cern. ch, lc=test 0, rc=WP 2 INFN Test Replica Catalog, dc=sunlab 2 g, dc=cnaf, dc=infn, dc=it objectclass: top objectclass: Globus. Top objectclass: Globus. Replica. Info uc: testbed 002. cern. ch path: /home/flavia/ffiles filename: test 10096 -0011 filename: test 10096 -0009 Flavia Donno The EDG Middleware User Interface ° 16

Grid Data Mirroring Package u client/server file replication software system for read-only files n initially designed to replicate Objectivity database files n now: for any kind of file format site 2 site 1 site 3 Based on: - Globus Security Infrastructure - Globus IO/ Globus Data Conversion - Grid. FTP - Globus Replica Catalogue Flavia Donno The EDG Middleware User Interface ° 17

GDMP (2) u General read-only file replication system n subscription - consumer/producer - on demand replication n several command line tools for automatic replication n it automates most of the details. u replication steps: n pre-processing: file type specific n actual file transfer: needs to be efficient and secure n post-processing: file type specific n insert into replica catalog: name space management Flavia Donno The EDG Middleware User Interface ° 18

GDMP Client Applications u gdmp_host_subscribe n subscribe to a remote host for notification of updates u gdmp_ping n check if remote server listens on the correct port u gdmp_register_local_file n register a file in a local file catalogue (for GDMP internal use) u gdmp_publish_catalogue n publish all files in the local file catalogue n update file in the Replica Catalogue (can also be switched of) u gdmp_replicate_get n retrieve a file from a remote site + update Replica Catalogue Flavia Donno The EDG Middleware User Interface ° 19

Catalogues used in GDMP Flavia Donno The EDG Middleware User Interface ° 20

Spitfire - SQLDatabase. Service u It provides: n Meta data storage and retrieval n Query, insert, delete, update, create table, drop table n Easy to use, efficient and scalable u Unified n n Grid enabled front-end to RDBMS Well defined language and platform neutral network protocol between client and server Enables interoperability and flexibility s C/C++, Java and Python clients … s access data in local or remote … s My. SQL, Oracle, DB/2 or Postgres database s client on Linux and servers on Solaris. Flavia Donno The EDG Middleware User Interface ° 21

Example 1: get. PFNs(LFN) 1. Client does http GET to http: //sql. cern. ch/get. Physical. File. Names? lfn=“some. LFN” 2. Server template: <xsql: query xmlns: xsql=“urn: oracle-xsql” connection = “RCCatalog” select * from catalog where LFN = ‘{@lfn)}’ </xsql: query> 3. Result. Set returned as canonical XML: <ROWSET> <ROW> </ROW> <lfn> host 1. cern. ch/somepath/file 1 </lfn> <size> 10000000 </size> <lfn> host 2. cern. ch/somepath/file 2 </lfn> <size> 50000000 </size> </ROWSET> Flavia Donno The EDG Middleware User Interface ° 22

WP 3 and WP 7 middleware u WP 3 R-GMA is a monitoring and information management service for distributed resources. n Relational model with SQL support. n Producers of info register with a registry (MDS producer sensor). n Consumers query the service to find info and localize producers. u WP 3 GRM/PROVE n It is an application monitoring tool. n Monitoring trace data are collected and visualized by PROVE. u WP 7 n edg-pinger (see Frank Bonnassieux talk). It monitor RTT between Datagrid sites Flavia Donno The EDG Middleware User Interface ° 23

Demonstration in the afternoon u More u Not all delivered code has been ported to GT 2 u More n details later in the afternoon session. testing is needed for GT 2 and EDG middleware software Detailed plan in progress u First release November 15 th Flavia Donno The EDG Middleware User Interface ° 24