The EU Data Grid The European Data Grid

The EU Data. Grid The European Data. Grid Project Team http: //www. eu-datagrid. org/ Erwin. Laure@cern. ch

Tutorial Roadmap Ø Project Introduction Ø Security Architecture Ø The EDG Testbed Coffee Break Ø Ø Specific Middleware Issues n Job Management n Data Management n Monitoring & Fabric Management Application Examples The EDG Intro– Tutorial - n° 2

The EU Data. Grid Project Introduction The European Data. Grid Project Team http: //www. eu-datagrid. org/

Contents Ø The EDG Project scope Ø Achievements Ø EDG structure Ø Middleware Workpackages: Goals, Achievements Ø Data. Grid in Numbers Ø Relation to Sister Projects The EDG Intro– Tutorial - n° 5

The Grid Vision Ø Flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions, and resource n Ø From “The Anatomy of the Grid: Enabling Scalable Virtual Organizations” Enable communities (“virtual organizations”) to share geographically distributed resources as they pursue common goals -- assuming the absence of… n central location, n central control, n omniscience, n existing trust relationships. The EDG Intro– Tutorial - n° 6

Grids: Elements of the Problem Ø Resource sharing n n Ø Sharing always conditional: issues of trust, policy, negotiation, payment, … Coordinated problem solving n Ø Computers, storage, sensors, networks, … Beyond client-server: distributed data analysis, computation, collaboration, … Dynamic, multi-institutional virtual orgs n Community overlays on classic org structures n Large or small, static or dynamic The EDG Intro– Tutorial - n° 7

Goals Ø Ø Data. Grid is a project funded by European Union whose objective is to exploit and build the next generation computing infrastructure providing intensive computation and analysis of shared large-scale databases. Enable data intensive sciences by providing world wide Grid test beds to large distributed scientific organizations ( “Virtual Organizations, Vos”) Start ( Kick off ) : Jan 1, 2001 Ø End : Dec 31, 2003 Ø Applications/End Users Communities : HEP, Earth Observation, Biology Ø Specific Project Objectives: n Middleware for fabric & grid management n Large scale testbed n Production quality demonstrations n n Collaborate and coordinate with other projects (Globus, Condor, Cross. Grid, Data. TAG, etc) Contribute to Open Standards and international bodies ( GGF, Industry&Research forum) The EDG Intro– Tutorial - n° 8

Data. Grid Main Partners Ø CERN – International (Switzerland/France) Ø CNRS - France Ø ESA/ESRIN – International (Italy) Ø INFN - Italy Ø NIKHEF – The Netherlands Ø PPARC - UK The EDG Intro– Tutorial - n° 9

Assistant Partners Industrial Partners • Datamat (Italy) • IBM-UK (UK) • CS-SI (France) Research and Academic Institutes • CESNET (Czech Republic) • Commissariat à l'énergie atomique (CEA) – France • Computer and Automation Research Institute, Hungarian Academy of Sciences (MTA SZTAKI) • Consiglio Nazionale delle Ricerche (Italy) • Helsinki Institute of Physics – Finland • Institut de Fisica d'Altes Energies (IFAE) - Spain • Istituto Trentino di Cultura (IRST) – Italy • Konrad-Zuse-Zentrum für Informationstechnik Berlin - Germany • Royal Netherlands Meteorological Institute (KNMI) • Ruprecht-Karls-Universität Heidelberg - Germany • Stichting Academisch Rekencentrum Amsterdam (SARA) – Netherlands • Swedish Research Council - Sweden The EDG Intro– Tutorial - n° 10

Project Schedule Ø Project started on 1/Jan/2001 Ø Testbed 0 (early 2001) n International test bed 0 infrastructure deployed s Ø Testbed 1 ( 2002 ) n First release of EU Data. Grid software to defined users within the project: s Ø Ø HEP experiments (WP 8), Earth Observation (WP 9), Biomedical applications (WP 10) Testbed 2 (End 2002) n Builds on Testbed 1 to extend facilities of Data. Grid n Focus on production quality Testbed 3 (2003) n Ø Globus 1 only - no EDG middleware Advanced functionality; currently being deployed. Project stops on 31/Dec/2003 The EDG Intro– Tutorial - n° 11

Data. Grid Work Packages Ø The EDG collaboration is structured in 12 Work Packages n WP 1: Work Load Management System n WP 2: Data Management n WP 3: Grid Monitoring / Grid Information Systems n WP 4: Fabric Management n WP 5: Storage Element n WP 6: Testbed and demonstrators – Production quality International Infrastructure n WP 7: Network Monitoring n WP 8: High Energy Physics Applications n WP 9: Earth Observation n WP 10: Biology n WP 11: Dissemination n WP 12: Management The EDG Intro– Tutorial - n° 12

Data. Grid Architecture Local Computing Grid Local Application Local Database Grid Application Layer Data Management Job Management Metadata Management Object to File Mapping Collective Services Information & Monitoring Replica Manager Grid Scheduler Underlying Grid Services SQL Database Services Computing Element Services Storage Element Services Replica Catalog Authorization Authentication and Accounting Logging & Bookkeeping Grid Fabric services Resource Management Configuration Management Monitoring and Fault Tolerance Node Installation & Management Fabric Storage Management The EDG Intro– Tutorial - n° 13

EDG Interfaces Application Developers System Managers Local Database Scientist s Grid Application Layer Data Management Job Management File Systems Local Application Metadata Management Object to File Mapping Certificate Authorities Collective Services User Accounts Information & Monitoring Replica Manager Grid Scheduler Underlying Grid Services SQL Database Services Computing Element Services Storage Element Services Replica Catalog Authorization Authentication and Accounting Logging & Bookkeeping Fabric services Resource Management Configuration Management Monitoring and Fault Tolerance Node Installation & Management Fabric Storage Management Operating Systems Mass Storage Systems HPSS, Castor Storage Elements Batch Systems PBS, LSF Computing Elements The EDG Intro– Tutorial - n° 14

WP 1: Work Load Management Local Application Grid Application Layer Data Management Job Management Ø Goals n Ø Maximize use of resources by efficient scheduling of user jobs Achievements n n Definition of architecture for scheduling & res. mgmt. and accounting & reservation Local Database Metadata Management Object to File Mapping Collective Services Information & Monitoring Replica Manager Grid Scheduler Underlying Grid Services SQL Database Services Computing Element Services Storage Element Services Replica Catalog Authorization Authentication and Accounting Logging & Bookkeeping Fabric services Resource Management Configuration Management Monitoring and Fault Tolerance Node Installation & Management Fabric Storage Management Development of "super scheduling" component using application data and computing elements requirements n Support for MPI jobs n Logical job check pointing n Interactive jobs The EDG Intro– Tutorial - n° 15

EDG middleware architecture: The Workload Management System (WP 1) Ø WP 1 is responsible for the Workload Management System (WMS). The WMS is currently composed by the following parts: n User Interface (UI) : access point for the user to the GRID ( using JDL) n Resource Broker (RB) : the broker of GRID resources, matchmaking n Job Submission System (JSS) : Condor-G; interfacing batch systems n n Information Index (II) : an LDAP server used as a filter to select resources Logging and Bookkeeping services (LB) : My. SQL databases to store Job Info The EDG Intro– Tutorial - n° 16

WP 1: Work Load Management Local Application Grid Application Layer Job Managem. Components Job Description Language Information Index User Interface Logging & Bookkeeping Service Ø Grid Scheduler UI : python (LB client : C++) n RB : C++ n Metadata Managem. Object to File Mapping Replica Manager Info & Monitor Underlying Grid Services SQL Database Services Computing Element Services Storage Element Services Replica Catalog Authorization Authentication Accounting Logging & Bookkeeping Fabric services Resource Managem. Implementation: n Data Managem. Collective Services Resource Broker Job Submission Service Local Database Ø Config Management Monitoring Fault Tolerance Node Installation Management Fabric Storage Management WMS main interfaces: n Globus Gatekeeper JSS : C++, python n WP 2 Replica Catalog APIs n II : LDAP server n WP 3 Information Systems LB: My. SQL, C++ n n WP 7 network monitoring info providers End User (using JDL files, on the UI) n Input/Output Sandboxes: Grid. FTP n The EDG Intro– Tutorial - n° 17

WP 2: Data Management Ø Goals n Ø Coherently manage and share petabyte-scale information volumes in high-throughput production-quality grid environments Achievements n n n Survey of existing tools and technologies for data access and mass storage systems Definition of architecture for data management Local Database Grid Application Layer Data Management Job Management Metadata Management Object to File Mapping Collective Services Information & Monitoring Replica Manager Grid Scheduler Underlying Grid Services SQL Database Services Computing Element Services Storage Element Services Replica Catalog Authorization Authentication and Accounting Logging & Bookkeeping Fabric services Resource Management Configuration Management Monitoring and Fault Tolerance Node Installation & Management Fabric Storage Management Deployment of Grid Data Mirroring Package (GDMP) in Testbed 1 Deployment of EDG Replica Manager in Testbed 2 Close collaboration with Globus, PPDG/Gri. Phy. N & Condor s n Local Application Common design of RLS Working with GGF on standards The EDG Intro– Tutorial - n° 18

EDG middleware architecture: WP 2 (Data Management ) Ø WP 2 is responsible for Data Management, which includes file and replica management, metadata access and data security. WP 2 components: Ø Ø Replica Manager: the main manager for triggering replica execution all over the GRID, including replica optimization and interfacing the replica catalog service Replica Catalog: a GRID service used to resolve Logical File Names into a set of corresponding Physical File Names – Globus Replica Catalog and Replica Location Service (RLS) GDMP: the GRID Data Mirroring Package, used to create replicas of any filetype all over the GRID Storage Elements in a synchronized way, by automatic updating the replica catalog Spitfire: provides a Grid enabled middleware service for access to relational databases : it consists of the Spitfire Server module and the Spitfire Client libraries and command line executables. The EDG Intro– Tutorial - n° 19

WP 2: Data Management Local Application Grid Application Layer Job Managem. Deployed Components Grid. FTP GDMP Spitfire Grid Scheduler SQL Database Services RM: C++ n Reptor: Java based Web Services RC : Globus Replica Catalog wrapper n Object to File Mapping Replica Manager Info & Monitor Computing Element Services Storage Element Services Replica Catalog Authorization Authentication Accounting Logging & Bookkeeping Fabric services Implementation: n Metadata Managem. Underlying Grid Services Resource Managem. Ø Data Managem. Collective Services Replica Manager - edg-replicamanager and Reptor Replica Catalog - globus-replicacatalog Local Database Ø Config Management Monitoring Fault Tolerance Node Installation Management Fabric Storage Management WP 2 main interfaces: n The GRID Storage Element n WP 1 Resource Broker APIs n WP 3 GRID Info services WP 7 network monitoring info providers n GDMP : C++ n n Spitfire : Java, Web Services n End User (using GDMP) The EDG Intro– Tutorial - n° 20

WP 3: Grid Monitoring Services Ø Goals n Ø Provide information system for discovering resources and monitoring status Achievements Local Application Grid Application Layer Information & Monitoring n Coordination of schemas in testbed 1 Fabric services n Object to File Mapping Replica Manager Grid Scheduler Underlying Grid Services Survey of current technologies n Metadata Management Collective Services n Development of Ftree caching backend based on Open. LDAP (Light Weight Directory Access Protocol) to address shortcoming in MDS v 1 Data Management Job Management SQL Database Services n Local Database Resource Management Computing Element Services Configuration Management Storage Element Services Replica Catalog Monitoring and Fault Tolerance Authorizat ion Logging & Authentication and Accounting Book-keeping Node Installation & Management Fabric Storage Management Relational Grid Monitoring Architecture (RGMA) GRM and PROVE adapted to grid environments to support end-user application monitoring The EDG Intro– Tutorial - n° 21

WP 3 : GRID Monitoring and Info Providers Ø WP 3’s task is to provide information about The Grid itself This includes information about resources (Computing. Elements, Storage. Elements and the Network), for which the Globus MDS is a common solution; and job status information (as implemented by WP 1's Logging and Bookkeeping). Grid applications This is information published by user jobs. This is used for performance monitoring. Ø R-GMA n relational implementation of the GGF GMA n interoperable with MDS The EDG Intro– Tutorial - n° 22

WP 3: GRID Monitoring Local Application Grid Application Layer Job Managem. Components SQL Database Services FTree : Open. LDAP, caching R-GMA : Java , C++, My. SQL, Tom. Cat n n GRM / PROVE : P-GRADE Info & Monitor Computing Element Services Storage Element Services Replica Catalog Authorization Authentication Accounting Logging & Bookkeeping Fabric services Implementation: n Object to File Mapping Underlying Grid Services GRM/Prove MDS : LDAP, Globus GRIS, GIIS Metadata Managem. Replica Manager Grid Scheduler R-GMA n Data Managem. Collective Services MDS / FTree Ø Local Database Resource Managem. Ø Config Management Monitoring Fault Tolerance Node Installation Management Fabric Storage Management WP 3 main interfaces: n WP 1 Resource Broker ( Info. Index) n WP 2 RM optimizer n n all GRID services producing info (SE, CE. . ) WP 7 network monitoring The EDG Intro– Tutorial - n° 23

WP 4: Fabric Management Ø Goals n Ø manage clusters (~thousands) of nodes Achievements n n n Survey of existing tools, techniques and protocols Local Application Local Database Grid Application Layer Data Management Job Management Metadata Management Object to File Mapping Collective Services Information & Monitoring Replica Manager Grid Scheduler Underlying Grid Services SQL Database Services Computing Element Services Storage Element Services Replica Catalog Authorization Logging & Authentication Book-keeping and Accounting Fabric services Resource Management Configuration Management Monitoring and Fault Tolerance Node Installation & Management Fabric Storage Management Defined an agreed architecture for fabric management Initial implementations deployed at several sites in testbed 1 & 2 The EDG Intro– Tutorial - n° 24

EDG middleware architecture: WP 4 : Fabric Management Components Ø Ø Ø WP 4 is responsible to deliver a computing fabric comprised of all the necessary tools to manage a center providing grid services on clusters of thousands of nodes. The computing fabric is called the Computing Element in EDG. User Job Control and Management (Grid and local jobs) on fabric batch and/or interactive CPU services n Gridification – Grid interface to fabric resources n Resource Management – manage underlying batch services Automated System Administration for Computing Fabric Elements. These subsystems are reserved for system administrators and operators for performing system maintenance n Configuration Management n Installation Management n Fabric Monitoring The EDG Intro– Tutorial - n° 25

WP 4: Fabric Management Local Application Local Database Grid Application Layer Job Managem. Data Managem. Metadata Managem. Object to File Mapping Components Collective Services LCFG Replica Manager Grid Scheduler Fabric Monitoring PBS & LSF info providers Info & Monitor Underlying Grid Services SQL Database Services Image installation Computing Element Services Storage Element Services Replica Catalog Authorization Authentication Accounting Logging & Bookkeeping Config. Cache Mgr Fabric services Resource Managem. Ø Implementation: n LCFG : C++, XML, HTTP Ø Config Management Monitoring Fault Tolerance Node Installation Management Fabric Storage Management WP 4 main interfaces: n WP 1 Resource Broker ( Info. Index) n WP 2 Data management n WP 5 Storage Element n WP 3 GRID Info Services The EDG Intro– Tutorial - n° 26

WP 5: Mass Storage Management Ø Goals n Ø Local Application Provide common user and data export/import interfaces to existing local mass storage systems Achievements n n n Review of Grid data systems, tape and disk storage systems and local file systems Definition of Architecture and Design for Data. Grid Storage Element Local Database Grid Application Layer Data Management Job Management Metadata Management Object to File Mapping Collective Services Information & Monitoring Replica Manager Grid Scheduler Underlying Grid Services SQL Database Services Computing Element Services Storage Element Services Replica Catalog Authorization Authentication and Accounting Logging & Bookkeeping Fabric services Resource Management Configuration Management Monitoring and Fault Tolerance Node Installation & Management Fabric Storage Management Collaboration with Globus on Grid. FTP/RFIO Collaboration with PPDG on control API First attempt at exchanging Hierarchical Storage Manager (HSM) tapes SRM compliant interface to MSS The EDG Intro– Tutorial - n° 27

WP 5 : Mass Storage Management Ø Ø Ø WP 5 delivers the Grid interface to Storage. Its service, the Storage Element (SE) is interfacing to underlying Mass Storage Systems or simple storage services. Main interfaces: n n n Data, gridftp will be used to transfer files over the WAN and the files will optionally be available to local nodes by NFS. Information, Existing MDS information providers will be extended to provide the extra information in the GLUE storage schema. Control, functions such as reservation, pinning, deletion, and transfer time estimation. Will provide an SRM 2 interface. The EDG Intro– Tutorial - n° 28

WP 5: Mass Storage Management Ø Achievements n n n Ø Local Application Local Database Grid Application Layer Job Managem. Definition of Architecture and Design for Data. Grid storage Element Data Managem. Metadata Managem. Object to File Mapping Collective Services Replica Manager Grid Scheduler Collaboration with Globus on Grid. FTP/RFIO Info & Monitor Underlying Grid Services Collaboration with PPDG on control API SQL Database Services Staging from/to CASTOR at CERN succesfully implemented and tested Computing Element Services Storage Element Services Replica Catalog Authorization Authentication Accounting Logging & Bookkeeping Fabric services Succesfully Interfaced to GDMP Resource Managem. Config Management Monitoring Fault Tolerance Node Installation Management Fabric Storage Management Supported Storage Systems: n n UNIX disk systems HPSS (High Performance Storage System) n CASTOR (through RFIO) n Grid. FTP servers n DMF n Enstore Ø WP 5 (SE) main interfaces: n WP 1 Resource Broker & JSS n WP 2 RM, RC n WP 7 for GRIDftp monitoring n WP 3 GRID Info Services The EDG Intro– Tutorial - n° 29

WP 6: Test. Bed Integration Ø Goals n Deploy testbeds for the end-to-end application experiments & demos n Ø Integrate successive releases of the software components Achievements n n n Integration of EDG sw and deployment Working implementation of multiple Virtual Organizations (VOs) s & basic security infrastructure Definition of acceptable usage contracts and creation of Certification Authorities group n Definition of test plan n User’s, administrator’s, and developer’s guides Local Application Local Database Grid Application Layer Data Management Job Management Metadata Management Object to File Mapping Collective Services Information & Monitoring Replica Manager Grid Scheduler Underlying Grid Services SQL Database Services Computing Element Services Storage Element Services Replica Catalog Authorization Authentication and Accounting Logging & Bookkeeping Fabric services Resource Management Monitoring and Fault Tolerance Configuration Management EDG release Globus Node Installation & Management Fabric Storage Management WP 6 additions to Globus Components Globus packaging & EDG config Build tools End-user documents The EDG Intro– Tutorial - n° 30

Tasks for the WP 6 integration team Ø Ø Ø Ø Testing and integration of the Globus package Exact definition of RPM lists (components) for the various testbed machine profiles (CE service , RB, UI, SE service , NE, WN, ) – check dependencies Perform preliminary centrally (CERN) managed tests on EDG m/w before green light for spread EDG testbed sites deployment Provide, update end user documentation for installers/site managers, developers and end users Define EDG release policies, coordinate the integration team staff with the various Work. Package managers – keep high inter -coordination. Assign the reported bugs to the corresponding developers/site managers (Bug. Zilla) Complete support for the i. Team testing VO The EDG Intro– Tutorial - n° 31

WP 6: Test. Bed Integration and demonstrators Local Application Ø WP 6 goals: the EDG testbed Grid Application Layer Job n n n Managem. Integration of EDG sw releases and deployment all over the EDG testbed Collective Services : the integration team Working implementation of multiple VOs & basic security infrastructure Definition of acceptable usage contracts and creation of Certification Authorities group Set up of the Authorization Working Group to manage authorization policies on the testbed 2 Testbeds: Local Database Data Managem. Metadata Managem. Object to File Mapping Replica Manager Grid Scheduler Info & Monitor Underlying Grid Services SQL Database Services Computing Element Services Storage Element Services Replica Catalog Authorization Authenticatio n Accounting Logging & Bookkeeping Fabric services Resource Managem. Config Management Monitoring Fault Tolerance Node Installation Management Fabric Storage Management Components Support for test-VO, mkgridmap tools s Dev. TB for integration Globus packaging & EDG config s Application TB for application usage Build tools, CVS central s/w repository s Certification TB planned End-user documents The EDG Intro– Tutorial - n° 32

WP 7: Network Services Local Application Ø Goals n n Ø Local Database Review the network service requirements for Data. Grid Establish and manage the Data. Grid network facilities Monitor the traffic and performance of the network Deal with the distributed security aspects Achievements n n n Analysis of network requirements for testbed 1 & study of available network physical infrastructure Use of European backbone GEANT since Dec. 2001 Initial network monitoring architecture defined and first tools deployed Collaboration with Dante & Data. TAG Working with GGF (Grid High Performance Networks) & Globus (monitoring/MDS) Network cost estimation for workload and data management Grid Application Layer Data Management Job Management Metadata Management Object to File Mapping Collective Services Information & Monitoring Replica Manager Grid Scheduler Underlying Grid Services SQL Database Services Computing Element Services Storage Element Services Replica Catalog Authorization Authentication and Accounting Logging & Bookkeepgin Fabric services Resource Management Configuration Management Monitoring and Fault Tolerance Node Installation & Management Fabric Storage Management Components network monitoring tools: Ping. ER Udpmon Iperf The EDG Intro– Tutorial - n° 33

Applications (WP 8 -10) Earth Observation Science Applications Biomedical Applications High Energy Physics The EDG Intro– Tutorial - n° 34

Grid aspects covered by EDG VO servers LDAP directory for mapping users (with certificates) to correct VO Storage Element Grid-aware storage area, situated close to a CE User Interface Submit & monitor jobs, retrieve output Replica Manager Replicates data to one or more CEs Job Submission Service Manages submission of jobs Replica Catalog Keeps track of multiple data files “replicated” on different CEs Information index Provides info about grid resources via GIIS/GRIS hierarchy Information & Monitoring Provides info on resource utilization & performance Resource Broker Uses Info Index to discover & select resources based on job requirements Grid Fabric Mgmt Configure, installs & maintains grid sw packages and environ. to Res. Broker Logging and Bookkeeping Collects resource usage & job status Computing Element Gatekeeper to a grid computing resource Network performance, Provides efficient network security and monitoring transport, security & bandwidth monitoring Testbed admin. Certificate auth. , user reg. , usage policy etc. The EDG Intro– Tutorial - n° 35

Data. Grid in Numbers People Testbeds >350 registered users >15 regular sites 12 Virtual Organisations >10’ 000 s jobs submitted 16 Certificate Authorities >1000 CPUs >200 people trained 278 man-years of effort >5 Tera. Bytes disk 3 Mass Storage Systems 100 years funded Software 50 use cases 18 software releases >300 K lines of code Scientific applications 5 Earth Obs institutes 9 bio-informatics apps 6 HEP experiments The EDG Intro– Tutorial - n° 36

Related Grid Projects Through links with sister projects, there is the potential for a truely global scientific applications grid Demonstrated at IST 2002 and SC 2002 in November The EDG Intro– Tutorial - n° 37