EGEE http goc gridsupport ac ukgridsitegocmain Enabling Grids

Мониторинг EGEE http: //goc. grid-support. ac. uk/gridsite/gocmain/ Enabling Grids for E-scienc. E НИВЦ МГУ 05. 2005 В. А. Ильин, В. В. Кореньков, А. А. Солдатов 4

Operations (SA 1, SA 2) Management Enabling Grids for E-scienc. E НИВЦ МГУ 05. 2005 В. А. Ильин, В. В. Кореньков, А. А. Солдатов 5

РДИГ SA 1 Enabling Grids for E-scienc. E Distributed ROC https: //edms. cern. ch/file/479460/4/EGEE-SA 1 -Exec. Plan-RU-v 1. 6. pdf IHEP, plus some functions provided by ITEP (user support), JINR (operational monitoring), IMPB RAS and PNPI: - serve 8 RCs, 5 new RCs to appear in next 3 months and next 5 to the end of 2005 - support RDIG SA 1 managers http: //mail. ihep. ru/Lists/roc_support/List. html - MW repository http: //grid-cvs. ihep. su - user support http: //ussup. itep. ru - Grid. Ice server http: //lcfgmon. itep. ru/gridice Distributed CIC - preparing to start in April 2005 (TA) Now some core servces are 8 x 5 supported (preparing to 24 x 7): - RB, IS, RC, My. Proxy, (regional) VO management (SINP MSU) - Grid monitoring and accounting (JINR) - CA (SINP MSU -> RRC KI), today about 300 (active) certificates, http: //lcg 20. sinp. msu. ru/CA/ - MW validation and documentation localization (KIAM RAS), http: //www. gridclub. ru НИВЦ МГУ 05. 2005 В. А. Ильин, В. В. Кореньков, А. А. Солдатов 8

Computing Resources: Feb 2005 Enabling Grids for E-scienc. E Country providing resources Country anticipating joining In LCG-2: ð 113 sites, 30 countries ð >10, 000 cpu ð ~5 PB storage Includes non-EGEE sites: • 9 countries • 18 sites НИВЦ МГУ 05. 2005 В. А. Ильин, В. В. Кореньков, А. А. Солдатов 19

Infrastructure metrics Enabling Grids for E-scienc. E Region countries sites cpu M 6 (TA) cpu M 15 (TA) cpu actual CERN 0 1 900 1800 942 UK/Ireland 2 19 100 2200 2398 France 1 8 400 895 886 Italy 1 20 553 679 1777 South East 5 7 146 322 133 South West 2 12 250 498 Central Europe 5 8 385 730 373 Northern Europe 2 4 2000 427 Germany/Switzerland 2 10 100 400 1207 Russia 1 6 50 152 238 EGEE-total 21 95 3084 9428 8879 USA 1 3 - - 458 Canada 1 6 - - 316 Asia-Pacific 6 8 - - 394 Hewlett-Packard 1 1 - - 100 Total other 9 18 - - 1268 Grand Total 30 113 - - 10147 НИВЦ МГУ 05. 2005 В. А. Ильин, В. В. Кореньков, А. А. Солдатов Countries, sites, and CPU available in EGEE production service EGEE partner regions Other collaborating sites 20

Service Usage Enabling Grids for E-scienc. E • VOs and users on the production service – Active HEP experiments: § 4 LHC, D 0, CDF, Zeus, Babar – Active other VO: § Biomed, ESR (Earth Sciences), Compchem, Magic (Astronomy), EGEODE (Geo-Physics) – 6 disciplines – Registered users in these VO: 500 – In addition to these there are many VO that are local to a region, supported by their ROCs, but not yet visible across EGEE • Scale of work performed: – LHC Data challenges 2004: § >1 M SI 2 K years of cpu time (~1000 cpu years) § 400 TB of data generated, moved and stored § 1 VO achieved ~4000 simultaneous jobs (~4 times CERN grid capacity) Number of jobs processed/month НИВЦ МГУ 05. 2005 В. А. Ильин, В. В. Кореньков, А. А. Солдатов 21

Current production software (LCG-2) Enabling Grids for E-scienc. E • Evolution through 2003/2004 – Focus has been on making these reliable and robust § rather than additional functionality • – Respond to needs of users, admins, operators The software stack is the following: – Virtual Data Toolkit § Globus (2. 4. x), Condor, etc – EU Data. Grid project developed higher-level components § § Workload management (RB, L&B, etc) Replica Location Service (single central catalog), replica management tools R-GMA as accounting and monitoring framework VOMS being deployed now – Operations team re-worked components: § Information system: MDS GRIS/GIIS LCG-BDII § edg-rm tools replaced and augmented as lcg-utils § Developments on: • Disk pool managers (d. Cache, DPM) o Not addressed by JRA 1 – Other tools as required: § e. g. Grid. Ice – EU Data. Tag project НИВЦ МГУ 05. 2005 В. А. Ильин, В. В. Кореньков, А. А. Солдатов 22

The deployment process Enabling Grids for E-scienc. E • Key point – a certification process is essential – However, it is expensive (people, resources, time) – But, this is the only way to deliver production quality services – LCG-2 was built from a wide variety of “research” quality code § Lots of good ideas, but little attention to the “mundane” needs of production … – Building a reliable distributed system is hard – § Must plan for failure, must provide fail-over of services, etc – Integrating components from different projects is also difficult § Lack of common standards for logging, error recovery, etc НИВЦ МГУ 05. 2005 В. А. Ильин, В. В. Кореньков, А. А. Солдатов 23

SA 1 – Operations Structure Enabling Grids for E-scienc. E • • Operations Management Centre (OMC): – At CERN – coordination etc Core Infrastructure Centres (CIC) – Manage daily grid operations – oversight, troubleshooting Run essential infrastructure services Provide 2 nd level support to ROCs UK/I, Fr, It, CERN, + Russia (M 12) Taipei also run a CIC • • НИВЦ МГУ 05. 2005 В. А. Ильин, В. В. Кореньков, А. А. Солдатов – – Regional Operations Centres (ROC) – Act as front-line support for user and operations issues – Provide local knowledge and adaptations – One in each region – many distributed User Support Centre (GGUS) – In FZK – manage PTS – provide single point of contact (service desk) – Not foreseen as such in TA, but need is clear 24

Grid Operations Enabling Grids for E-scienc. E • • RC RC ROC RC CIC RC RC • ROC RC CIC RC RC The grid is flat, but Hierarchy of responsibility – Essential to scale the operation CICs act as a single Operations Centre – Operational oversight (grid RC – – – OMC CIC RC RC RC ROC RC RC RC • RC RC = Resource Centre НИВЦ МГУ 05. 2005 RC ROC • operator) responsibility rotates weekly between CICs Report problems to ROC/RC ROC is responsible for ensuring problem is resolved ROC oversees regional RCs – ROCs responsible for organising the operations in a region – Coordinate deployment of middleware, etc CERN coordinates sites not associated with a ROC В. А. Ильин, В. В. Кореньков, А. А. Солдатов 25

Accounting views Enabling Grids for E-scienc. E Summed CPU (Seconds) consumed by resources in selected Region Select date range Accounting menu may be used to select different views of the data Aggregate data across an organisation structure (Default= All ROCs) НИВЦ МГУ 05. 2005 В. А. Ильин, В. В. Кореньков, А. А. Солдатов Select VOs (Default = All) Web form to apply selection criteria on the data 26

Policy – Joint Security Group Enabling Grids for E-scienc. E Incident Response Certification Authorities Usage Rules Security & Availability Policy Audit Requirements Best practice Guides Application Development & Network Admin Guide User Registration http: //cern. ch/proj-lcg-security/documents. html НИВЦ МГУ 05. 2005 В. А. Ильин, В. В. Кореньков, А. А. Солдатов 27

g. Lite Services for Release 1 Software stack and origin (simplified) Enabling Grids for E-scienc. E • Computing Element – Gatekeeper (Globus) – Condor-C (Condor) – CE Monitor (EGEE) – Local batch system (PBS, LSF, • Catalog – File and Replica Catalog (EGEE) – Metadata Catalog (EGEE) • Information and Monitoring Condor) – R-GMA (EDG) • Workload Management • Security – WMS (EDG) – VOMS (Data. TAG, EDG) – Logging and bookkeeping (EDG) – GSI (Globus) – Condor-C (Condor) – Authentication for C and Java based (web) services (EDG) • Storage Element – File Transfer/Placement (EGEE) – glite-I/O (Ali. En) – Grid. FTP (Globus) – SRM: Castor (CERN), d. Cache (FNAL, DESY), other SRMs НИВЦ МГУ 05. 2005 В. А. Ильин, В. В. Кореньков, А. А. Солдатов 28

Main Differences to LCG-2 Enabling Grids for E-scienc. E • Workload Management System works in push and pull mode • Computing Element moving towards a VO based scheduler guarding the jobs of the VO (reduces load on GRAM) • Distributed and re-factored file & replica catalogs • Secure catalogs (based on user DN; VOMS certificates being integrated) • Scheduled data transfers • SRM based storage • Information Services: R-GMA with improved API and registry replication • Prototypes of additional services – Grid Access Service (GAS) – Package manager – DGAS based accounting system – Job provenance service • Move towards Web Services НИВЦ МГУ 05. 2005 В. А. Ильин, В. В. Кореньков, А. А. Солдатов 29

Standards Enabling Grids for E-scienc. E • Web Services Fast moving area – Follow WSRF and related standards but are not early adopters – WS-I compatibility is a target § Challenging to write WSDL which is WS-I compatible AND can be processed by all the tools – Industry strength tooling not always available – Trying to keep back from the bleeding edge • Work on standards bodies – Active contributions to § GGF OGSA-WG § § • GMA in OGSA • Data Design team GGF INFOD-WG OASIS WS-N GGF GSM-WG (SRM) • Co-chairing WG Replica Registration Service – And following many, many others – Adopting mature standards is a goal НИВЦ МГУ 05. 2005 В. А. Ильин, В. В. Кореньков, А. А. Солдатов 30

Release Timeline Enabling Grids for E-scienc. E First public nightly build (B 39) RC 1 (I 20041217 B 151) Release 1. 0 Builds First automated build (B 1) RC 1 (I 20050204 B 206) First Integration build (I 20041020 B 80) VOMS, Site Configuration Prototype Available to ARDA users I/O Client I/O Server June 2004 НИВЦ МГУ 05. 2005 July 2004 Aug 2004 Data Local Transfer Service, Single Catalog CE, LB, WMS, WN Functionality May 2004 Ali. En, R-GMA Sep 2004 Oct 2004 В. А. Ильин, В. В. Кореньков, А. А. Солдатов Nov 2004 Dec 2004 Jan 2005 Feb 2005 March 2005 Today April 2005 31

RDIG in MW evaluation and testing Enabling Grids for E-scienc. E Testing/adaptation MW components (SA 1): IHEP, PNPI, JINR IHEP will participate in the pre-production testing/adaptation of g. Lite (SA 1) Testing new MW components (NA 4 ARDA): - Metadata catalog, Fireman catalog, grid. FTP, . . . (JINR, SINP MSU) - testing g. Lite for ATLAS and CMS (PNPI, SINP MSU) EGEE work plan : January-March 2005: evaluation OMII (JINR, KIAM RAS) April-October 2005: evaluation GT 4 (SINP MSU, JINR, KIAM RAS) НИВЦ МГУ 05. 2005 В. А. Ильин, В. В. Кореньков, А. А. Солдатов 32

Enabling Grids for E-scienc. E SINP MSU+ INFN (Padua) new mw - improved job flow НИВЦ МГУ 05. 2005 В. А. Ильин, В. В. Кореньков, А. А. Солдатов CERN-INTAS meeting, 14 March 2005, CERN 33

Enabling Grids for E-scienc. E SINP MSU: new mw - monitoring of application jobs § No LCG MW modification required (wrappers + additional server); § Access to the intermediate job output via Web-interface; § Authorization is based on the standard GSI certificates and proxy certificates § Starting Web-page for interested users (with instructions): http: //grid. sinp. msu. ru/acgi-bin/welcome. cgi НИВЦ МГУ 05. 2005 В. А. Ильин, В. В. Кореньков, А. А. Солдатов CERN-INTAS meeting, 14 March 2005, CERN 34

Enabling Grids for E-scienc. E New MW: JINR, KIAM RAS, SINP MSU OGSA/Globus evaluation for data intensive applications • Based on the experience with OGSA/GT 3 evaluation in 2003 -2004 (T. Chen et al. “OGSA Globus Tolkit Evaluation Activity at CERN”, in Proc. of ACAT 03, NIMA 534 (2004) 80) • Release of the Globus Toolkit 4 is currently scheduled for April 29, 2005 www-unix. globus. org/toolkit/docs/development/4. 0 -drafts/GT 4 Facts • Therefore: testing/evaluation of other OGSA/WS systems potentially interesting for LCG/EGEE НИВЦ МГУ 05. 2005 В. А. Ильин, В. В. Кореньков, А. А. Солдатов CERN-INTAS meeting, 14 March 2005, CERN 35

Enabling Grids for E-scienc. E Testing the OMII basic functionality (KIAM RAS+JINR): § Applications must be pre-installed on the (Job Service) server; an execution of programs prepared on client side is impossible. § No such core services like RB, IS, RC § Management of (grid) accounts is not well scalable, not well suitable for management of large dynamic VOs § Clients must be installed for each user separately (e. g. , not under root) § Failed to deploy a new custom service into the OMII container § Report was submitted to JRA 1 and OMII Support The OMII 1. 1. 1 Job service was found to be robust in a test with 20 concurrent clients The maximal job submission rate ~ 6 jobs/min no bulk batch mode for job submission --> problem for submitting large number of jobs The Data Service was found to work stable with up to 5 concurrent clients and a file size of up to 10 MB (no tests beyond this limits yet). § § НИВЦ МГУ 05. 2005 В. А. Ильин, В. В. Кореньков, А. А. Солдатов CERN-INTAS meeting, 14 March 2005, CERN 36

New Deployment – new CIC/ROC Enabling Grids for E-scienc. E YAIM Release(s) Update Release Notes Update User Guides GIS Release Notes Installation Guides Every Month EIS User Guides Every 3 months on fixed dates ! Certification is run daily Release Client Release 11 Every Month Deploy Major Releases (Mandatory) ROCs Re-Certify Deploy Client Releases (User Space) НИВЦ МГУ 05. 2005 GIS Deploy Service Releases (Optional) CICs CIC at own pace RCs В. А. Ильин, В. В. Кореньков, А. А. Солдатов 37

g. Lite Enabling Grids for E-scienc. E • • Differences – Unit and functional testing already performed by JRA 1 – Releases have to be synchronized between JRA 1, SA 1 based on NA 4’s priorities New Sequence – Certification Testbed (CERN) § Installation/config tests § Rerun functional tests (to validate configuration) § Synthetic stress tests – Preproduction Service § Sites • Krakow, FZK, IN 2 P 3, CNAF, Padua, Bari, NIKHEF, SNIC, Protvino-IHEP, UOM, LIP, PIC, RAL • sites test installation and configuration § Applications test by using their production software and give feedback on reliability and functionality • Status – Documentation of process is in draft state – Certification Testbed § g. Lite pre-release installed – Preproduction Service § Sites are installing current LCG 2 release as a platform for the g. Lite components – Porting of tests and collection of existing tests (JRA 1, NA 4, ARDA) started НИВЦ МГУ 05. 2005 В. А. Ильин, В. В. Кореньков, А. А. Солдатов 38