Accomplishments of the project from the end user

Accomplishments of the project from the end user point of view General Perspective Federico. Carminati@cern. ch EC Review – 01/03/2002 – F. Carminati – Accomplishments of the project from the end user point of view– n° 1

Application objectives for year 1 u Define use cases for the applications u Define application requirements u Deploy realistic applications on the Test. Bed 1 u Evaluate Test. Bed 1 EC Review – 01/03/2002 – F. Carminati – Accomplishments of the project from the end user point of view– n° 2

multi-level trigger filter out background reduce data volume Hz leve l 1 (40 TB/ sec) s peci 7 5 al h K leve ardw l 2 - Hz (7 a 5 5 KH embedde GB/sec) re d pr z( leve l o B/se cessor s c) PCs 5 G 100 3 (100 Hz M B/se data c) r e c offli o ne a rding & naly sis les. robertson@cern. ch online system 40 M

The LHC Detectors ATLAS CMS ~6 -8 Peta. Bytes / year ~108 events/year ~103 batch and interactive users LHCb EC Review – 01/03/2002 – F. Carminati – Accomplishments of the project from the end user point of view– n° 4

CERN’s Network in the World Europe: 267 institutes, 4603 users Elsewhere: 208 institutes, 1632 users EC Review – 01/03/2002 – F. Carminati – Accomplishments of the project from the end user point of view– n° 5

HEP & Distributed computing u The investment for LHC computing is massive n 1. 25 GB/s in HI mode n ~5 PB/y of tape n ~1. 5 PB of disk n ~1800 k. SI 95/exp (~70, 000 PC 2000) n ~ order of 60 MSFr of hardware s u Without media + personpower + infrastructure and networking Politically, technically and sociologically it cannot be concentrated in a single location n It is unlikely that countries will make deploy massive computing at CERN n Competence is naturally distributed n Cannot ask to people to travel to CERN so often EC Review – 01/03/2002 – F. Carminati – Accomplishments of the project from the end user point of view– n° 6

The Monarc Model Lyon RAL BNL Tier 3 Universit y 1 Tier 3 Universit y 2 Tier 2 centre 20 k. SI 95 20 TB disk 622 MB/s Robot 622 MB/s Tier 3 Universit y. N Tier 2 centre 20 k. SI 95 20 TB disk Robot Tier 1 centre 200 k. SI 95 300 TB disk Robot 1500 MB/s CERN Tier 0+1 centre 800 k. SI 95 500 TB disk Robot EC Review – 01/03/2002 – F. Carminati – Accomplishments of the project from the end user point of view– n° 7

Why we need the GRID? u Every physicist should have equal access to data and resources u The system will be extremely complex n n Number of sites and components in each site Different tasks performed in parallel: simulation, reconstruction, scheduled and unscheduled analysis u We need transparent access to dynamic resources u Bad news is that the basic tools are missing n u Distributed resource management, file and object namespace and authentication n Local resource management of large clusters n Data replication and caching Good news is that we are not alone n All the above issues are central to the new developments going on in the US and Europe under the collective name of GRID EC Review – 01/03/2002 – F. Carminati – Accomplishments of the project from the end user point of view– n° 8

The Grid Vision Researchers perform their activities regardless geographical location, interact with colleagues, share and access data The GRID: networked data processing centres and ”middleware” software as the “glue” of resources. Scientific instruments and experiments provide huge amount of data EC Review – 01/03/2002 – F. Carminati – Accomplishments of the project from the end user point of view– n° 9

Biomedical Applications Genomics, post-genomics, and proteomics Explore strategies that facilitate the sharing of genomic databases and test grid-aware algorithms for comparative genomics Medical images analysis Process the huge amount of data produced by digital imagers in hospitals. EC Review – 01/03/2002 – F. Carminati – Accomplishments of the project from the end user point of view– n° 10

Grid added value for biomedical applications u u Data mining on genomics databases (exponential growth). Indexing of medical databases (Tb/hospital/year). Collaborative framework for large scale experiments (e. g. epidemiological studies). Parallel processing for n Databases analysis n Complex 3 D modelling EC Review – 01/03/2002 – F. Carminati – Accomplishments of the project from the end user point of view– n° 11

Earth Observations ESA missions: • about 100 Gbytes of data per day (ERS 1/2) • 500 Gbytes, for the next ENVISAT mission (2002). Data. Grid contribute to EO: • enhance the ability to access high level products • allow reprocessing of large historical archives • improve Earth science complex applications (data fusion, data mining, modelling …) Source: L. Fusco, June 2001 EC Review – 01/03/2002 – F. Carminati – Accomplishments of the project from the end user point of view– n° 12

Achievements u A wide-ranging dialogue has been launched within the applications on the opportunities opened by the GRID technology n n n u Each HEP experiment is a world-wide distributed community of some 1000 -2000 researchers Dialogue and coordination at several levels has been necessary to reach a common understanding Understand how to do things differently and not more of the same To improve their understanding of the issue, applications have asked to have a GLOBUS only Test. Bed deployed immediately n Applications produced a set of requirements for Test. Bed 0 in 1 Q 2001 n Testbed 0 has been in operation since 2 Q 2001 n A very large amount of unfunded effort, particularly from WP 8, has made this possible EC Review – 01/03/2002 – F. Carminati – Accomplishments of the project from the end user point of view– n° 13

Achievements u u Realistic applications using GLOBUS authentication have been deployed on Test. Bed 0 Applications have expressed their requirements in June 2001 n n n u u For this it has been necessary to develop detailed “long term use cases” A very large exercise leveraging essentially unfunded effort (some 30 -40 person-month) This has left few months to the developers to produce the middleware Funded WP 8 effort has participated to the deployment and configuration of the Test. Bed Funded WP 8 effort has performed a thorough validation of the Test. Bed with “generic” HEP applications developed by them for this purpose n Resource Broker n Replica catalogue functionality EC Review – 01/03/2002 – F. Carminati – Accomplishments of the project from the end user point of view– n° 14

Achievements u Detailed Test. Bed evaluation plans have been elaborated and applied n n Few hours after the official opening of the Test. Bed on December 9 a standard physics simulation job has been run Although TB 1 had not reached stability much middleware functionality was demonstrated by the applications in a short period of time EC Review – 01/03/2002 – F. Carminati – Accomplishments of the project from the end user point of view– n° 15

Achievements u Mechanisms to provide feedback to the developers have been put in place n n n u Priority list constantly updated by WP 8 and discussed at weekly WP manager meetings Feedback on the release plan Detailed user requirements for crucial components (e. g. Storage Element) in preparation Large unfunded participation from experiments n Probably as much as 200 person-month in the first year EC Review – 01/03/2002 – F. Carminati – Accomplishments of the project from the end user point of view– n° 16

Issues and actions u Configuration and management of the VO’s on the Test. Bed has been unexpectedly difficult n n u Procedures should be worked out to automate it Assignment and escalation of problems is complicated n u The responsibility is shared, some of our needs were not expressed too clearly and were not given enough attention It has worked well thanks to the heroic efforts of all people involved, but it will have to be better formalised Need tight coordination between site-managers, Test. Bed and applications Coexistence of pseudo-production and development environment n We have done it for years on a local environments n On a distributed ones is more difficult and we are learning Deployment plans to all Test. Bed sites are still unclear n This will be clarified in the next minor releases EC Review – 01/03/2002 – F. Carminati – Accomplishments of the project from the end user point of view– n° 17

What we want from a GRID u This is the result of our experience on TB 0 & TB 1 Specific application layer VO common application layer GRID architecture GLOBUS team ALICE ATLAS CMS LHCb LHC Other apps High level GRID middleware Basic Services OS & Net services EC Review – 01/03/2002 – F. Carminati – Accomplishments of the project from the end user point of view– n° 18

Common use cases It bebetter easiertotodefine arrive at Orwill even If we manage Specific application layer VO use cases & requirements ALICE ATLAS LHCb LHC Other apps Common core useuse cases case MW 1 GLOBUS team CMS MW 2 MW 3 MW 4 Middle. Ware MW 5 Bag of Services (GLOBUS) OS & Net services EC Review – 01/03/2002 – F. Carminati – Accomplishments of the project from the end user point of view– n° 19

Plan for the next year u u Continue exploitation of TB 1 n Provide continuous feedback to developers n Deploy larger and more realistic applications on TB 1 n Use the TB 1 for “data challenges” and “production challenges” n Use more sites as TB 1 expands Refine the user requirements and long term views n We should be able to express them in a more uniform way n A common set of requirements / use cases to define common solutions n This will also facilitate the relation with other GRID MW projects s n n Collaboration with Data. Tag WP 4 has MW interoperability as specific goal A Requirement Technical Assessment Group (RTAG) has been launched in the context of the LHC GRID Computing Project with this mandate This has been possible because of the work of the Data. GRID project and is heavily based on WP 8's findings to date EC Review – 01/03/2002 – F. Carminati – Accomplishments of the project from the end user point of view– n° 20

Summary u Applications have been able to deploy and demonstrate large applications on the Test. Bed n u u The side effect is that the pressure is high from the users for n Support n Stable environment n Functionality n Documentation n Wide deployment These are healthy signs n u The LHC experiments are participating with enthusiasm to the project Continued positive reaction to the feedback from the users will maintain their high level of enthusiasm and participation The evolution of TB 1 leading to TB 2 will be a crucial test of the project’s ability to reach its final goals EC Review – 01/03/2002 – F. Carminati – Accomplishments of the project from the end user point of view– n° 21