Grid Computing and the Nordu Grid Balzs Knya

Grid Computing and the Nordu. Grid Balázs Kónya LDC NETinfo-meeting 28 May 2002

What is the Grid? ● ● ● Grid is a technology to share and seamlessly access computing resources of the world The future infrastructure of computing and data management a kind of distributed computing Middleware World Wide Web access to information World Wide Grid access to computing capacity and. . . 28/05/2002 balazs. konya@quark. lu. se 1

Grid: The beginning ● ● Ian Foster, Karl Kesselman, July, 1998: Blueprint for a new Computing Infrastructure The Computational Grid <-> Power Grid analogy was suggested Early Testbeds in the USA connected supercomputing centers The birth of the Globus toolkit 28/05/2002 balazs. konya@quark. lu. se 1

Grid: Today ● ● The media is full of Grid announcements Grid is expected to become the NEXT “big thing” after the WEB Governments and Research Foundations rush to support Grid projects all over the word ● Europe seems to lag behind in the race Huge commercial interests: ● IBM ● ● ● Platform Computing ● 28/05/2002 Grid-enable the company’s entire product portfolio Close cooperation with the open source Globus development community and Global Grid Forum standards Recently announced the first commercial Grid software, the Platform Globus balazs. konya@quark. lu. se 1

what is behind? ● ● ● Powerful PCs are everywhere Clusters are commodity Network & Storage & Computing exponentials: ● ● Networking speed grows faster than computing power Even data storage outperforms the CPUs 28/05/2002 taken from Scientific American, Jan 2001 balazs. konya@quark. lu. se 1

the future. . . ● ● ● Cohen Communication Group: ● Grid computing will be the driving force behind the 150% annual internet traffic expansion in 2005 ● in contrast to the 60% predicted growth rate accounted mainly for video streaming and video file transfer forecasted by Mc. Kinsey - JP Morgan IBM: ● Grid is the next utility in the line of the water, drainage, gas and electricity systems ● people will pay their “computing bills” Ian Foster: People used to overestimate the short term impact of change but underestimate the long-term effect 28/05/2002 balazs. konya@quark. lu. se 1

Grid & Supercomputers ● The present day Supercomputers are the PC clusters ● Grid will provide a uniform access to all the resources ● The Supercomputing centers will be the power plants of the Grid Cray T 3 E Sun E 10000 28/05/2002 balazs. konya@quark. lu. se PC cluster The Grid 1

Grid projects ● USA ● Tera. Grid ($53 million) ● ● ● ● $53 million from the NSF 13. 6 teraflops of Linux clusters 450 terabytes of data storage 4 sites 40 Gbits/sec (later 50 -80) network connections Globus based Grid toolkits Visualisation environment Nasa Information Power. Grid (IPG) ● Earth System Grid Asia ● Asia-Pacific Grid (Ap. Grid) ● Japanese Grid initiatives ● Thai. Grid ● Korean. Grid ● ● 28/05/2002 balazs. konya@quark. lu. se 1

Grid projects II ● Europe ● ● ● ● European Data Grid (EDG) (9. 8 M Euro) UK Grid. PP (26 M Pounds) UK E-science project (98 M Pounds) INFN Grid, Italy 10 M Euro in 2002 Finland’s Grid research budget (connected to Data. Grid) 0. 5 M Euro /year EU 6 th Framework Programme: Grid activities expect 40 MEuro/year for 5 years, plus 30 MEuro/year from Information Society Programme. Cross Grid projects in order to link together Grid projects for more projects see: www. gridcomputing. com 28/05/2002 balazs. konya@quark. lu. se 1

Grid in Scandinavia (history) • PDC: participation in the early Globus and Condor tests – development of tools • March 2000 Lund workshop: first discussion on using Grid for basic research in particle physics • During 2000: Data. Grid and Nordu. Grid projects were formed. 28/05/2002 balazs. konya@quark. lu. se 1

Grid in Scandinavia (Data. Grid) Elementary Particle Physics LU & the EU Data. Grid Project: ● (WP 6) Testbed: through the Nordu. Grid project ● (WP 8) HEP Applications: through the Nordu. Grid project more info on EDG: www. eu-datagrid. org 28/05/2002 balazs. konya@quark. lu. se 1

Grid in Scandinavia: The Nordu. Grid Project Nordic Testbed for Wide area Computing and Data Handling www. nordugrid. org The pioneering Grid project in Scandinavia 28/05/2002 balazs. konya@quark. lu. se 1

Nordu. Grid: objectives 28/05/2002 ● Introduce the Grid to Scandinavia ● Pursue basic research on Grid Computing ● Develop Middleware Solutions ● Create a Grid infrastructure in Nordic countries ● Survey current Grid technologies ● Operate a functional Testbed ● Expose the infrastructure to end -users of different scientific communities balazs. konya@quark. lu. se 1

Nordu. Grid: participants 28/05/2002 balazs. konya@quark. lu. se 1

Nordu. Grid: project details ● ● Start: May 2001, end: October 2002 The project is financed via the Nordunet 2 programme. Budget 2. 2 MDKK. Coordinator J. Renner Hansen, NBI. The project was initiated by the high energy physics community ● ● LHC data challenge of Cern* (several Petabytes / year) Operates as part of the EU Data. Grid Testbed Physicists are among the driving forces of the World Wide Grid developement A heighly acknowledged member of the Grid community * ”Cern, where the web was born” 28/05/2002 balazs. konya@quark. lu. se 1

Nordu. Grid: resources ● Network: Nordunet network ● Hardware: dedicated test clusters at 5 sites (3 -5 machines), separate test machines at 2 sites: 7 Grid-connected sites ● Manpower: 5 postdoc researcher (3 of them financed by Nordu. Grid) ● The Nordic supercomputing centers are about to put their resources onto the Grid ● NSC clusters from Linköping ● Nordic Grid Consortium 28/05/2002 balazs. konya@quark. lu. se 1

Nordu. Grid: organization ● ● Steering panel for management Technical working group: ● ● To carry out the project’s activities Fully distributed research project: Researchers are scattered all over Scandinavia (A. Wäänänen Copenhagen, O. Smirnova & B. Kónya Lund, M. Ellert Uppsala, A. Konstantinov Oslo) ● ● Project is succesfully run via the web, mailing lists & phone conferences Bi-monthly technical meetings Nordu. Grid Workshops ● ● 28/05/2002 March 2001 Lund, November 2001 Oslo 22 -23 May 2002 Helsinki: First Presentation of the Nordu. Grid Toolkit balazs. konya@quark. lu. se 1

Nordu. Grid: activities ● The fabric of the Nordu. Grid was laid down by June 2001 ● ● Setting up the dedicated clusters, machines Nordu. Grid Authentication System was put into operation in May 2001 ● Nordu. Grid Certificate Authority in Copenhagen ● ● ● Issues certificates for Nordic Grid users Recognized by major Grid projects The first middleware was deployed and the sites were Grid-enabled by July 2001 ● ● 28/05/2002 Early version (1. 1. 4) of Globus Part of Datagrid Testbed 0 balazs. konya@quark. lu. se 1

Nordu. Grid: activities cont. ● Active participation in Globus version 2 alpha testing (August-October) ● ● ● Installation and detailed investigation of the Grid. FTP service over the Nordu. Grid resources (October) ● ● ● Nordu. Grid was among the first sites successfully running Globus version 2 Globus v. 2 beta was officially released in November http: //arxiv. org/abs/cs. DC/0205023 The Globus project showed interest in our results Further Grid services were put into operation (November-December): ● ● 28/05/2002 Nordu. Grid User Management System (Virtual Organization) Nordu. Grid Information System Grid Data Mirroring Package (GDMP) Data replication catalog balazs. konya@quark. lu. se 1

Nordu. Grid: activities cont. ● ● Deployment & evaluation of the first (Testbed 1) release of the EDG Middleware (December-January) Design and implementation of the Nordu. Grid Middleware Architecture ● ● ● information system, gridmanager, userinterface, broker, xrsl language http: //arxiv. org/abs/cs. DC/0205021 March 28, 2002: first REAL job is submitted from Lund to Oslo and successfully returned all the expected output May 23, a production level Grid System was demonstrated in Helsinki at the Nordu. Grid Workshop The Atlas Data Challenge 1 will use the Nordu. Grid Testbed ● 28/05/2002 Petabytes of data (107 events) will be generated, stored analyzed in a distributed way balazs. konya@quark. lu. se 1

Nordu. Grid: Summary ● ● The underlying structure of a Trans-Nordic Computational Grid is set up by the Nordu. Grid project: ● all the presently available Grid-services are enabled ● the Testbed is ready for production ● the Nordu. Grid Testbed is about to expand. . . Nordu. Grid carries out a pioneering role in building a Grid infrastructure in Scandinavia: ● ● ● Pursues generic research on Grid Deploys & operates the Nordu. Grid Continuously evaluates the rapidly changing Grid technologies Actively contributes to the Grid middleware development Strong ties with the Grid research community: ● ● 28/05/2002 Contribute to the Testbed of EU Data. Grid Nordu. Grid is acknowledged by and has a good working relation to the “industry standard” Globus project. balazs. konya@quark. lu. se 1

Future. . . ● Strategic importance of Grid-competence has been emphasized by Research Council ● In a few years, it is foreseen that a Nordic Grid-Centre will be created: ● ● The Nordic Centre will be a so-called Tier-1 Centre with a capacity of about 1700 x (Dual 1 GHz Intel PIII) plus fileservers and tape robots. Financial scope 150 MSEK (16 MEuro). The Centre will serve all interested fields of science. N Meanwhile ● ● 28/05/2002 Nordu. Grid will be continued: funding frame 0. 5 MEuro/year (4. 5 MSEK), personnel and dedicated hardware. Nordu. Grid continues to operate and extend its Testbed balazs. konya@quark. lu. se 1

Nordu. Grid in action a brief demo of a Grid session: ● ● ● job submission resource brokering Grid status monitoring job status information result retrieval (gridftp) 28/05/2002 balazs. konya@quark. lu. se 1

you are welcome ● if you happen to have a cluster that you want to put on the Grid ● if you are interested in Grid research and want to solve some of the outstanding questions ● if you have an application which is embarrassingly parallel and “cries” for the Grid ● or you simple would like to check out what the Grid can do for you Nordu. Grid is planning to organize Grid Tutorials and demonstrations 28/05/2002 balazs. konya@quark. lu. se 1