Enabling Grids for Escienc E EGEE Tutorial for

Enabling Grids for E-scienc. E EGEE Tutorial for MAGIC Please download files from: http: //homepages. nesc. ac. uk/~mjm/MAGIC/ www. eu-egee. org INFSO-RI-508833

Tutorial goals Enabling Grids for E-scienc. E • 1. Introduce the key concepts of grid computing, whereby grid services allow resources of data and computation in different institutes to be shared and used as a "virtual computer". 2. Explain the goals of the EGEE project 3. Outline how the grid services provided by EGEE can be used, with an emphasis on gaining practical experience of commandline interfaces. 4. Describe how the MAGIC community is building with EGEE. The tutorial will benefit members of the MAGIC project who do not yet have grid or more specifically, LCG or g. Lite, experience • • Introductory talk Introductory practicals Lunch MAGICAL practical INFSO-RI-508833 “The Grid” and EGEE Tutorial for MAGIC Oct. 16 2005 2

Enabling Grids for E-scienc. E “The Grid” and EGEE Mike Mineter mjm@nesc. ac. uk www. eu-egee. org INFSO-RI-508833

Acknowledgements Enabling Grids for E-scienc. E • This talk was prepared by Mike Mineter of Ne. SC and includes slides from previous tutorials and talks delivered by: – – – Dave Berry, Richard Hopkins (National e-Science Centre) the EDG training team Ian Foster, Argonne National Laboratories Jeffrey Grethe, SDSC EGEE colleagues INFSO-RI-508833 “The Grid” and EGEE Tutorial for MAGIC Oct. 16 2005 4

Goals of this module Enabling Grids for E-scienc. E • To introduce the concepts of Grid computing assuming no previous knowledge • To enable members of the MAGIC virtual organisation to understand the context in which they are working • To introduce the practicals that follow INFSO-RI-508833 “The Grid” and EGEE Tutorial for MAGIC Oct. 16 2005 5

Contents Enabling Grids for E-scienc. E • • What and why of grids Current status of grids The EGEE project Introduction to the production middleware INFSO-RI-508833 “The Grid” and EGEE Tutorial for MAGIC Oct. 16 2005 6

The Grid Vision Enabling Grids for E-scienc. E Mobile Access G R I D Workstation M I D D L E W A R E Supercomputer, PC-Cluster Data-storage, Sensors, Experiments Visualising Internet, networks INFSO-RI-508833 “The Grid” and EGEE Tutorial for MAGIC Oct. 16 2005 7

“The Grid” vision Enabling Grids for E-scienc. E • The grid vision is of “Virtual computing” (+ information services to locate computation, storage resources) – Compare: The web: “virtual documents” (+ search engine to locate them) • pooling their resources • MOTIVATION: collaboration through dynamic configuration of resources for: – Research – Commerce – engineering, … “the knowledge economy” – Public service – health, environment, … INFSO-RI-508833 “The Grid” and EGEE Tutorial for MAGIC Oct. 16 2005 8

e-Science: the first driver of Grid computing Enabling Grids for E-scienc. E • What is e-Science? Collaborative science that is made possible by the sharing across the Internet of resources (data, instruments, computation, people’s expertise. . . ) – Often very compute intensive – Often very data intensive (both creating new data and accessing very large data collections) – data deluges from new technologies – Crosses organisational boundaries • Grid computing enables e-Science (e-Research) INFSO-RI-508833 “The Grid” and EGEE Tutorial for MAGIC Oct. 16 2005 9

Astronomy Enabling Grids for E-scienc. E No. & sizes of data sets as of mid-2002, grouped by wavelength • 12 waveband coverage of large areas of the sky • Total about 200 TB data • Doubling every 12 months • Largest catalogues near 1 B objects INFSO-RI-508833 “The Grid” and EGEE Data and images courtesy Alex Szalay, John Hopkins University EGEE Tutorial for MAGIC Oct. 16 2005 10

Earth Observation Enabling Grids for E-scienc. E ESA missions: • 100’s of Gbytes of data per day Grid contribution to EO: • Enhance the ability to access high level products • Allow reprocessing of large historical archives • Improve Earth science complex applications (data fusion, data mining, modelling …) Federico. Carminati , EU review presentation, 1 March 2002 INFSO-RI-508833 “The Grid” and EGEE Derived from: L. Fusco, June 2001 EGEE Tutorial for MAGIC Oct. 16 2005 11

Large Hadron Collider at CERN Enabling Grids for E-scienc. E • Data Challenge: – 10 Petabytes/year of data !!! – 20 million CDs each year! • Simulation, reconstruction, analysis: – LHC data handling requires computing power equivalent to ~100, 000 of today's fastest PC processors! • Operational challenges Mont Blanc (4810 m) – Reliable and scalable through project lifetime of decades INFSO-RI-508833 “The Grid” and EGEE Tutorial for MAGIC Oct. 16 2005 Downtown Geneva 12

Enabling Grids for E-scienc. E DAME: Grid based tools and Inferstructure for Aero-Engine Diagnosis and Prognosis Engine flight data London Airport Airline office New York Airport Grid • “A Significant factor in the success of the Rolls-Royce campaign to power the Boeing 7 E 7 with the Trent 1000 was the emphasis on the new aftermarket support service for the engines provided via DS&S. Boeing personnel were shown DAME as an example of the new ways of gathering and processing the large amounts of data that could be retrieved from an advanced aircraft such as the 7 E 7, and they were very impressed”, DS&S 2004 Diagnostics Centre Maintenance Centre American data center European data center XTO Companies: Rolls-Royce DS&S Cybula INFSO-RI-508833 Universities: York, Leeds, Sheffield, Oxford “The Grid” and EGEE Engine Model Case Based Reasoning EGEE Tutorial for MAGIC Oct. 16 2005 13

Political realities Enabling Grids for E-scienc. E • Grid investors speak of entering the “knowledge society” from the “industrial society” – Infrastructures that § “Unleash creativity” § Enable and require new forms of organisations (VO’s) • Lisbon strategy: Research and Innovation will be the most important factors in determining Europe’s success through the next decades • HEP is where it started, but science alone will not be seen as justifying the investment… grids must develop to underpin research, learning, … and commerce INFSO-RI-508833 “The Grid” and EGEE Tutorial for MAGIC Oct. 16 2005 14

Grid projects Enabling Grids for E-scienc. E Many Grid development efforts — all over the world • UK – OGSA-DAI, Reality. Grid, Geo. Dise, • NASA Information Power Grid Comb-e-Chem, Discovery. Net, DAME, • DOE Science Grid Astro. Grid, Grid. PP, My. Grid, GOLD, e. Diamond, Integrative Biology, … • NSF National Virtual Observatory • Netherlands – VLAM, Polder. Grid • NSF Gri. Phy. N • Germany – UNICORE, Grid proposal • DOE Particle Physics Data Grid • France – Grid funding approved • NSF Tera. Grid • Italy – INFN Grid • DOE ASCI Grid • Eire – Grid proposals • DOE Earth Systems Grid • Switzerland - Network/Grid proposal • DARPA Co. ABS Grid • Data. Grid (CERN, . . . ) • Hungary – Demo. Grid, Grid proposal • NEESGrid • Euro. Grid (Unicore) • Norway, Sweden - Nordu. Grid • Data. Tag (CERN, …) • DOH BIRN • Astrophysical Virtual Observatory • NSF i. VDGL • GRIP (Globus/Unicore) • GRIA (Industrial applications) • Grid. Lab (Cactus Toolkit) • Cross. Grid (Infrastructure Components) • EGSO (Solar Physics) INFSO-RI-508833 “The Grid” and EGEE Tutorial for MAGIC Oct. 16 2005 15

Key concepts Enabling Grids for E-scienc. E • Virtual organisation: – people collaborating in shared endeavours – pooling their resources • A grid provides “virtual computing” across administrative domains • Single sign-on – I connect to one machine – some sort of “digital credential” is passed on to any other resource I use, basis of: § Authentication: How do I identify myself to dynamically chosen resources without username/password for each resource I use? § Authorisation: what can I do? Determined by • My membership of VO • VO negotiations with resource providers INFSO-RI-508833 “The Grid” and EGEE Tutorial for MAGIC Oct. 16 2005 16

A grid Enabling Grids for E-scienc. E • • Grid middleware runs on each shared resource to provide – Data services – Computation services – Single sign-on Users join VO’s Virtual organisation negotiates with sites to agree access to resources INTERNET Distributed services (both people and middleware) enable the grid for multiple VOs INFSO-RI-508833 “The Grid” and EGEE Tutorial for MAGIC Oct. 16 2005 17

Enabling Grids for E-scienc. E Current status INFSO-RI-508833 “The Grid” and EGEE Tutorial for MAGIC Oct. 16 2005 19

Grids: still a work in progress!!! Enabling Grids for E-scienc. E • Many key concepts identified and known • Many grid projects have tested these • Major efforts now on establishing: – Standards (a slow process) and “service orientation” (Global Grid Forum, http: //www. gridforum. org/ ; IETF, OASIS, W 3 C) – Production Grids for multiple VO’s § “Production” = Reliable, sustainable, with commitments to quality of service so researchers can commit to building with it § Operational procedures and services (people, policy, . . ) § Interoperability: International grid that federates national grids – New user communities • … whilst research & development continues INFSO-RI-508833 “The Grid” and EGEE Tutorial for MAGIC Oct. 16 2005 20

EGEE – building e-infrastructure Enabling Grids for E-scienc. E EGEE is building a large-scale production grid service to: • Underpin research, technology and public service • Link with and build on national, regional and international initiatives • Foster international cooperation both in the creation and the use of the einfrastructure INFSO-RI-508833 “The Grid” and EGEE Pan-European Grid Operations, Support and training Collaboration Network infrastructure & Resource centres EGEE Tutorial for MAGIC Oct. 16 2005 24

In the first 2 years EGEE will Enabling Grids for E-scienc. E • Establish production quality sustained Grid services – 3000 users from at least 5 disciplines – integrate 50 sites into a common infrastructure – offer 5 Petabytes (1015) storage • Demonstrate a viable general process to bring other scientific communities on board Pilot Added • Proposed a second phase for 2 more years from ~April 2006 INFSO-RI-508833 “The Grid” and EGEE Tutorial for MAGIC Oct. 16 2005 25

Activities Definition Enabling Grids for E-scienc. E • Network Activities – NA 1: Project Management – NA 2: Dissemination and Outreach – NA 3: User Training and Induction – NA 4: Application Identification and Support – NA 5: Policy and International Cooperation • Service Activities – SA 1: Grid Support, Operation and Management – SA 2: Network Resource Provision • Joint Research Activities – JRA 1: Middleware Reengineering + Integration g. Lite services – JRA 2: Quality Assurance – JRA 3: Security – JRA 4: Network Services Development INFSO-RI-508833 “The Grid” and EGEE Emphasis in EGEE is on operating a production grid and supporting the end-users EGEE Tutorial for MAGIC Oct. 16 2005 26

LCG and EGEE Enabling Grids for E-scienc. E • EGEE committed to “hit the ground running” in the proposal • Current service (“LCG-2”) based on work done in LCG – Middleware components to be LCG : Large Hadron Collider Compute Grid upgraded by “g. Lite” services as they are proven • EGEE profits from the resources - no funded computing/data resources in EGEE – Provided by the VOs • LCG obtains additional production and operation efforts INFSO-RI-508833 “The Grid” and EGEE Tutorial for MAGIC Oct. 16 2005 28

Current production grid: LCG-2 Enabling Grids for E-scienc. E Application level services User interfaces Applications EU Data. Grid “Collective” services App monitoring system VDT (Condor, Globus, GLUE) User access Workload management Data management “Basic” services Information system NFS, … Scientific Linux Operating system Information schema System software File system Data transfer Security PBS, Condor, LSF, Torque… Local scheduler Hardware Computing cluster INFSO-RI-508833 Network resources “The Grid” and EGEE HPSS, CASTOR… Data storage EGEE Tutorial for MAGIC Oct. 16 2005 31

Current production m’ware: LCG-2 Enabling Grids for E-scienc. E “User interface” Input “sandbox” Replica Catalogue Information Service Data. Sets info Output “sandbox” SE & In CE i nfo s tu St a Jo b nfo Job Status “The Grid” and EGEE Publish san t“ INFSO-RI-508833 tpu Ou Logging & Book-keeping Job Query Job Submit Event Author. &Authen. I er ok Br ”+ ox ” db ox san db t“ pu Resource Broker Storage Element Computing Element EGEE Tutorial for MAGIC Oct. 16 2005 32

Enabling Grids for E-scienc. E • Now its time to begin to use the current production middleware… INFSO-RI-508833 “The Grid” and EGEE Tutorial for MAGIC Oct. 16 2005 36