Introduction to Grids and Grid applications Gergely Sipos

Introduction to Grids and Grid applications Gergely Sipos MTA SZTAKI www. lpds. sztaki. hu sipos@sztaki. hu

What is Grid? ● ● A Grid is a collection of computers, storages, special devices, services that can dynamically join and leave the Grid They are heterogeneous in every aspect They are geographically distributed and connected by a wide-area network They can be accessed ondemand by a set of users Internet

Why use a Grid? • A user has a complex problem that requires many services/resources in order to • • Reduce computation time Access databases Share equipments Collaborate with other users Internet

Typical Grid application areas • Demand for computation capacity • High-performance computing (HPC) • Shorten the execution time of a single parallel application • Reguirement: parallel computing • High-throughput computing (HTC) • Execute as many similar jobs as possible during a given period • Requirement: exploit spare CPU cycles • Demand for large data storage • With the involvement of physically distributed data bases • Demand for collaborative work • Integrate several users’ knowledge in order to solve a complex problem

Example: Large Hidron Collider, CERN, Genf Mont Blanc (4810 m) ATLAS Downtown Geneva ~10 -15 Peta. Bytes /year ~108 events/year ~103 batch and interactive users LHCb CMS

Example: Rolls Royce aircraft engines 1 Gb data per engine per flight ● ● Real time download to basis airport Historical data mining ● ● Evaluation, analysis ● ● Across airports Pattern search on distributed platform Preparing maintenance crew

Other examples • In silico drug discovery – molecule simulations to find drug candidates • Earth science, space research – sharing, analyzing satellite pictures • Archeology – digital archives, virtual simulations • Weather prediction – data integration, model selection, simulations, evaluation • Engineering – simulation of buildings, vehicles in extreme conditions • . . .

Why to use grids? ● Most of these problems were solved by supercomputers and clusters 5 -10 years ago. Now grids are used: ● ● A Grid is able to utilize spare cycles without extra investments Grids can share risk, can lower participation cost Grids can integrate resources – HW, SW, data Grids provide flexible access to resources

Grid vision Mobile Access G R I D Workstation Visualising M I D D L E W A R E Supercomputer, PC-Cluster Data-storage, Sensors, Experiments Internet, networks

Problems to solve ● Standard access to resources ● ● ● Computers Storage resources Special devices Software Data Access policy, security Load balancing Monitoring of resources Monitoring of applications Error handling Application methodology, programming models. . .

If “The Grid” vision leads us here… … then where are we now? • • • Utility computing Cloud computing E-Infrastructure Cycle scavenging … IBM Grid HP Grid Oracle Grid …

Generic Grid modell Donating free resources Inst 1 Inst 2 Inst 4 Internet Requiring resources Inst 3

Two players of the Grid • Resource donors = D • Resource users = U • Relationship between the two characterizes the Grid: • if U ~ D generic Grid model • if U >> D utility Grid model • if U << D desktop Grid model

Generic Grid model is complex… ● Endless possible usage patterns Involved security solutions Real time information system Complex brokering, load balancing architecture Flexible programming architecture ● Simplifications were made to achieve something useful: ● ● ● Utility grids Desktop grids

Utility grids

Utility Grid model Inst 1 Donor and user User 1 Donating resources static 7/24 mode Internet Dynamic resource requirements Inst 2 Donor and user User N

Characteristics of the utility Grid model • Donors must be “professional” resource providers who provide production service (7/24 mode) Simplification • Homogeneous resources Simplification • Anybody can use the donated resources for solving her/his own applications • Asymmetric relationship between donors and users: U >> D

Utility Grid example: EGEE ● ● ● ● > 200 sites in 40 countries ~ 38 000 CPUs ~ 5 PB storage 98 k jobs/day > 200 Virtual Organizations g. Lite middleware The World’s largest multi-disciplinary Grid http: //www. eu-egee. org/

Utility Grid example: Open Science Grid 30 Virtual Organizations 105 Resources 26 Support Agencies Middleware: – Virtual Data Toolkit (VDT): collection of grid tools – – – http: //www. opensciencegrid. org/ Condor Globus VO Management Service

Utility Grid example: Nordu. Grid http: //www. nordugrid. org/ Dynamic Grid ~ 33 sites, ~1400 CPUS Production Grid – – – Applications from various scientific disciplines Sites operate 24/7 Mostly unattended by administrators Middleware: – Advanced Resource Connector (ARC)

Utility Grid example: UK National Grid Service ● http: //www. grid-support. ac. uk/ 4 core sites: Leeds, Oxford, Manchester, Rutherford ● 6 partner sites ● 3 affiliate sites ● Middleware: Globus Toolkit 2 ● Additional SW services: ● OGSA DAI ● Storage Resource Broker ● NGS Oracle DB ● P-GRADE Portal ● . . .

Virtual Organizations and Utility Grids • Grid: – Resources that host the same middleware version – People who use them • VO: – Logical subset of sites and users – Security policy – Dynamic? • Atlas VO tens of years • WISDOM data challenge few weeks Virtual Organization Grid

Virtual Organizations and Utility Grids • Grid: – Resources that host the same middleware version – People who use them • The Grid problem is to enable Virtual VO: “coordinated resource sharing – Logical of sites and solving andsubset problem in dynamic, Organization users multi-institutional. Grid virtual – Security policy organizations. ” – Dynamic? • Atlas VO tens of years • WISDOM data challenge From VO few weeks ”The Anatomy of the Grid” by Ian Foster, Carl Kesselman, Steven Tuecke

Utility Grids: Based on Service Oriented Architecture Registry Register an available service Send name & description Service Consumers Services

Architecture of Service Grids: Service Oriented Architecture Registry Request a service Send a description Service Consumers Services

Architecture of Service Grids: Service Oriented Architecture Registry Set (possibly empty) of matching services Service Consumers Services

Architecture of Service Grids: Service Oriented Architecture Registry Service Consumers Request service operation Services

Architecture of Service Grids: Service Oriented Architecture Registry Service Consumers Return result or Error Services

Architecture of Service Grids: Service Oriented Architecture Registries Server programs run on the resources • High availability is a must • Standard protocols expected • Security architecture is Services complicated Consumers • Requires expertise at every site

SOA meets the Grid Supercomputing (PVM/MPI) Clusters Network Computing (sockets) Cluster computing Web Computing (scripts) OO Computing (CORBA) Client/server High-throughput High-performance computing Condor Object Web Globus, LCG-2 Web Services Service Oriented Grids Semantic Grid Utility Grid Systems

Desktop Grids

Desktop Grid model Dynamic resource donation Company/ univ. server Donor: Company/ Univ. or private PC Application Internet Donor: Company/ univ. or private PC Work package distribution

Characteristics of the desktop Grid model • Anybody can donate resources • Heterogeneous resources, that dynamically join and leave • One or a small number of projects can use the resources Simplification • Resources run clients: Expertise only at the server Simplification • Asymmetric relationship between donors and users: U << D • Advantage: • Donating a PC is extremely easy • Setting up and maintaining a DG server is much easier than installing the server sw of utility grids

Types of Desktop Grids • Global Desktop Grid • Aim is to collect resources for grand-challenge scientific problems • Example: • BOINC (SETI@home) • Local Desktop Grid • Aim is to enable the quick and easy creation of grid for any community (company, univ. city, etc. ) to solve their own applications • Example: • SZTAKI Desktop Grid

SETI: a global desktop grid ● SETI@home ● 3. 8 M users in 226 countries ● 1200 CPU years/day ● ● ● 38 TF sustained (Japanese Earth Simulator is 32 TF sustained. Currently 30 rd on TOP 500) Highly heterogeneous: >77 different processor types Infrastructure is separated now from application: BIONC

SZTAKI Desktop Grid: a global and local DG system • Extension of BOINC • • Simplifies the creation of DG applications Simplifies the installation and maintenance of DG servers Global and local configuration Global installation: http: //www. lpds. sztaki. hu/desktopgrid/ • Mathematical problem: Search for number dimension • Installation package is available • Three steps to try and use the system: 1. Donate one PC to the Global system 2. Port application to the global system 3. Set up a DG for your own community Step 1 is easy. SZTAKI helps in steps 2 and 3

FP 7 Project - EDGe. S ● ● ● Enabing Desktop Grids for e-Science Two years, started on the 1 st of January, 2008. Integrate Desktop Grids and Utility Grids ● ● ● Including BIONC and g. Lite technologies (besides other DG tools) DG jobs UG UG jobs DG ● Integrated portal environment to develop and manage applications on DG, UG platforms Coordinator MTA SZTAKI, Hungary – www. lpds. sztaki. hu ● Watch for news at www. edges-grid. eu ●

Programming the grid

Available parallelism in grids • Utility Grids – Master-slave (parameter study) – Inter-site parallelism – Intra-site parallelism – Workflow – Combination of the above • Desktop Grids – Master-slave (parameter study)

Master-slave (parameter study) parallelism Master Work package 1 Work package 2 Work package 3 Work package N Internet

Inter-site parallelism Internet

Intra-site parallelism Internet

Workflow parallelism Internet

Combined parallelism Example: Inter-site and parameter study Internet

Goal of the course ● ● ● This is not a middleware developer course This is a user course specialised on g. Lite and related technologies. Why g. Lite? ● ● A utility grid implementation Anybody can access to execute applications Widely used, well supported, large community behind it Several potential use cases Several extra tools „around” it

Conclusion • Generic grid model is good, but hard to implement • Simplification in practice: • Utility grids • Desktop grids • Existing production installation from both types • EGEE, US OSG, Nordu. Grid, UK NGS Various patterns • BOINC, SZTAKI DG Master-slave • Course focus on • Utility grids • g. Lite middleware and related tools • Application development and usage, installation, administration

Thank you for the attention ? sipos @ sztaki. hu