European Grid Infrastructure EGIIn SPIRE EGI for ELIHU

European Grid Infrastructure EGI-In. SPIRE EGI for ELI-HU Gergely Sipos EGI. eu Technical Outreach Manager User Community Support Team gergely. sipos@egi. eu EGI-In. SPIRE RI-261323 www. egi. eu

Outline • EGI overview – Infrastructure, communities, projects, . . . • EGI services for RIs – Infrastructure, collaboration, user tools, capacity • Possible next steps for EGI – ELI-HU collaboration EGI-In. SPIRE RI-261323 2 www. egi. eu

European Grid Infrastructure EGI • European – Over 35 countries • Grid – Secure sharing of IT resources • Infrastructure – – Computers Data Applications …. and beyond!! EGI-In. SPIRE RI-261323 3 www. egi. eu

EGI Resource Infrastructure Providers: National Grid Infrastructures (Snapshot: April 2012) EGI-In. SPIRE RI-261323 4 www. egi. eu

A multi-disciplinary e-infrastructure http: //www. egi. eu/community/ Virtual Research Community VO Virtual Organisations VO Research communities Virtual Research Community VO User Sites EGI-In. SPIRE RI-261323 User Sites User • • • w. LCG - High Energy Physics We. NMR – Structural biology LSGC – Life sciences HMRC – Hydro-Meteorology CLARIN - Humanities Largest VO communities: • • High Energy Physics Astronomy & Astrophysics Life Sciences Earth Sciences Computational Chemistry Fusion … Members User Current VRCs: ~25. 000 users in ~200 Registered VOs http: //operations-portal. egi. eu/vo 5 www. egi. eu

EGI’s Strategic Focus • Operational Infrastructure – Operate a European wide grid infrastructure – Build a federated cloud infrastructure • Virtual Research Environments – Integration & operation of scientific environments • Community & Coordination – Events: Community Forum, Manchester, April 8 -12 http: //cf 2013. egi. eu – Marketing, technical outreach and support • Focus on Reseach Infrastructures, such as ELI http: //go. egi. eu/EGI 2020 EGI-In. SPIRE RI-261323 6 www. egi. eu

EGI, EGI. eu, EGI-In. SPIRE • EGI – an open collaboration – To support the digital European Research Area through a pan. European research infrastructure based on services federed from the NGIs • EGI. eu – a Dutch fundation owned by the NGIs – To coordinate the work of EGI (operations, technology, user support, policy, community & communications, administration) – Sustainable, small organisation • EGI-In. SPIRE – an FP 7 project – Supports EGI. eu and EGI to transition to sustainable operational model – May 2010 – April 2014 – 24 m. Euro EC contribution, 72 m. Euro in total EGI-In. SPIRE RI-261323 7 www. egi. eu

What can EGI offer to RIs? EGI-In. SPIRE project Middleware projects: • EMI, • IGE, • EDGI, . . . Technology projects: • SCI-BUS, • SHIWA, • MAPPER, • . . . 1. Operation tools Monitoring, accounting, service registry, . . for a distributed infrastructure 2. Collaboration tools Training Marketplace, Software registry & repository, VM image marketplace, . . . 3. Application developer & user tools VRC and VO projects: • w. LCG, • We. NMR, • Scala. Life • HMRC, . . . EGI-In. SPIRE RI-261323 NGIs and EGI. eu 4. Computing, storage & hosting capacity (existing grid and cloud sites) Registered in EGI Applications Database (mostly) From resource centres (Centrally managed resource pool is under setup) . . . 8 www. egi. eu

1. Operation Tools for RIs To operate a multi-site, multi-national IT infrastructure. . . 1. Service registry (GOCDB) – Repository to maintain topology and other type of basic information about distributed services 2. Accounting infrastructure – To gather, process and visualise usage of distributed services 3. Service Availability Monitor (SAM) – To gather, process and visualise availability and reliability information about distributed services 4. Service discovery capability (GLUE and LDAP) – To store detailed and dynamic information about services. To discover services by features. 5. Helpdesk (GGUS, x. GUS) – To troubleshoot problems or provide guidance about products and services. EGI-In. SPIRE RI-261323 9 www. egi. eu

2. Collaboration Tools for RIs To exhange & integrate knowledge in a multi-national community. . . 1. Applications Database – Registry of high level software environments, software components, developers, publications 2. Training Marketplace – Registry of training events, training materials, training resources, university courses 3. Client Relationship Manager – To capture leads and needs of scientific communities EGI-In. SPIRE RI-261323 10 www. egi. eu

Applications Database Benefits: • Gives recognition to reusable scientific applications, application developer tools, portals, workflow systems, etc. • Gives recognition to application developers (people profiles) • Access through web page AND web gadget • Community features such as commenting, rating, tag-based groupings http: //appdb. egi. eu EGI-In. SPIRE RI-261323 11 www. egi. eu

Training Marketplace Benefits: • Register & share • training events, expertise, services, materials, resources, online courses, university courses • Browse and search items • Community features such as commenting, rating • Access through web page and web gadget http: //training. egi. eu EGI-In. SPIRE RI-261323 12 www. egi. eu

Client Relationship Manager • To capture leads and needs of scientific communities • EGI CRM – main capabilities (based on v. Tiger CRM): – Register projects/communities – Register personal leads – Register details about ‘Potential for EGI use’ and ‘Interest in e-infrastructures’ • EGI instance is available to NGI International Liaisons and their collaborators at http: //crm. egi. eu EGI-In. SPIRE RI-261323 13 www. egi. eu

Web Gadgets: Embed EGI collaboration services into Websites! • The simplest way to reuse EGI services: embed EGI gadgets into your website • Benefits – Customisability – Reusability – Compatibility • Existing gadgets: – – Application Database Training Marketplace Requirement Tracker Grid Relational DBs Catalogue • Use and develop gadgets & share them through EGI: www. egi. eu/user-support/gadgets EGI-In. SPIRE RI-261323 14 www. egi. eu

3. Application Developer and user tools for RIs To develop custom environments for scientists. . . • EGI Applications Database stores – Ready-to-use solutions on EGI – Reusable frameworks – Check them at http: //appdb. egi. eu • EGI provides these in partnership with projects. Some examples: – SCI-BUS: Grid/cloud portal framework – ER-flow: Workflow platform and repository – DIRAC: Application developer tool • Numbers (February 2013): – – – 460 software in total 387 scientific application 70 developer tool/framework/toolkit 46 science gateway & framework 26 workflow system & workflow EGI-In. SPIRE RI-261323 Community specific Virtual Research Environment Service access Science gateways, Portals Service integration Workflow systems Domain services Domain Off-the- specific applications Scientific shelf and specific Data workflows bases and Domain applications custom repositories and algorithms Domain VM imagesspecific databases 15 www. egi. eu

4. Grid and Cloud capacity for RIs Community specific Virtual Research Environment Science gateways, Portals Workflow systems Domain specific applications Domain specific workflows Domain specific databases Domain specific services Grid middleware: g. Lite, ARC, Unicore, Desktop Grid OS OS Grid sites at ~350 sites EGI-In. SPIRE RI-261323 EGI Cloud federation (Iaa. S) Local hypervisors OS OS OS Cloud sites at ~10 sites 16 www. egi. eu

EGI-In. SPIRE Possible next steps for EGI – ELI-HU collaboration EGI-In. SPIRE RI-261323 17 www. egi. eu

CRISP reference architecture T 0 Processing QA Calibration Policies Data Management 2 -3 Days DAQ Buffer PBs-1 Controls loa n ow D ds ta Da Trigger Farm Collaboration Recording Buffer Real-time Queue Instrument (x 10 s) Writes Data to GBs-1 Primary Storage & Archive PB Replicates Data to GBs-1 Secondary Storage Datapresentation. in This is a slide from a recent. Registers CRISP n ta i m Fro rs ste i g e Da R Could we refine this architecture to specfy Downloads Data From Data Stager the ELI-HU File IT &infrastructure? Metadata ar Se che r. D s fo in Catalog Runs Analysis on User (x 10 K) Data Policy Accounting and Billing Data Analysis Facility (100 ks Cores) Laurence. Field@cern. ch 18

Follow up at national level • Hungarian NGI: Magyar Grid Kompetencia Központ http: //mgkk. hu • • • BME ELTE KFKI-RMKI MTA SZTAKI NIIFI 3 D SOFT KFT. • EGI projects with Hungarian partners • • EGI-In. SPIRE RI-261323 EDGI - desktop grid SCI-BUS – portal framework for developers and users ER-flow – workflow platform & repository CRISP – common needs of physics ESFRIs www. egi. eu

Follow-up at European level: EGI Virtual Team projects • Virtual Team projects: – Short (up to 6 month) project with multiple NGIs involved – Setup through NGI International Liaisons and EGI. eu – https: //wiki. egi. eu/wiki/Virtual_Team_Projects • Example VT projects that support RIs: – NGI - ELIXIR ESFRI collaboration (Started in October) – Technology study for CTA ESFRI (Started in January) – Towards a Chemistry, Molecular & Materials Science and Technology Virtual Research Community (Started in February) – Science Gateway Primer (Amost over) • Setup a Virtual Team project for ELI? – With partners common to ELI – CRISP - EGI-In. SPIRE? EGI-In. SPIRE RI-261323 20 www. egi. eu

EGI-In. SPIRE Extra slides EGI-In. SPIRE RI-261323 21 www. egi. eu

EGI Infrastructure: The platform model Virtual Research Communities VM VM VM EGI Collaboration platform EGI Cloud Infrastructure Platform Community Platform (g. Lite Grid) EGI Core Infrastructure Platform Grid services EGI resources (cores & storage) EGI-In. SPIRE RI-261323 22 www. egi. eu

The current community platform: Unified Middleware Distribution (UMD) • External technology providers: EMI (ARC, g. Lite, UNICORE, d. CACHE), IGE (Globus), EDGI (desktop grids), SAGA, … • UMD capabilities captured in UMD roadmap (last version 2012 May: https: //documents. egi. eu/document/612) By EGI user Externalcommunities Technologyand Providers other stakeholders Requirements Core infrastructure platform Software Operations Criteria Definition Criteria Verification Staged Rollout Production Deployed Software Helpdesk unit Provisioning Infrastructure EGI-In. SPIRE RI-261323 23 www. egi. eu

EGI Grid use case 1: workload management with UMD User environment Create job definition Submit job (batch executable + small inputs)Broker Information System service query create Proxy (~login) e e or f in ru st ur ct a pl Retrieve status & (small) output files m or Retrieve output tf ra Job status Logging Site X of YOUR VO C VO Management Service (VOMS) EGI-In. SPIRE RI-261323 publish state Submit job Computing service Job status Logging and bookkeeping service process Storage Service Read/write files 24 www. egi. eu

EGI grid use case 2: data & meta-data management with UMD Command line or Portal Information System Query create proxy rm e e or f in u tr s a ur ct a pl Register file Lookup file Upload file Download file tfo r Site X of YOUR VO C VO Management Service (VOMS) EGI-In. SPIRE RI-261323 File content publish state Computing service Storage service File Catalog and/or Metadata catalog File references 25 www. egi. eu

EGI cloud roadmap http: //go. egi. eu/cloud • Sept 2011 – March 2013: Federated Cloud Task Force – Identify open source technologies (Open. Stack, Open. Nebula, . . . ) and open standard (OCCI, CDMI, GLUE 2, . . . ) – Deploy a testbed – Write blueprint for resource centres – Identify issues from non-technical areas (policy, dissemination) • August 2012 – March 2013: Pilot use cases – Support early adopters – Collect and feedback requirements – Establish user-facing services • April 2013 (EGI Community Forum): : Open the platform – Routine usage by users and providers EGI-In. SPIRE RI-261323 26 www. egi. eu

Some of the cloud infrastructure pilot use cases • Structural biology – We-NMR project: Gromacs training environments • Ecology – Bio. Vel project: Remote hosting of Open. Modeller service • Linguistics – CLARIN project: Scalable ‘British National Corpus’ service (BNCWeb) • Software development – SCI-BUS project: Simulated environments for portal testing • Space science – ASTRA-GAIA project: Data integration with scalable workflows Join as pilot use case, or as provider of high level service on top of the EGI Cloud! http: //go. egi. eu/cloud EGI-In. SPIRE RI-261323 27 www. egi. eu

Services of the Collaborative platform 4: Requirements tracker User Community Support Team of EGI. eu VOs VRCs projects NGIs Input channels for community requirements EGI Requirements Tracker events User Community Board Structured scientific communities Technology Coordination Board Technology providers Operations Management Board Resource centres EGI Helpdesk NGIs & projects EGI-In. SPIRE RI-261323 http: //go. egi. eu/requirements 28 www. egi. eu

EGI guiding Principles • Be an open provider – Resource providers can federate their sites into the Core Platform – Software providers can join into the Core, Cloud and Collaboration Platforms • Be a neutral provider – Collaboration Platform for the fair representation of various stakeholders and their achivements • Be a flexible provider – Cloud and Core platform allows communities to deploy their own services Community Platforms – Give communities the power to meet their own needs EGI-In. SPIRE RI-261323 29 www. egi. eu