European Grid Infrastructure EGIIn SPIRE The European Grid

European Grid Infrastructure EGI-In. SPIRE The European Grid Infrastructure collaboration Gergely Sipos EGI. eu Technical Outreach Manager User Community Support Team gergely. sipos@egi. eu EGI-In. SPIRE RI-261323 EGI Champions briefing webinar 20/Mar/2013 www. egi. eu

Outline • EGI overview – Infrastructure, communities, projects, . . . • EGI services for research communities – Individual researchers Research infrastructures Break • EGI services for research communities – Details • Homework – Next steps for Our Champions EGI-In. SPIRE RI-261323 2 www. egi. eu

European Grid Infrastructure • European – Over 35 countries • Grid – Secure sharing of IT resources • Infrastructure – – Computers (clusters) Data Applications …. and beyond!! EGI-In. SPIRE RI-261323 3 www. egi. eu

Key resource providers in EGI: National Grid Infrastructures Metric EGI-In. SPIRE RI-261323 Value (March 2013) Sites 315 Nb. of CPU cores 372, 612 Disk (PB) 180 PB Tape (PB) 167 PB 4 www. egi. eu

A multi-disciplinary e-infrastructure http: //www. egi. eu/community/ Virtual Research Community VO Virtual Organisations VO Research communities Virtual Research Community VO User Sites EGI-In. SPIRE RI-261323 User Sites User • • • w. LCG - High Energy Physics We. NMR – Structural biology LSGC – Life sciences (HMRC – Hydro-Meteorology) (CLARIN – Humanities) Most active VO groups: • • High Energy Physics Astronomy & Astrophysics Life Sciences Earth Sciences Computational Chemistry Fusion … Members User Current VRCs: ~22. 000 users in ~230 Registered VOs http: //operations-portal. egi. eu/vo 5 www. egi. eu

EGI, EGI. eu, EGI-In. SPIRE • EGI – A collaboration. An ecosystem. – Operates the infrastructure – and does other things too! (see later) – Aim: Support the digital European Research Area through a pan-European research infrastructure based on services federed from the NGIs • EGI. eu – A Dutch fundation owned by the NGIs – To coordinate the work of EGI (operations, technology, user support, policy, community & communications, administration) – Sustainable, small organisation • EGI-In. SPIRE – An FP 7 project – Supports EGI. eu and EGI to transition to sustainable operational model – May 2010 – April 2014 EGI-In. SPIRE RI-261323 6 www. egi. eu

What is an NGI? http: //www. egi. eu/about/ngis/ • The country's single point of contact for government, research communities and resource centres as regards ICT services for escience. • Manage the computing resources and support services they provide to the European Grid Infrastructure. • Has named contact for: – International liaison (NILs) – User support – Infrastructure Operation EGI-In. SPIRE RI-261323 7 www. egi. eu

EGI-In. SPIRE Project Integrated Sustainable Pan-European Infrastructure for Researchers in Europe A 4 year project with € 25 M EC contribution – Project cost € 72 M – Total Effort ~€ 330 M – Effort: 9261 PMs (192 FTE) Project Partners (50) EGI. eu (coordinator) 38 NGIs, 2 EIROs Asia Pacific (9 partners) EGI-In. SPIRE RI-261323 8 www. egi. eu

EGI-In. SPIRE EGI. eu National Grid Infrastructures (NGIs) European Intergovernmental Research Organisations (EIROs) EGI-In. SPIRE RI-261323 Software + services Mo. Us, SLAs Technologies: • EMI, • IGE, • EDGI, . . . Specialised services: • SCI-BUS, • SHIWA, • MAPPER, • . . . VRCs and VOs: • w. LCG, • We. NMR, Infrastructures: • Scala. Life • CHAIN • HMRC, • EUDAT. . . Projects Legal entities EGI contributors 9 www. egi. eu

EGI’s strategic focus areas Virtual Research Environments • Integration & operation of scientific environments Operational Infrastructure • Operate a European wide grid infrastructure • Build a federated cloud infrastructure EGI-In. SPIRE RI-261323 Community & Coordination • Governance • Communication (inreach, outreach) 10 www. egi. eu

EGI’s strategic focus areas Consultance Gadgets Virtual Research Environments. VTs App. DB NILs Community • Integration & operation of scientific environments Webinars Requirements. CRMNewsletter & Platforms Outreach UCB Coordination Training Case studies Pilots UMDmiddleware Case studies • Governance Press • Communication Task forces (inreach, outreach) Mo. U • Operate a European wide grid infrastructure OMB Policy • Build a federated cloud infrastructure Core Cloud Accounting Marketplace Design Operational Capacity. Infrastructure TCB Monitoring EGI-In. SPIRE RI-261323 Registry 11 www. egi. eu

EGI service offerings 1. Ready-to-use scientific applications 2. Grid & cloud sites 3. Application porting / develeloper tools 4. Collab. & communic. services 5. Operation tools EGI-In. SPIRE RI-261323 Individual researchers Research groups . . . . Research 12 infrastructures www. egi. eu

1. Ready-to-use scientific applications • Registered in EGI App. DB: http: //appdb. egi. eu – Browse by scientific discipline – Browse by Virtual Organisation Applications • Application access varies: – Public access – Via portal (username/pwd) – Register in VO (X. 509 cert) • Interact! – Follow – Rate – Comment – Contact EGI-In. SPIRE RI-261323 13 www. egi. eu

2/1. Grid sites You may not need site-level access to use scientific applications on EGI. But if you do need then. . . Domain specific applications Domain specific workflows Domain specific databases Grid middleware: UMD Unified Middleware Distribution (g. Lite, ARC, Unicore, d. Cache, SAGA, Desktop Grid) Linux OS Grid sites at ~350 sites • Obtain an X. 509 certificate: – http: //www. igtf. net/ • Join one of the existing VOs: – http: //operations-portal. egi. eu/vo – 200 -220 Virtual Organisations • 8 science discipline specific VO groups • 1 VO group for other disciplines • 1 group for Multi-disciplinary VOs OR • Setup a new VO: – http: //operationsportal. egi. eu/vo/registration. Welcome – Request to support the VO is sent to site managers Get help from NGI User Support Teams EGI-In. SPIRE RI-261323 14 www. egi. eu

EGI Grid use case example 1: workload management with UMD User environment Create job definition Submit job (batch executable + <20 MB inputs) Information System Broker service query Login to your VO (With X 509 cert) Retrieve status & (small) output files ti a is rg EG irt V I on an O al u VO Management Service (VOMS) EGI-In. SPIRE RI-261323 Retrieve output Job status publish state Submit job Logging Site X of YOUR VO Computing service Job status Logging and bookkeeping service job job Storage Service Read/write files Your files 15 www. egi. eu

Implementation: Auger VO Central services for the VO the are in Czech Serving Pierre Auger Collaboration Republic (by CESNET) User environment Create job definition Submit job (batch executable + <20 MB inputs) Information System Broker service query Login to your VO (With X 509 cert) Retrieve status & (small) output files Processing software is I G distributed to E n o consortium n o i members at is an rg l. O Retrieve output Job status u A r ge VO Management Service (VOMS) EGI-In. SPIRE RI-261323 Submit job Logging ua rt Vi Copied from Colorado (USA), Malargüe (Arg) to publish state grid storage Lyon (Fr) (CCIN 2 P 3) Site of Auger. VO CPU intensive Computing service simulations done at various sites and results copied back Job status to Lyon job job Logging and bookkeeping service Storage Service Read/write files Cosmic ray files 16 www. egi. eu

EGI grid use case example 2: data & meta-data management with UMD User environment Information System Query Login to your VO (With X 509 cert) ti a is on Register file Lookup file Upload file Download file File content n ga r EG irt V I O al u VO Management Service (VOMS) EGI-In. SPIRE RI-261323 publish state Site X of YOUR VO Computing service File Catalog and/or Metadata catalog Metadata Storage service Your files 17 www. egi. eu

Implementation: g. Library http: //glibrary. ct. infn. it De Roberto, an Italian writer of the XIX/XX century, born in Naples, but spending his life in Catania (4) Browse (1) Digitalise 3 TByte in total TIFF (100 MB/A 3 page) PDF (40 -400 MB/work) (3) Catalogue + Meta-data (2) Store EGI-In. SPIRE RI-261323 18 www. egi. eu

2/2 Cloud sites: EGI Federated Cloud Domain specific services hosted on EGI as VM Images EGI Fed. Cloud interfaces Local hypervisors OS OS Cloud resources at ~10 EGI sites EGI-In. SPIRE RI-261323 • Any service, not only “batch processing” and “file based data management” • • • OCCI: VM Image management BDII: Information system X 509: Authentication VOMS: Authorization APEL: Accounting • • • Open. Stack Open. Nebula Emotive. Cloud (Spain) Okeanos (Open. Stack impl. in GR) WNo. De. S (Italy) OS 19 www. egi. eu

EGI Federated Cloud roadmap http: //go. egi. eu/cloud • Sept 2011 – March 2013: Federated Cloud Task Force – Identify open source technologies (Open. Stack, Open. Nebula, . . . ) and open standard (OCCI, CDMI, GLUE 2, . . . ) – Deploy a testbed – Write blueprint for resource centres – Identify issues from non-technical areas (policy, dissemination) • August 2012 – March 2013: Pilot use cases – Support early adopters – Collect and feedback requirements – Establish user-facing services • April 2013 (EGI Community Forum): : Open the platform – Routine usage by users and providers – Demo and discussions at EGI Community Forum 2013 • Including early adopter use cases EGI-In. SPIRE RI-261323 20 www. egi. eu

3. Application porting / developer tools To create custom environments for scientists. . . • EGI Applications Database stores – Ready-to-use solutions on EGI – Reusable frameworks – Grouped in various categories (see below) • Categories (numbers in February 2013): – 70 developer tool/framework/toolkit – 46 science gateway & gateway framework – 26 workflow system & workflow • http: //appdb. egi. eu Very often only this part is unique Domain specific services hosted on EGI Service access Science gateways (Portals) Service integration Domain specific services hosted on EGI Workflow systems Science domain specific services Scientific Off-the- algorithms, applications Scientific shelf and workflows Data bases and Scientific applications custom repositories and algorithms Scientific databases VM images EGI Grid EGI-In. SPIRE RI-261323 EGI Fed. Cloud 21 www. egi. eu

4/1. Communication services To facilitate information exchange & knowledge integration in the community. . . What we provide: • Creation of content/materials – Case studies, news stories, newsletters, articles, prints, designs, blog posts, videos, leaflets, social network content, etc. • Events – EGI annual meetings – F 2 f trainings and Webinars – Training Marketplace: http: //training. egi. eu • Outreach – International Science Grid This Week, HPCwire, HPCinthecloud and Datanami, etc. – Coordinated outreach (to ESFRIs) via the network of ‘NGI International Liaisons’ EGI-In. SPIRE RI-261323 In the pipeline: • EGI Community Forum – Machester, 8 -12 April: http: //cf 2013. egi. eu – Including several training sessions • EGI Webinar Programme – DIRAC services, 22 March – https: //wiki. egi. eu/wiki/EGI_Webinar_Programme • European Geosciences Union – Vienna, 8 -12 April: http: //www. egu 2013. eu – Exhibition booth – Presentation and PICO in ESSI 2. 8 session about EGI Federated Cloud • Leaflets – EGI Service offerings 22 www. egi. eu

4/2. Collaboration services • Virtual Team projects – Short (up to 6 month) project with multiple NGIs involved – Setup through NGI International Liaisons and EGI. eu – https: //wiki. egi. eu/wiki/Virtual_Team_Projects • Example VT projects that support Research Communities: – NGI - ELIXIR ESFRI collaboration (Started in October) – Technology study for CTA ESFRI (Started in January) – Towards a Chemistry, Molecular & Materials Science and Technology Virtual Research Community (Started in February) – Science Gateway Primer (Primer document in final editing) • Setup a Virtual Team project for your community? – What would be the scope? Which countries should be involved? EGI-In. SPIRE RI-261323 23 www. egi. eu

5. Operation Tools Services to operate a multi-site, multi-national Research Infrastructure. . . Research infrastructure Service EGI’s services: 1. Service registry (GOCDB) Scientific services – Repository to maintain topology and other type of basic information about distributed services 2. Accounting infrastructure Service – To gather, process and visualise usage of distributed services Service Registry Accounting Monitoring Heldesk Community portals 3. Service Availability Monitor (SAM) – To gather, process and visualise availability and reliability information about distributed services 4. Service discovery capability (GLUE and LDAP) Local NGIs Infrastructure services – To store detailed and dynamic information about services. To discover services by features. 5. Helpdesk (GGUS, x. GUS) – To troubleshoot problems or provide guidance about products and services. EGI-In. SPIRE RI-261323 24 www. egi. eu

Take home messages • EGI is based on a federated model – NGIs - national resource and service providers • Established grid, emerging cloud infrastructure • EGI is not just computing and storage – – Scientific applications Application developer tools & frameworks Infrastructure operation tools Collaboration and communication services EGI enables customised Virtual Research Environments EGI-In. SPIRE RI-261323 25 www. egi. eu

Homework: What can our Champions do? Promote EGI within your scientific domain 1. Ready-to-use applications: – Check the applications of your domain. Is there any key application missing? 2. Virtual Organisations: – Check the VOs that exist in EGI in your scientific domain. What services and processes do they provide to new users? Are these sufficient for a newbie to start? 3. Web gadgets: – EGI App. DB, Training Marketplace, etc. can be easily embedded into any website as ‘web gadgets’. Customise EGI gadgets for your community at http: //go. egi. eu/gadgets 4. Events: – Is the local NGI aware of your community’s upcoming event? Will they send a representative? Get in touch with the NGI International Liaison: http: //www. egi. eu/about/ngis/NILs. html 5. Research Infrastructures: – Which research infrastructures (e. g. ESFRI) exist in your scientific discipline? What is their status? Which EGI services would these benefit from at this stage? EGI-In. SPIRE RI-261323 26 www. egi. eu

EGI-In. SPIRE Questions EGI-In. SPIRE RI-261323 27 www. egi. eu

EGI-In. SPIRE Backup slides EGI-In. SPIRE RI-261323 28 www. egi. eu

VO/community, user statistics (March 2012 – March 2013) VO activity level (March 2013) High activity: Medium activity: Low activity: Very low activity: More than 1 CPU-Year/Week More than 1 CPU-Month/Week More than 1 CPU-Day/Week Less than 1 CPU-Day/Week Communities’ share of CPU consumption 25 17 12 183 High-Energy Physics Astronomy and Astrophysics Life Sciences Remaining disciplines 93. 78% 2. 78 % 1. 31% 2. 13% We need you to increase this! EGI-In. SPIRE RI-261323 29 www. egi. eu

Web Gadgets: Embed EGI collaboration services into Websites! • The simplest way to reuse EGI services: embed EGI gadgets into your website • Benefits – Customisability – Reusability – Compatibility • Existing gadgets: – – Application Database Training Marketplace Requirement Tracker Grid Relational DBs Catalogue • Use and develop gadgets & share them through EGI: www. egi. eu/user-support/gadgets EGI-In. SPIRE RI-261323 30 www. egi. eu

EGI Infrastructure: The platform model EGI Cloud Infrastructure Platform (Fed. Cloud) Virtual Research Communities Community Platform VM VM VM EGI Collaboration platform Collaboration Platform (App. DB, Training Marketplace, Doc. DB, . . . ) Community Platform (UMD) w. LCG Community Platform EGI Core Infrastructure Platform (Operation tools) EGI resources (cores & storage) EGI-In. SPIRE RI-261323 31 www. egi. eu

The current community platform: Unified Middleware Distribution (UMD) • External technology providers: EMI (ARC, g. Lite, UNICORE, d. CACHE), IGE (Globus), EDGI (desktop grids), SAGA, … • UMD capabilities captured in UMD roadmap (last version 2012 May: https: //documents. egi. eu/document/612) By EGI user Externalcommunities Technologyand Providers other stakeholders Requirements Core infrastructure platform Software Operations Criteria Definition Criteria Verification Staged Rollout Production Deployed Software Helpdesk unit Provisioning Infrastructure EGI-In. SPIRE RI-261323 32 www. egi. eu

Some of the EGI Fed. Cloud pilot use cases • Structural biology – We-NMR project: Gromacs training environments • Ecology – Bio. Vel project: Remote hosting of Open. Modeller service • Linguistics – CLARIN project: Scalable ‘British National Corpus’ service (BNCWeb) • Software development – SCI-BUS project: Simulated environments for portal testing • Space science – ASTRA-GAIA project: Data integration with scalable workflows Join as pilot use case, or as provider of high level service on top of the EGI Cloud! http: //go. egi. eu/cloud EGI-In. SPIRE RI-261323 33 www. egi. eu