Enabling Grids for Escienc E The EGEE Production

Enabling Grids for E-scienc. E The EGEE Production Grid Dr. Ian Bird EGEE Grid Operations & Management Leader IT Department, CERN www. eu-egee. org EGEE-II INFSO-RI-031688 EGEE and g. Lite are registered trademarks

EGEE Enabling Grids for E-scienc. E • Flagship grid infrastructure project co-funded by the European Commission • Now in 2 nd phase with 91 partners in 32 countries Objectives • • • Large-scale, production-quality grid infrastructure for e-Science Attracting new resources and users from industry as well as science Maintain and further improve g. Lite Grid middleware EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 2

Outline Enabling Grids for E-scienc. E • EGEE infrastructure & services – – – How we got to this point Overview of services Status Middleware Training etc. • Applications • EGEE Project Activities • Management, • Dissemination , • etc. • Training • 12% • 5% • Application support • 16% • Service • 54% – Some key successes • Interoperation/interoperability – … and related projects • Middleware Development • 13% • EGEE and standards … • Open issues • What next? EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 3

Evolution of production grid Enabling Grids for E-scienc. E -Starts from LCG - Shared production infrastructure - Extended production service to other applications - Growth from 40 to 190 sites Middleware & test-beds for an operational grid Continued expansion of resources and applications communities Globus Condor 2001 EGEE-II INFSO-RI-031688 Deploying results of EDG to provide 1 st production service for LHC 2002 2004 2006 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 4

Applications Enabling Grids for E-scienc. E • Many applications from a growing number of domains – Astrophysics – Computational Chemistry – Earth Sciences – Financial Simulation – Fusion – Geophysics – High Energy Physics – Life Sciences – Multimedia – Material Sciences – … ~ 200 Virtual Organisations Applications list: https: //edms. cern. ch/file/722132/3/EGEE-II-DNA 4. 2. 1 -722132 -v 2. 5 -1. pdf EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 5

The EGEE Infrastructure Enabling Grids for E-scienc. E Support Structures & Processes Training infrastructure EGEE-II INFSO-RI-031688 Training activities Ian Bird - OGF/EGEE User Forum - May 9 th 2007 6

Growth Enabling Grids for E-scienc. E ROC CERN France De/CH Italy UK/I CE NE SEE SWE Russia A-P Total EGEE-II INFSO-RI-031688 Partner - Partner Do. W actual 1800 3548 1252 2550 1852 2695 2280 3539 2010 4527 1163 1622 1860 2473 1289 2552 898 1535 445 527 801 841 15650 26409 Total % non partner 5943 2700 3364 3628 7720 1875 3031 2568 1593 583 1632 34637 40% 6% 20% 2% 41% 13% 18% 1% 4% 10% 48% 24% Ian Bird - OGF/EGEE User Forum - May 9 th 2007 7

CPU, countries, sites Enabling Grids for E-scienc. E • Russia; 583 • SWE; 1593 A-P; 1632 • • CPU / ROC CERN; 4 • Countries / • ROC • A-P; 8 • CERN; 5943 • France; 1 • De/CH; 2 • Italy; 1 • SEE; 2568 • Russia; 2 • UK/I; 2 • SWE; 2 • France; 2700 • CE; 7 • NE; 3031 • SEE; 8 • NE; 8 • De/CH; 3364 • CE; 1875 • A-P; • 20 Sites / • CERN; ROC 12 • France; 10 • Russia; 15 • Italy; 3628 • De/CH; 14 • SWE; 15 • UK/I; 7720 35000 CPU 45 countries (31 partner countries) 237 sites (131 partner sites) EGEE-II INFSO-RI-031688 • Italy; 37 • SEE; 38 • UK/I; 25 • NE; 27 • CE; 24 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 8

Workload Enabling Grids for E-scienc. E • 3000000 • No. jobs / month - all 98000 jobs/day • 2500000 • 2000000 • LHC • 1500000 • Non-LHC • 1000000 • OPS • 500000 -0 7 7 • м • ф ев ар -0 -0 7 нв • я • н • д ек оя -0 6 • о ен • с • No. jobs / month - exc. LHC + Ops • biomed • compchem • egeode • egrid • esr • fusion • geant 4 ев -0 7 ар -0 7 • planck • Other VOs • м • ф -0 7 нв • я -0 6 ек • д -0 6 оя • н -0 6 кт • о -0 6 ен • с 6 -0 вг • а 06 лю • и 06 ню • и ай -0 6 • magic • м пр -0 • а EGEE-II INFSO-RI-031688 кт -0 6 6 -0 вг л 06 ю • а • 450000 • 400000 • 350000 • 300000 • 250000 • 200000 • 150000 • 100000 • 50000 • 0 6 13000 jobs/day • и 06 ню • и -0 6 ай • м • а пр -0 6 • 0 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 9

CPU time delivered Enabling Grids for E-scienc. E • Normalised CPU hours - all 14000 CPU-month/month • LHC • Non-LHC 7 ар -0 • м 7 ев • ф • я -0 -0 7 нв ек • д оя • н -0 6 • Normalized CPU hours - exc. LHC + Ops • biomed • compchem • 2500000 • egeode • 2000000 • egrid • esr • 1500000 • fusion • 1000000 • OPS ар ев -0 -0 7 7 • Other VOs • ф -0 7 нв • я -0 6 ек • д -0 6 оя • н -0 6 кт • о -0 6 • с ен 6 -0 вг • а ю • и ню • и ай • м пр - л 06 • planck 06 • 0 -0 6 • magic 06 • 500000 • а EGEE-II INFSO-RI-031688 • geant 4 • м 3600 CPU-month ~ 1/3 of total • о ен • с • а • 3000000 кт -0 6 6 вг -0 л 06 ю • и 06 ню • и ай -0 6 • OPS • м • а пр - 06 • 10000000 • 9000000 • 8000000 • 7000000 • 6000000 • 5000000 • 4000000 • 3000000 • 2000000 • 1000000 • 0 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 10

Overall load Enabling Grids for E-scienc. E • • Cumulative no. jobs • 2, 50 E+07 • 2, 00 E+07 – 56000 per day sustained average – Peak of 98000 – Non-LHC 13500 /day • 1, 50 E+07 • LHC • 1, 00 E+07 • non-LHC • OPS • 5, 00 E+06 ар -0 7 7 § Level of total in EGEE in 2005 • м • ф ев -0 7 -0 нв -0 • я • н • д ек оя -0 6 6 6 -0 кт • о ен -0 6 6 • с • а вг -0 6 -0 • и ю л ю н- 06 6 • и ай -0 • м • а пр -0 6 • 0, 00 E+00 • • Cumulative norm. CPU hours • 8, 00 E+07 • 7, 00 E+07 • 6, 00 E+07 • 5, 00 E+07 • 4, 00 E+07 • LHC • 3, 00 E+07 • Non-LHC • 2, 00 E+07 • OPS • 1, 00 E+07 8400 CPU-years delivered in 1 year 7 – ~1/3 of total available sustained over the year – Peak of 50% of available in Feb ’ 07 – ~1/3 of total was non-LHC in Dec ‘ 06 • м ар -0 7 -0 ев • ф -0 7 нв • я -0 6 • д ек -0 6 • н оя -0 6 кт • о -0 6 • с ен 6 • а вг -0 06 л- • и ю 06 н- • и ю -0 6 ай • м • а пр - 06 • 0, 00 E+00 EGEE-II INFSO-RI-031688 19. 6 million jobs run in 1 st year of EGEE-II Ian Bird - OGF/EGEE User Forum - May 9 th 2007 11

Grid Middleware Enabling Grids for E-scienc. E Applications Higher-Level Grid Services Workload Management Replica Management Visualization Workflow Grid Economies. . . Foundation Grid Middleware Security model and infrastructure Computing (CE) and Storage Elements (SE) Accounting Information and Monitoring EGEE-II INFSO-RI-031688 • Higher-Level Grid Services – Additional functionality • Foundation Grid Middleware – Robustness – Coexistence – Interoperability Ian Bird - OGF/EGEE User Forum - May 9 th 2007 12

g. Lite Grid Middleware Services Enabling Grids for E-scienc. E Access CLI API Security Information & Monitoring Authorization Information & Monitoring Auditing Authentication Data Management Application Monitoring Workload Management Metadata Catalog File & Replica Catalog Accounting Job Provenance Package Manager Storage Element Data Movement Site Proxy Computing Element Workload Management Overview paper http: //doc. cern. ch//archive/electronic/egee/tr/egee-tr-2006 -001. pdf EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 13

Middleware and Certification Enabling Grids for E-scienc. E • • The goal is to produce a middleware distribution that can be deployed widely Certification testing: – Installation and configuration – Component (service) functionality – System testing (trying to emulate real workloads and stress testing) EGEE-II INFSO-RI-031688 • Test-beds • • Virtual test-beds for individual testers ( ~5 ) Dynamically allocated test nodes ( > 50 nodes) Central certification test-bed Distributed test-beds for specific functions Ian Bird - OGF/EGEE User Forum - May 9 th 2007 14

Pre-production service Enabling Grids for E-scienc. E • Pre-production service is now ~ 27 sites in 16 countries • Provides access to some 3000 CPU – Some sites allow access to their full production batch systems for scale tests • Sites install and test different configurations and sets of services • Services may be initially demonstrated in this environment • Before further development • New VO-s: adapt their applications & gain experience • (e. g. DILIGENT) EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 15

Grid Management Structure Enabling Grids for E-scienc. E • Operations Coordination Centre – Management, oversight, coordination • Regional operations Centres • Grid User Support (GGUS) – Core support infrastructure – Coordination, management of user support • EGEE Network Operations Centre (ENOC) – Coordination with NRENs & GEANT 2 EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 16

Grid Operations Enabling Grids for E-scienc. E • Fully distributed – key are the Regional Operations Centres – Many of the ROCs are themselves distributed organizations – Grid Operator on Duty § Weekly rotation of teams § Critical activity in maintaining usability and stability of sites § Important tools • • • Site Availability Monitoring and Testing(SAM) Information system monitoring GGUS system for trouble ticket management § Portal for operations : https: //cic. gridops. org • Significant work on operations procedures – Evolved throughout EGEE and EGEE-II – Contribute to establishment of regional grid infrastructures through related projects – well beyond Europe now EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 17

User Support Enabling Grids for E-scienc. E • GGUS – now well established – Use continues to grow – Most ROCs provide dedicated effort to manage the process – similar to operator on duty teams – Setting up user support advisory groups to steer the priorities • GGUS tool used for all support activities No. Tickets Processed Operations Network User All – Interlinks many local ticketing systems EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 18

Policy & Security Enabling Grids for E-scienc. E • Joint Security Policy Group (JSPG) – Produces and maintains security policy and procedures § for EGEE, OSG, NDGF, WLCG, and other EU Grid infrastructures – Achieved common policy between EGEE and OSG (for interoperation) – New Grid Site Operations Policy & Updated top-level Security Policy – Grid User AUP accepted by e. IRG as good approach – Current work §New policy addressing User-level Accounting (data privacy issues) § New policy on VO and Grid service responsibilities • Operational Security Coordination Team (OSCT) focuses on: – Incident Response & improvement APGrid. PMA EUGrid. PMA TAGPMA – Security Monitoring – Best practice for system managers – Pan-regional security coordination • Grid Security Vulnerability Group The Americas Grid PMA –New group analyzing potential vulnerabilities EGEE-II INFSO-RI-031688 European Grid PMA Asia. Pacific Grid PMA Ian Bird - OGF/EGEE User Forum - May 9 th 2007 19

Grid Monitoring Enabling Grids for E-scienc. E • Becoming a critical activity to achieve reliability and stability System Management Fabric management Best Practices Security ……. • “ … improving system management practices, • Provide site manager input to requirements on grid monitoring and management tools • Propose existing tools to the grid monitoring working group • Produce a Grid Site Fabric Management cook-book • Identify training needs EGEE-II INFSO-RI-031688 Grid Services Grid sensors Transport Repositories Views ……. • “… To help improve the reliability of the grid infrastructure …” • “ … provide stakeholders with views of the infrastructure allowing them to understand the current and historical status of the service …” System Analysis Application monitoring …… • “ … to gain understanding of application failures in the grid environment and to provide an application view of the state of the infrastructure …” Ian Bird - OGF/EGEE User Forum - May 9 th 2007 20

Monitoring Enabling Grids for E-scienc. E • Important to have standard solutions for: – Sensors – Repository schema – Interfaces EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 21

Experiment Dashboard Enabling Grids for E-scienc. E Information sources INPUT Multiple sources of information Monitoring systems (RGMA, Grid. Ice, • Increasing the reliability SAM, and ICRTMDB, • Providing both global very detailed view Mona. Alisa, BDII, Generic Grid. View…) Services Experiment specific services Can satisfy users with various roles: will be shown Experiment work • Generic. This user running his jobs load management on the Grid and data • Site administrator management • VO manager, production or analysis systems group coordinator, Jobs data transfer coordinator… instrumented to report monitoring information EGEE-II INFSO-RI-031688 OUTPUT Providing output in various formats (Web pages, xml, csv, image formats) Collect data of VO interest coming from various sources Store it in a single location Provide UI following VO requirements Analyze collected statistics Define alarm conditions in the demo Can be used by various clients VO users both users session with and applications various roles • Potentially other • Clients: • PANDA, ATLAS production • <XML, CSV, image formats> Ian Bird - OGF/EGEE User Forum - May 9 th 2007 22

Training Enabling Grids for E-scienc. E • Broad range of courses to many disciplines and clients with very different backgrounds • Close relationships with applications and infrastructure activities for provision of material and lecturers • Needs are expanding rapidly with new communities and ‘beginner’ users EGEE-II INFSO-RI-031688 • 110 events; 1600 participants Ian Bird - OGF/EGEE User Forum - May 9 th 2007 23

Infrastructure for training Enabling Grids for E-scienc. E • GILDA is an effective t-Infrastructure for EGEE and other European projects, providing resources and knowledge for training events • Besides training events, GILDA is available around the clock for grid novices, with dedicated facilities • The GILDA t-Infrastructure is currently supported by 12 sites, managed on a besteffort basis • GILDA is also available for application porting EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 24

Interoperability/interoperation Enabling Grids for E-scienc. E • Well established with Open Science Grid in U. S. – – – In production use by CMS – submits work to OSG from EGEE Weekly operations meetings attended by OSG staff Processes set up with OSG for operations and user support workflows OPS VO defined to support joint operations – for testing/monitoring use Collaboration on monitoring tools and procedures • EGEE also working with other grid projects on specific interoperability at the level of middleware: – NAREGI, Unicore, NDGF(ARC) • Effort in GIN in several areas key for EGEE • Important to have a user community/use case driving this EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 25

Worldwide Grid Infrastructures Enabling Grids for E-scienc. E • APAC • DEISA • EGEE • Naregi • NDGF • NGS • OSG • Pragma • Teragrid • G I N EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 26

Collaborating e-Infrastructures Enabling Grids for E-scienc. E TWGRID Potential for linking ~80 countries EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 27

Registered Collaborating Projects Enabling Grids for E-scienc. E 24 projects have registered as on February 2007: web page Infrastructures geographical or thematic coverage EGEE-II INFSO-RI-031688 Applications Support Actions improved services for academia, industry and the public key complementary functions Ian Bird - OGF/EGEE User Forum - May 9 th 2007 28

Applications on EGEE Enabling Grids for E-scienc. E • Multitude of applications from a growing number of domains – – – Astrophysics Computational Chemistry Earth Sciences Financial Simulation Fusion Geophysics High Energy Physics Life Sciences Multimedia Material Sciences …. . This is an exciting year for science – LHC, the largest scientific instrument ever built, comes on-line - Grids are key to the success of LHC analysis EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 29

Virtual Organizations Enabling Grids for E-scienc. E Total VOs: 204 Registered VOs: 116 Median sites per VO: 3 Total Users: 5034 Affected People: 10200 Median members per VO: 18 EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 30

Active VOs Enabling Grids for E-scienc. E • Number of “active” VOs growing with time. • Turnover not shown: not same VOs every week! EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 31

Reported Applications Enabling Grids for E-scienc. E • Disciplines: 10 • Sub-disciplines: 36 • See growth and diversification of applications. • Reported apps. only PM 3 PM 11 Astronomy & Astrophysics 2 8 Computational Chemistry 6 27 16 16 Fusion 2 3 High-Energy Physics 9 11 23 39 4 14 62 118 Earth Science Life Sciences Others Total EGEE-II INFSO-RI-031688 Condensed Matter Physics Comp. Fluid Dynamics Computer Science/Tools Civil Protection Ian Bird - OGF/EGEE User Forum - May 9 th 2007 32

High Energy Physics Enabling Grids for E-scienc. E • 9000000 • LHC Experiment workloads • 8000000 • Normalized CPU – k. SI 2 k. hours • 7000000 • 6000000 • 5000000 • alice • 4000000 • atlas • 3000000 • cms • 2000000 • lhcb • 1000000 EGEE-II INFSO-RI-031688 7 7 ар -0 -0 • м ев • ф -0 7 нв • я -0 6 • д ек -0 6 • н оя -0 6 кт • о -0 6 ен 6 • с -0 вг • а л 06 ю 06 • и ню -0 6 • и ай • м • а пр - 06 • 0 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 33

User Analysis with Ganga Enabling Grids for E-scienc. E • Used ATLAS and LHCb experiments, • developed with the contribution of EGEE NA 4 • ~ 550 different users, ~100 users weekly Usage monitoring started end 2006 • ~60% Atlas • ~25% LHCb • ~15% others • Easter EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 34

CMS analysis Enabling Grids for E-scienc. E • CRAB Jobs @ FNAL (OSG) • Users on the grid: • - April 2007 statistics • CMS users submitting jobs to Grids via CRAB • (developed by CMS) • CRAB Jobs @ CERN (EGEE) • Over 1, 000 job/day Efficiency over 90% EGEE-II INFSO-RI-031688 IT/PSS Group Meeting 35

ALICE Grid Access Service Enabling Grids for E-scienc. E • ALICE Grid Access (commands executed) • Slope changes because of • optimised access (less command executed • to interact with data management) EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 36

High Energy Physics Enabling Grids for E-scienc. E • Data management: – Demonstrated data transfers at nominal rates: 1. 6 GB/s through FTS – 1 GB/s with real (simulated) workloads – 2 large experiments transferred >1 PB/month in summer 2006 • Workload management – CMS – computing service challenge achieved 50 k jobs/day – CMS aim this year for 100 k jobs/day; ATLAS for 60 k • Reliability and availability – Significant effort to ensure Tier 1 sites meet Mo. U commitments – using site and service monitoring • Grid is now the primary source of computing resources for LCG EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 37

Biomedical applications on different layers Enabling Grids for E-scienc. E 12 applications ported on the EGEE grid in areas of Medical Data management, Imaging, Bioinformatics and Drug Discovery Infrastructure level Applications EGEE-II INFSO-RI-031688 High-level interfaces Generic portals Application specific interface Specific biomedical services Medical Data Management Data-intensive workflow management Middleware Resources Communication layer Ian Bird - OGF/EGEE User Forum - May 9 th 2007 38

WISDOM Enabling Grids for E-scienc. E • WISDOM (http: //wisdom. healthgrid. org/) – Developing new drugs for neglected and emerging diseases with a particular focus on malaria. – Reduced R&D costs for neglected diseases – Accelerated R&D for emerging diseases • Three large calculations: – – – WISDOM-I (Summer 2005) Avian Flu (Spring 2006) WISDOM-II (Autumn 2006) • WISDOM calculations used Flex. X from Bio. Solve. IT in addition to Autodock. EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 39

Docking Results Enabling Grids for E-scienc. E Targets Compounds CPUyears Duration (wk) Max. CPUs Size of Results (TB) 1 M 80 6 1700 1 WISDOM-I (Q 3’ 05) PBD Avian Flu (Q 2’ 06) H 5 N 1 300 k 105 6 1700 0. 750 WISDOM-II (Q 4’ 06) GST DHFR Tubulin 125 M 420 8 5000 2 EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 40

Confirming in vitro the results obtained in silico Enabling Grids for E-scienc. E LPC Clermont-Ferrand: Biomedical grid SCAI Fraunhofer: Knowledge extraction, Chemoinformatics CEA, Acamba project: Biological targets, Chemogenomics Univ. Modena: Biological targets, Molecular Dynamics Health. Grid: Biomedical grid, Dissemination ITB CNR: Bioinformatics, Molecular modelling Univ. Los Andes: Biological targets, Malaria biology Chonnam nat. univ. : In vitro testing New Academica Sinica: Grid user interface Biological targets In vitro testing Univ. Pretoria: Bioinformatics, Malaria biology I Avian flu data challenge: in the selection of 2250 compounds out of initial 308585 compounds, an enrichment factor of 111 was observed. Experimental trial confirms 7 actives out of 123 tested gave “potential hits”. Data challenges on malaria: the 25 most promising compounds out of 500. 000 are now being tested in vitro at Chonnam National University EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 41

Earthsystem Sciences Enabling Grids for E-scienc. E • Goal: learn about the past, the present, and possible futures of the earth system • Community: internationally and interdisciplinary distributed but strongly interconnected • Method: Analysing, comparing and processing data • Input: data from observations and/or other modelling studies EGEE-II INFSO-RI-031688 Typical workflow Scenario data Model Data Observation Data Distributed Climate Data 1 Find & Select Data description 2 Collect & Prepare Analysis Dataset 3 Analyse Result Dataset 4 Visualize Ian Bird - OGF/EGEE User Forum - May 9 th 2007 42

An example workflow: “qflux” Enabling Grids for E-scienc. E Location Various data centers & portals Find & Select relevant & available datasets 1 Specific Temperature Wind speed humidity Distributed Climate Datavolume Several PB ~3, 1 TB (300 -500 files) Institutional storage & computing facilities Collect & Prepare a temporal and spatial subset of the data 2 Analysis Dataset local facilities Personal Computer Analyse the integrated, transport 3 of humidity between selected levels Result Dataset 4 Visualize selected ~10, 3 GB (28 files) ~76 MB ~66 KB result EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 43

Potential use of grid technology Enabling Grids for E-scienc. E Current issues • Search & select – Different portals with different authentications and data descriptions • Collect & prepare – Different access mechanisms of the different providers – Pre-processing requires sufficient local facilities • Analyse – Existing tools and already processed data are available locally and miss proper description • • Central unique authentication to a common catalogue with standardized metadata • Shared resources with standardized access hiding proprietary access mechanisms • Commonly defined tool description • Log processing steps and automatically republish processed data Visualize – Detached from the remaining workflow EGEE-II INFSO-RI-031688 • Integrate basic visualization (first peep) into the workflow Ian Bird - OGF/EGEE User Forum - May 9 th 2007 44

Presentations in User Forum on applications in EGEE and Related Projects Enabling Grids for E-scienc. E • Specific applications – Atmosphere and Ocean Models – Earthquake modelling – Fusion – Range of biomedical applications – Computational Chemistry – Astrophysics – Space applications – HEP (LHC and non-LHC) EGEE-II INFSO-RI-031688 • Applications in Related Projects – – – – – EUMEDgrid Baltic. Grid EELA EUChina. Grid EUIndia. Grid G-Eclipse Sym. Grid DILIGENT Be. In. Grid Ian Bird - OGF/EGEE User Forum - May 9 th 2007 45

Sustainability: Beyond EGEE-II Enabling Grids for E-scienc. E • Need to prepare permanent, common Grid infrastructure • Ensure the long-term sustainability of the European e-infrastructure independent of short project funding cycles • Coordinate the integration and interaction between National Grid Infrastructures (NGIs) • Operate the European level of the production Grid infrastructure for a wide range of scientific disciplines to link NGIs EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 46

EGEE and standards Enabling Grids for E-scienc. E • EGEE and other grid infrastructures need to co-exist and interoperate – At many levels – campus, local, national, regional, international • A large production system has inertia – cannot change quickly – Introducing new software and standards is slow, need to maintain backward compatibility – Cannot frequently change the infrastructure • g. Lite choice of standard adoption is based on interoperability needs and impact assessment on the infrastructure • Operational experience essential – Leads to best practices which in turn should drive standardization efforts – Actively pushing convergence for most pressing needs • The EGI/NGI era will rely on interoperability and coexistence – Appropriate and workable standards will be essential – Care not to fix standards too soon – this is not mature technology See also: http: //egee-na 5. web. cern. ch/egee-na 5/NA 5 Standardisation. html EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 47

Examples Enabling Grids for E-scienc. E EGEE has worked on real community implementations of standards • Example 1: SRM (Storage Resource Manager) – SRM v 2. 2 defined > 1 year ago to satisfy LCG requirements – Dedicated effort to reach today with beta versions of real interoperating implementations (5) – and this was vital for LCG § Needed many iterations on details of the specifications § Interoperation test suites and real use case testing was essential – Also required changes to all clients – the APIs were completely changed from SRM v 1. 1 • Example 2: GLUE (information system schema) – Today this is the accumulated knowledge of experience in real large scale production of EGEE, OSG, ARC over 5 years – The information systems are not perfect – we see scalability problems – The experience is in the schema – It can and should evolve to something better – but it must evolve – Is an OGF working group EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 48

Areas of standardization Enabling Grids for E-scienc. E Driven by the need for interoperation, co-existence, etc. EGEE is actively involved in many areas, including with OGF • Security (AAA) – Policy work & IETF wg on Incident Response – VOMS and proxy certificates – Interoperability with Shibboleth • Data Management – SRM, FTS • Accounting & monitoring – Common usage record, schema, sensors • Job Management – Gatekeeper interfaces • Information system – Common schema • Important for coexistence/interoperability: – areas close to fabric (accounting, monitoring, sensors, etc. ) need to be common EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 49

Open Issues Enabling Grids for E-scienc. E General issues: • Making grid tools easily usable by non-experts • Failures not easy to understand – Lack of consistent or thorough error reporting • Lack of consistent administrative interfaces makes them hard to manage EGEE issues: • Portability of current g. Lite distribution prevents wider acceptance and coexistence EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 50

Summary Enabling Grids for E-scienc. E • EGEE is operating the world’s largest multi-disciplinary grid for science – In continuous use for production work at significant scale • Can bring experience at operating at this scale to the community and the standardization process – But we have to prioritize carefully • There is a long way to go to improve: – Usability, manageability, reliability, security – Interoperability and coexistence • It is time to move towards ensuring the long term sustainability of these infrastructures – Will rely on carefully selected common solutions for key services and processes EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 51

EGEE’ 07 Conference Enabling Grids for E-scienc. E Building Bridges… • Between Science and business • Between users and infrastructures • Between countries • Between scientific disciplines • Between projects http: //www. eu-egee. org/egee 07 • EGEE-II INFSO-RI-031688

OGF and EGEE THANK OUR EVENT COORDINATING PARTNERS and SPONSORS © 2006 Open Grid Forum

OGF 20/EGEE User Forum Coordinating Partners © 2006 Open Grid Forum Ian Bird - OGF/EGEE User Forum - May 9 th 2007

OGF 20/EGEE User Forum Event Sponsors Premier Standard Technische Universitat Berlin GRIDtoday © 2006 Open Grid Forum Media

User Forum agenda Enabling Grids for E-scienc. E Wednesday Astro Workshop Opening Plenary Grids Mean Business g. Lite GIN OMII-Europe Poster and Demonstrations Data Management Thursday Experience with Users in the wider grid Application Domains community Workflow Poster and Demonstrations Data Management Grid Monitoring & Accounting Friday Experience with Users in wider grid Application Domains community Workflow Interactivity & portals User/VO community support Closing Plenary EGEE-II INFSO-RI-031688 Ian Bird - OGF/EGEE User Forum - May 9 th 2007 56