Enabling Grids for Escienc E EGEE 4 Summary

Enabling Grids for E-scienc. E EGEE 4 – Summary of summaries The expurgated version Steve Fisher www. eu-egee. org INFSO-RI-508833

Demo session (I/II) Enabling Grids for E-scienc. E • Very successful session – > 16 demos from JRA 1, JRA 4, NA 3 and NA 4 § NA 4 Biomed: GPS@, Pharmacokinetics, Simri@web, SRM Dicom and Metadata, WISDOM, GROCK § NA 4 HEP: experience on data analysis, user activity on the EGEE grid § NA 4 Generic: GILDA application demos, P-Grade portal, Charon, EGRID, Earth Sciences § EGEE digital library (NA 3) § BAR (JRA 4), NPM (JRA 4), § Pro. Active g. Lite bridge (JRA 1) – A lot of visitors despite parallel sessions (SA 1, PMB, CB, . . . ) INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 2

Demo session (II/II) Enabling Grids for E-scienc. E • Lessons learnt – Usage of booths allows much more interactivity with audience – Need to have a common template for all the demos (poster, leaflet) • Perspectives – Demos for EC review in December – Demos for User Forum INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 3

NPM DT Scenario (1) Enabling Grids for E-scienc. E • Step 1: Access the NPM Diagnostic Tool. –The Diagnostic Tool can be accessed using a standard web browser, which users are individually authorised to use. –In the future, we plan to use VOMS for authorisation. –Please mail us for access! INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 4

Network Services in EGEE-II Enabling Grids for E-scienc. E • JRA 4 absorbed in SA 1 • NPM continues – This conference established the potential of the NPM work for Operations and fixed our position within SA 1 – EGEE-II work will include the hardening of e 2 emonit and the coordination of its deployment – Collaboration with middleware will also continue to provide network performance information to the WMS • BAR parks INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 5

EGEE first User Forum (I/II) Enabling Grids for E-scienc. E • • Dates: March 1 -3 2006 Location: CERN, Switzerland Target attendance: 150 participants Goals – Get a consistent understanding across the EGEE related projects of expectation, present status and possible evolution – Promote cross-application fertilisation – Prepare EGEE-II • Participation open to external projects and EGEE members • Format: 3 -day workshop – Presentations by thematic areas selected by invitation and through a call for contributions – EGEE presentations (integration of new applications, access to resources, status of middleware, . . . ) – With a lot of time for discussion • Contact: Massimo. Lamanna@cern. ch INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 6

EGEE user forum (II/II) Enabling Grids for E-scienc. E • Success of User Forum requires involvment of other project activities – User Forum is an opportunity for EGEE users to ask questions to middleware, operation and training experts – NA 4 counts on other activities (JRA 1, SA 1, NA 3, . . . ) to send teams of experts to participate to the event – Format for interaction with users will be discussed in the coming weeks • Call for application talks soon to be announced – If you have an application you wish to show, be ready to respond • User Forum will be the opportunity to collect data for the second EGEE User Survey INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 7

Preparation of EGEE review Enabling Grids for E-scienc. E • NA 4 requested to focus only on the reviewer recommendations of first review report – Have all current applications migrated to g. LIte with a very good user satisfaction § migration of several existing applications to g. Lite is achieved – Based on the experience of previous FP 5 projects, capture full requirements of future user groups, assess needs for new grid services and plan accordingly for later implementation § Requirements database on Savannah include hundreds of requirements – Clarify the true motivation of users from new application areas right from the beginning § 5 Mo. U’s are ready to be approved by PEB. A lot of experience has been acquired in the process. • Selection of demos – To highlight new g. Lite features – To Illustrate application deployment on the infrastructure INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 8

Service Metrics Enabling Grids for E-scienc. E • RB: % passing a job submission test • BDII: Query time measurement • My. Proxy: % passing a test (register a proxy, access the proxy, delete the proxy) • SRM-SE: % passing file movement and deletion tests • Catalogue: % passing register, query, delete • VOMS: % passing VOMS test • RGMA: TBD INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 9

Metric discussion Enabling Grids for E-scienc. E • We are going to publish these values • Is it acceptable for all sites? • Provide guidelines to publish correct specint and correlated benchmark • Goal: – provide understandable metrics with comparable values – Most of the metric are numbers with an associated scale – Other metrics are judgement of quality – Measure the quality of the infrastructure • Applications need to measure the quality of their application – Basic numbers can be retrieved via with application SFT • Start publish some of these metrics soon as prototype, – end of the year – Service metrics LCG requirement by December – Other metric sets to follow same timeline INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 10

Proposal for PPS Evolution Enabling Grids for E-scienc. E • PPS can hardly be defined the Pre-Production compared with the ‘production services – PPS includes only g. Lite services – PPS does not share Monitoring and Operation Procedures with the Production – PPS is far away to be a preview of what really will be in the next LCG release • How PPS should change – Improve Operation Procedures – Same Support model as in production – To be included in Monitoring Infrastructure – Include all production services really in use: Catalogs/LFC, BDII etc • How Production should change – Adding deployable g. Lite Services to existing production services INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 11

Deployment and Release Issues Straw Man I Enabling Grids for E-scienc. E • Upgrade and introduce services as soon as ready (SC and g. Lite) – Whenever upgrades are ready and tested – T 1 s experience can be used as additional deployment test – Handle these as upgrades to the last major release – Make client libs available in user space – Version tracking via the information system • With sufficient material accumulated – Cut a new release – Starting point for sites to join – Reference point for external project • Security upgrades – Upgrades + message to the site security contacts • Work continuously on tests and documentation INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 12

Deployment and Release Issues Straw Man II Enabling Grids for E-scienc. E • New services should be “integrable” with current CIC on duty – Tick list produced at the operations workshop INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 13

Deployment and Release Issues Straw Man III Enabling Grids for E-scienc. E • Integrated releases on the pre-production service – Now: § production: lcg + g. Lite components § pre-production: g. Lite only – Better: § production: lcg + g. Lite components § pre-production: lcg + current g. Lite + new g. Lite components • New name for integrated production stack needed • The winner is: –g. Lite X. Y INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 14

Overview of porting projects Enabling Grids for E-scienc. E • CERN Openlab & SPACI – Itanium port available and tested for LCG 2. 6. 0 (all nodes) • CERN/UVienna/Apple – Mac. OS X port available (focus on UI: WMS, …) • Grid-Ireland – WN ports available for Cent. OS 4. 1, Suse 9. 3, Red. Hat 7. 3/9 – Work in progress on Mac. OS X, Solaris, EMT 64, FC 4, AIX, IRIX • GSI (Germany) – Debian port (UI and WN? ) • IRB (Croatia) – Debian: tar fixes (UI), chroot (CE+WN), converting RPMs to DEBs (ongoing); Free. BSD: tar (UI) • HPC 2 N Umea (Sweden) – Porting g. Lite to Ubuntu (Debian) • EGRID (Italy) – Live. CD with all service nodes, UI-only relocatable installation INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 15

Issues on porting Enabling Grids for E-scienc. E • The future is to provide common infrastructure for all platform with no need to check out the code • Installation of different platforms must not be an issue • The code has to be 64 bit compatible • Interoperability is also very important • Significant interest to run Grid service in different Linux distribution non only SLC INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 16

SA 1 in EGEE-II Enabling Grids for E-scienc. E • Consolidate and build on what has been done in EGEE • Integration, certification, release preparation moved into SA 3 • Maintain hierarchical structure, but simplify ROC/CIC ROCs • Strengthen (operational and user) support for many more applications and users • Security – take operational tasks from JRA 3; set up real response teams INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 17

Organisation Enabling Grids for E-scienc. E INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 18

SA 3 Tasks Enabling Grids for E-scienc. E • • • T 1: Integration and packaging T 2: Testing and certification T 3: Support, analysis, debugging, problem resolution T 4: Interoperability T 5: Capture Requirements • Partners: CERN + – PSNC, TCD, �IMPERIAL, INFN, UKBH – UCY, GRNET, CSIC, PIC, CESGA, FZJ – Partners active in testbeds, tests, and specific interoperation tasks INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 19

Enabling Grids for E-scienc. E INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 20

TCG Overview Enabling Grids for E-scienc. E • The EGEE-II proposal defines a Technical Coordination Group (TCG): The TCG brings together the technical activities within the project in order to ensure the oversight and coordination of the technical direction of the project, and to ensure that the technical work progresses according to plan. • Basically coordinating the work of SA 1, SA 2, SA 3, NA 4, and JRA 1 – Membership from all these activities but still remain a “small” team – Additional experts will join based on the topic of discussion – Working groups will be spawn off to solve specific problems • Focus on practical short term solutions – Long term projects will be sourced out to middleware providers • The group must have executive power – Not just a discussion forum! – Decisions taken by the group must be honoured by the affected activities INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 21

Report from discussion Enabling Grids for E-scienc. E • The TCG should have control on the architecture/design team and have a solid requirement management process • The most important members of the TCG should be users, sites and developers representatives with a distillation process to avoid TCG become too big • User documentation is very important and has to be part of the certification process is to verify documentation • Define strict acceptance criteria to avoid testing things that doesn't work. • Avoid confusion if debugging and fixing is in SA 3 or in JRA 1: SA 3 just minor fixes • SA 3 and SA 1 are middleware providers only for small components, glue between services • Make sure the contributions coming from SA 3 follow the same criteria as the rest • The deployment policies? is not a short term discussion and will be done in the TCG, certain decisions will also be taken by deployment managers depending on site needs INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 22

g. Lite 1. 4 Certification Status Enabling Grids for E-scienc. E • WMS – push/pull: OK – Integration with the BDII: still waiting for the fix for the empty cache (it is not supposed to be released with 1. 4. 1, which may mean that likely the WMS will not be ready to be included in the next LCG release as proposed) – Bulk submission: not tested, test left to PPS users – Upgrade: OK – Status: Moved to PPS • • IO, Fireman (Mysql), R-GMA : – Upgrade: OK – Status: Deployable FTS : – Configuration instructions still not sufficient – It took several days to get it working – We could not get it working with dpm – Still does not work with catalogs – It works with url-copy – Upgrade: not tested – Status: Not deployable out-of-the-box. Expert support needed for installation, limited functionality, keep version 1. 3 INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 23

Deployment tools Enabling Grids for E-scienc. E • Currentl situation: YAIM – All changes concentrated in 1 file, easy to customize and extend it, appreciated by many site managers – Balance between simplicity and flexibility • Glite configuration approach is different – For each glite module, there is a corresponding packages list + XML data + 1 configuration script – 1 XML document can describe the configuration of 1 site • There is an overlap between the two tools – Discussion how to find a good compromise for site manager, integrators, developers – Everybody agree that the configuration for the sitemanager must be kept simple whenever possible – A. Di Meglio, O. Keeble, A. Forti will work for a temporary smooth transition INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 24

A digital library for EGEE Enabling Grids for E-scienc. E • E-Learning – Phase one: digital library and content management system – Based on open source repository system - Fedora – Web Services API & W 3 C XForms user interface – Standard Metadata: Dublin core, Learning Objects, etc. – On track with timeline presented at Athens – Very sucessful demonstartion on Wednesday – Next: “wiki in e-learning” developments • Collaboration meeting with DILIGENT and visit – Sharing content/metadata § All g. Lite documentation (DILIGENT -> NA 3) § Training material archive (NA 3 -> DILIGENT) – Showing the benefits of adhering to standards § Dublin core/LO etc. § Also promoting Google cross searching etc. • WS services – clients can be constructed to meet varied needs • Requests for the personalisation services already being built in. INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 25

Proposed for EGEE II Enabling Grids for E-scienc. E • How does EGEE improve documentation – Identified core? § Fluent and simple sharing for the rest of our creative channels § Segmenting audience • Both of these begun by UIG as part of its workplan agreed with the PEB – A workflow for accredited & QA’ed Documents? – Metadata & Structure? – Properly resourced Documents in Core: § UIG with staff act as Editorial Board • • § § INFSO-RI-508833 Establish policy & criteria With teeth and an executive staff With resources With multi-activity buy in and contributions Part of Requirements Part of Release schedule Staff time allocated – in all relevant activities Write this in the EGEE II technical annex 4 th EGEE conference - 28 th October 2005 26

JRA 1 management Enabling Grids for E-scienc. E • Change of JRA 1 management on November 1 st • New Activity Manager: Claudio Grandi – Previously in CMS experiment @ LHC § Grid Integration Coordinator 2000 -2004 – Started using grid in 1999 (test of data productions with Globus) – Member of EU Data. Grid (WP 8) and then EGEE (NA 4) • Many thanks to Frédéric and Erwin for the job done – I’ll rely on their help in the next months – They’ll do most of the work for the EU review! INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 27

Summary Enabling Grids for E-scienc. E by F. Hemmer, 24/10 • g. Lite releases have been produced – Tested, Documented, with Installation and Release notes – Subsystems used on § Service Challenges § Pre-Production Services § Production Service – And by other communities (e. g. DILIGENT) • g. Lite processes are in place – Closely monitored by various bodies – Hiding many technical problems to the end user • g. Lite is more than just software, it also about – Processes, Tools and Documentation – International Collaboration INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 28

External projects integration session Enabling Grids for E-scienc. E Many (new) EU projects (will) use g. Lite middleware ISSe. G EU GRID INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 29

by P. Pagano, 25/10 Enabling Grids for E-scienc. E INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 30

Enabling Grids for E-scienc. E by A. Di. Meglio, 25/10 INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 31

Job Statistics - Conclusion Enabling Grids for E-scienc. E • by G. Zaquine, 25/10 Summary of the current situation – Various tools for various purposes (statistics, monitoring, accounting). Each tool with advantages and inconvenient depending where input data come from: § Input from RBs (JRA 2 RB stats, Job Provenance stats): do not take into account jobs not submitted through RBs. About 90% RBs are collected § Input from CEs (APEL): do not take into account what is happened before CE § DGAS will offer both. About 90% sites are collected – Data Challenge and end users statistics: Each DC has to build it own statistics tool § No basic solution currently even if JRA 2 statistics helped Wisdom Biomed DC § JDL “Application Tag” will help • Next steps – Better understand job throughput distribution between jobs using RBs from other jobs submission mechanisms (direct access to the CE, Dirac…). § No basic solution currently – Common work in order to provide common tool INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 32

PPS emoticons Enabling Grids for E-scienc. E • • • In some cases the release does not reflect the proposed architecture (e. g. the pull mode, use of BDII) User Guide: Once a new server is installed and configured it is really painful to understand how to use it, even for basic tests Error Messages: Often not useful, sometimes misleading WMS Performance decays (observed with 1. 2, 1. 3, 1. 4) VO enabling and handling on the system should be made easier The upgrade procedure is officially not supported but in principle the tools are there (sometimes not working mostly due to rpm names changing) Quite a number of failures mostly due to configuration errors. SFT should make things better Log files are located in a single place. This makes debugging easier Installation Documentation: Release Notes, Installation documents and XML templates are of very high quality Support: JRA 1 very reactive and effective on both mailing lists (discuss and PPS) People in PPS starts getting used to XML and python scripts. After the first impact and it is not perceived anymore as "difficult" by default. by A. Retico, 27/10 INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 33

EAC Feedback Enabling Grids for E-scienc. E • “The Project has once again made a very impressive progress” • “Clearly visible by surpassing both quantitative and qualitative criteria” • “Constitutes an excellent basis for EGEE II” • “EGEE has been very successful in the acquisition of application in a wide spectrum of scientific disciplines” • “The EAC encourages EGEE to take a leading role in some initiatives to share its experience with the Grid community, especially in the areas of grid middleware, deployment and interoperation” INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 34

Issues Enabling Grids for E-scienc. E • Applications/Demos: Reinforce the Grid and EGEE added value in the different applications and boost the usage of the productions system Why is the Grid/EGEE needed? • Documentation and Training Material: reconfirms the need for the User Information Group (UIG), more effort is needed in quality and access (not quantity) • Better coordination between the partners within (large) activities Partners have to provide better feedback on their work contribution to the activities!!! INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 35

Issues Enabling Grids for E-scienc. E • Middleware: More and more confidence in stability of the software made availability is seen by the community, but confusion about what is the actual EGEE project – Strategic importance to converge towards a single g. Lite distribution – Goal for the end of EGEE: to have a SINGLE software distribution called g. Lite including all the components for the production service and the evolution of the production service • Testing and Education still needs manpower reinforcement – Management board is working on a solution to that – Contributions are welcome INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 36

Next Events Enabling Grids for E-scienc. E • November 7, 2005: Hearing on EGEE-II • December 6 -7, 2005: Focused Review • March 1 -3, 2006: User Forum • March 31, 2006: End of EGEE • April 1, 2006: Beginning of EGEE-II • May 2006: Final Review + EGEE-II All-Activity Meeting INFSO-RI-508833 4 th EGEE conference - 28 th October 2005 37