Project Report Alberto Di Meglio Project Director CERN
Project Report Alberto Di Meglio, Project Director, CERN ETICS Final Review CERN, Geneva - 15 February 2008 www. eu-etics. org INFSOM-RI-026753
Introduction • Overall project report, providing an overview of: – – – Review Agenda Project Goals First Review Recommendations Major achievements Challenges Project metrics (deliverables and milestones, financial status, manpower levels) INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 2
Review Agenda Morning 09: 00 - Welcome 09: 15 – Project Report 10: 15 – WP 1 (Management) 10: 30 – WP 2 (Infrastructure) 11: 20 – WP 3 (Build System) 11: 50 – WP 4 (Test System) 12: 20 – WP 5 (Dissemination, certification and web tools) Frédéric Hemmer (CERN) Alberto Di Meglio (CERN) Miron Livny (Univ. of Wisconsin) Elisabetta Ronchieri (INFN) Eva Takacs (4 D SOFT) Marc-Elian Bégin (CERN) Afternoon 14: 00 – Grid-QCM 14: 30 - Demos Adriano Rippa (Engineering) Lorenzo Dini (CERN), Tomasz Kokoszka (CERN), Paolo Fabriani (Engineering), Marc-Elian Bégin (CERN) 15: 30 - Concluding remarks Alberto Di Meglio (CERN) 15: 35 - Reviewers Closed Session followed by Feedback INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 3
The ETICS Partners Build system, software configuration, service infrastructure, dissemination, EGEE, g. Lite, project coord. The Condor batch system, distributed testing tools, service infrastructure, NMI Software configuration, service infrastructure, dissemination Web portals and tools, quality process, dissemination, DILIGENT Test methods and metrics, unit testing tools, EBIT INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 4
Project Structure • WP 1 • PROJECT ADMINISTRATIVE AND TECHNICAL MANAGEMENT • CERN (Lead) • WP 2 • INFRASTRUCTURE • Uo. W (Lead) • CERN, INFN • WP 3 • SOFTWARE CONFIGURATION, BUILD, INTEROPERABILITY • INFN (Lead) • CERN INFSOM-RI-026753 • WP 4 • SOFTWARE TESTING • 4 DSoft (Lead) • Uo. W • WP 5 • DISSEMINATION, TRAINING, USER INTERFACES • CERN (Lead) • Engineering ETICS Final Review - Project Report - CERN, 15 February 2008 5
Project Structure Project Manager (PM) Project Management Board INFSOM-RI-026753 Project Technical Committee WP 2 WP 3 WP 4 WP 5 ETICS Final Review - Project Report - CERN, 15 February 2008 6
Project Goals To provide within a two year programme: • A professionally managed, integrated system of services and resources for running automated builds and test suites for distributed software projects • A repository of packages, test reports and interoperability information that development projects can use to develop, validate and certify their products • A forum where to discuss and promote software quality initiatives and collect common software engineering tools and processes, software configuration information and documentation. INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 7
What has ETICS delivered? • Infrastructure – Three resource sites at CERN, INFN and Uo. W with more than 200 CPUs and more than 40 platforms. Additional third-party resources from OMIIEurope, GARR (EUChina. Grid), IN 2 P 3 (EGEE/SA 2) and DILIGENT – A Service Level Agreement and documented deployment procedures • Services – A set of web services and tools for configuring, building and testing software – Web and multiplatform CLI clients – A rich repository of reports and packages with connectors for other types of repositories like APT and OMII-Europe – The ‘multi-node testing’ feature (co-scheduling) that allows automated deployment and execution of complex multi-node tests • Community Support – Build and test support for almost 20 projects and in particular EGEE (both middleware and applications), DILIGENT and OMII-Europe – A structured user support mechanism organised in weekly shifts with dedicated personnel to handle user tickets • Dissemination, Training and Certification – 6 major training events (EGEE’ 06, DILIGENT TCOM, INFN-CNAF in 2006, OGF 19, JRA 1 All-Hands, ETICS User Workshop in 2007) – The BTC Community Group in OGF, participation to several conferences – The Grid-QCM model INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 8
Architecture ETICS Portal (my. ETICS, Build/Test, Repository, Administration) Web Service Repository Service Via browser Report Build/Test Project Artefacts DB DB Metronome Execution Engine Clients WNs INFSOM-RI-026753 Metronome Client Wrapper ETICS Infrastructure ETICS Final Review - Project Report - CERN, 15 February 2008 9
The ETICS Timeline (Year 1) April 2006 ETICS Alpha programme started (first g. Lite components registered) July 2006 ETICS Beta programme started (g. Lite 3. 1 managed only with ETICS), DILIGENT using for testing, OMIIEurope project created February 2007 ETICS Service v 1. 0 Released • 1 st Review All-hands meeting (Bologna) All-hands meeting (Budapest) • Kick-off Jan 06 INFSOM-RI-026753 OGF CCG Launched Jun 06 September 2006 ETICS Pre-release programme started, first major training event in Geneva Dec 06 ETICS Final Review - Project Report - CERN, 15 February 2008 10
1 – Strong Collaborations and Repository To establish a stronger collaboration with other projects (not only EGEE and DILIGENT) and to promote the use of the ETICS services by as many projects as possible from the e-Infrastructure Unit and else (to defend the use of the SSA instrument). Especially it is proposed to cooperate closely with OMII-Europe towards the establishment of a “European scientific repository of software tools and reusable components”. As it concerns the collaboration with EGEE and DILIGENT, ETICS has to achieve that its services are used for all build, testing, configuration and distribution of their software INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 11
1 – Strong Collaborations and Repository Dissemination or standardization activities Technical activities European Repository Software build, test and distribution INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 12
2 – Software Quality Assurance Process To promote a Software “Quality Certification Process” that projects may want to adopt in order to assess and promote their products • One of the major achievements ETICS is the definition of the so-called Grid Quality Certification Model (Grid-QCM) • More details on Grid-QCM and the active dissemination activities around it are provided later today in a dedicated presentation INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 13
3 – Scientific Production To intensify scientific production and contribution to standards • The scientific production of ETICS in the second year has been high, several papers have been produced and presented at international conferences or published on peer-reviewed journals • The contribution to standards is more in the QA process area. Again the Grid-QCM plays an important role. • More details on this subject in the dedicated presentations about dissemination (WP 5) and the Grid. QCM model INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 14
4 – User feedback and user interface improvements • • • To structure and integrate the users’ feedback and significantly improve the user interfaces During the second year of activity ETICS has conducted several training and dissemination events, collecting user feedback An on-line questionnaire on usage of the system has been prepared and has been filled by more than 40 users (about 1/3 of the user base at the time) All replies and suggestions have been categorized and used to develop new requirements The Web Application has been redesigned following user recommendations The client performance and usability have been improved and more improvements are under development or planned INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 15
5 – Commercial viability To pay emphasis to elements which are crucial for future commercial viability, such as charging, registration, service level agreements, extensibility, scalability and use of third party resources to run the ETICS services • The elements mentioned in the recommendation are of great importance to ETICS. The following activities have taken place: – User registration is one of the services provided by ETICS. A user registration application is provided to control access to the service and its features – Project registration and SLAs: ETICS policies in the first phase has been ‘open’ due to user requirements. Users have so far asked to have less formal relationships. Now that the services are stable and used, the policy is going to change in ETICS 2. Mo. U are being agreed for example with EGEE and D 4 Science and will be established with other infrastructures like DEISA – Extensibility is based on the ‘plugin framework’, which allows to plug in the system additional tools or functionality. It is based on a published interface that users can also adopt to add custom tools (more details in WP 4 presentation) – Scalability is based on the cross-site submission feature, which allows to create distributed pools of resources and automatically dispatch jobs where resources are available. – Third-party resources can be plugged in directly into a resource pool or using the cross-site submission. An example implemented with EUChina. Grid is described in the WP 2 presentation INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 16
6 – Validation of Testing Components To involve at least two projects to validate the testing components • The testing components have been validated by all major projects using ETICS and of course by ETICS itself. More details are given in the WP 2 and WP 4 presentations and in the demos. In summary: – ETICS uses ETICS to build and test ETICS – EGEE performs routinely code tests, unit tests, deployment tests and some regression testing with ETICS, Usage of ETICS is limited by the availability of automated tests, not by ETICS functionality – DILIGENT performs routinely code tests, unit tests, deployment tests and regression tests with ETICS – OMII-Europe has based its entire test strategy on ETICS and performs standard compliance and interoperability tests with ETICS – EGEE and EUChina. Grid have implemented with ETICS an IPv 6 interoperability infrastructure distributed between CERN, IN 2 P 3 and GARR to test IPv 6 code compliance and regression testing in IPv 6 environments INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 17
7 – Costs Acceleration • • • To justify the cost acceleration, if the project plans to recover the current underspent The budget carried over to year two has been used to recruit additional effort for the tasks that were affected by the initial delay In particular INFN has recruited an additional person for 12 months to work on interoperability and development of unit tests and CERN has recruited a technical student for 3 months to work on deployment testing tools, which compensate the underspent In the case of Engineering the underspent in year 1 was done as planned, since the Quality Assurance process feasibility study where Engineering has been mostly involved is a year 2 activity INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 18
8 – Draft deliverables 8 For some of the above recommendations to be well planned, it is proposed to prepare between month 16 and month 18 a draft version of the following deliverables (originally planned on or after month 20): D 4. 3, D 5. 7, D 5. 9. It is up to the project management to propose a time schedule for such drafts. The final delivery remains at the planned months • The drafts have been prepared and submitted to the EC as per recommended schedule INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 19
The ETICS Timeline (Year 2) Repository Service Design New ETICS Portal Design Co-scheduling development started, first definition of Grid-QCM November 2007 ETICS Service v 2. 0 Released Final Review All-hands meeting (CERN) OGF 19, Joint OMII-Europe Training All-hands meeting (Wisconsin) Jan 07 INFSOM-RI-026753 Jun 07 October 2007 EGEE/JRA 1 Training Dec 07 ETICS Final Review - Project Report - CERN, 15 February 2008 20
Relations with Other Projects • In Year 2 ETICS has spent again considerable effort in establishing and maintaining relationships with other projects: – EGEE: SA 3 (certification), JRA 1 (development), NA 4 (applications), SA 2 (networks) – DILIGENT: integration and testing – OGF: co-chair of a Community Group on Build, Test and Certification of grid software – OMII-Europe: provider of the build and test infrastructure – EUChina. Grid: implementation of plugins for IPv 6 compliance tests – ICEAGE: participation to the Grid Computing School 2007 – Health-e-Child: initial discussions about service testing – Be. In. Grid: on-going discussions on building and testing – ETSI: Standard compliance testing, IPv 6 testing • ETICS has actively taken part to the EC/IST activities: – e. IRG Workshops (Heidelberg) – IST Workshops and Concertation Meetings (Sophia Antipolis) INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 21
International Collaboration One of the major achievements of ETICS has been the establishment of a truly international collaboration between Europe and the US This relationship has established the foundations not only for common software development standards, but also to improve interoperability among different middleware implementations and infrastructures INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 22
Challenges • Effort-constrained projects – Even projects that do believe that doing testing and QA gives benefits, often do not have enough effort to write tests. They like ETICS, but when they realize that ETICS doesn’t write the tests for them, they adopt the build part, but postpone the adoption of the test part to ‘more relaxed times’ • Developer-centric decision processes – In many occasions technical decisions are taken bottom-up, but every developer has different habits or preferred tools and tends to resist to changes especially when he/she is afraid of ‘loosing control’ of the software • Software not designed for automation – Some software we have dealt with in the past two years is not designed for test automation. The deployment and configuration operations are often manual and do not provide ways of easily scripting or instrumenting the procedures • QA-unaware projects – A number of short-term projects are mainly concerned with releasing as soon as possible and believe that doing testing slows their work and cannot improve the quality within the projects lifespan INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 23
Deliverables and milestones • All deliverables/milestones (PM 13 -24) done – 11 deliverables and 4 milestones in 12 months • Additional draft versions of three deliverables submitted to the Commission as per Reviewers’ recommendation n. 8 INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 24
Deliverables List Del. no. Deliverable name WP no. Lead participant Delivery date (PM) D 4. 3 (D) Actual delivery date Distributed test execution system (final release) Coherent documentation of the selected software testing tools and methodology Quality certification feasibility study and sustainability strategy 4 4 DSOFT PM 18 17/07/2007 5 CERN PM 18 27/08/2007 D 5. 9 (D) Plan for using and disseminating Knowledge 5 CERN PM 18 17/07/2007 D 5. 7 Quality certification feasibility study and sustainability strategy 5 CERN PM 20 21/12/2007 D 2. 4 Status of certification, integration and validation test 2 bed setup (final) Uo. W PM 22 21/12/2007 D 3. 3 Final evaluation report and configuration, build and 3 integration system (final release) of public Restricted instead INFN PM 22 21/12/2007 D 5. 8 ETICS portal testing support service (final) due to–further standardization 5 ENG PM 22 21/12/2007 D 4. 3 activities Distributed test execution system (final release) Coherent documentation of the selected software testing tools and methodology 4 4 DSOFT PM 22 21/12/2007 D 5. 9 Plan for using and disseminating Knowledge 5 CERN PM 23 24/01/2008 D 2. 5 Final service usage report 2 CERN PM 24 29/01/2008 D 1. 3 Final project report 1 CERN PM 24 29/01/2008 D 5. 7 (D) INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 25
Financial status (2 nd Period) • The provisional Financial Report covering the 2 nd reporting period shows a total cost of 944 877 € and requests a total EC contribution of 822 087 €. It represents 59% of the total maximum EC contribution as per the Annex I of the Contract. • Internal Cost Claims (Form C) are being submitted by all partners to the Coordinator and from the Coordinator to the Commission. Audit certificates are being finalized. • The breakdown of expenses is as follows: – Direct Costs: 79% – Indirect Costs: 21% – Personnel: 89% – Travel, subsistence and other costs: 11% INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 26
Financial status (Full project) • The provisional Financial Report covering the full project duration shows a total cost of 1 551 769 € and requests a total EC contribution of 1 400 000 € as per the Annex I of the Contract. • The breakdown of expenses is as follows: – Direct Costs: 76% – Indirect Costs: 24% – Personnel: 87% – Travel, subsistence and other costs: 13% INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 27
Manpower levels • PPT tool to monitor project effort consumption (timesheets) – 28 people registered across Europe and USA – Total equivalent of ~ 11 FTE/year A large fraction of the unfunded effort has been provided by University of Wisconsin Consumed Effort Vs. Total Planned Effort at Dec 2007 Funded 112% INFSOM-RI-026753 Unfunded 105% Total 111% ETICS Final Review - Project Report - CERN, 15 February 2008 28
Summary • ETICS has closely followed the planned work program and it has delivered a fully functional distributed service for Building and Testing software on more than 15 different platforms and a large repository or packages and reports • It has met the expected number of supported projects and users for the two years of its plan (3 major active projects, 18+ registered projects, ~220 users). The major projects (EGEE (JRA 1, SA 3 and NA 4), DILIGENT and OMII-Europe) use its services for production build and test tasks • ETICS has established strong relationships with other projects and initiatives in the field of software quality assurance and certification and has established itself as a reliable service provider and partner in this area and has developed the Grid-QCM quality assurance model • The project is close to the expected levels of effort and budget and has completed all the planned tasks INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 29
Thanks http: //www. eu-etics. org INFSOM-RI-026753 ETICS Final Review - Project Report - CERN, 15 February 2008 30
- Slides: 30