The ETICS Build and Test Service ETICS Project

The ETICS Build and Test Service ETICS Project, CERN www. eu-etics. org INFSOM-RI-026753

Contents • • • Introduction Use Cases Service – – – ETICS Features Service architecture Data model NMI execution engine Reporting and repository • Conclusions and future work INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 2

Introduction INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 3

ETICS in a nutshell • ETICS stands for e-Infrastructure for Testing, Integration and Configuration of Software • Project started on January 1 st, 2006 and has a duration of two years • Goal is to provide efficient automated tools for building and testing • ETICS is born from the experience of developing, integrating and testing grid and distributed software INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 4

Usecases INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 5

Basic build usecase • • • Context: A user develops, for example, a Grid service (as a single component) This service relies on existing software (dependencies) User performs following actions – For each required dependencies – – – – Find documentation Identify correct version to use (release, maintenance, beta, etc) Identify if a binary package is adequate In case building is required (very often the case), find build instructions Build code Install and configure code (using available documentation) Verify that the code is built, installed and configured correctly Find developer contact information in case of problems – – – Inter-dependencies must be understood and version clashes resolved Write a script to build all dependencies Write a script to build the current component (Grid service), including fetching code, build and test instructions/command – Store the component in a repository for others to use • • This above process has to be executed, in full or in part, for each targeted platforms For each platform, a machine has to be setup and maintained If regular builds and tests are required, a scheduled task must be setup for each required platform Each version of the software must be advertised to potential user communities INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 6

What does this usecase tells us? • The above usecase can take between a few hours to a few weeks, depending on the number of dependencies • When complexity grows, so does the time required to get started • Different software teams will define the required information differently, which takes longer for the developers to integrate their work • Unless well controlled, builds and tests might not be reproducible • Significant cost in maintaining different machines for build and test INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 7

Same Usecase using ETICS • Context: A user develops, for example, a Grid service (as a single component) • This service relies on existing software (dependencies) • Using ETICS, user: – Enters the commands to checkout, build and test the component – Selects the dependencies this component requires (from the rich catalog of existing components in ETICS) – Builds and tests the component locally – Submit build and test for parallel execution on all required platforms – All build and test artefacts are registered in the ETICS repository for public access INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 8

Service INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 9

ETICS Features • Out-of-the-box automatic build/test system • Multiplatform support for development and testing • Powerful and flexible automatic dependency management (internal and external dependencies) • Trigger coding convention checks, unit test, documentation tools and collect and publish the results • Possible to build everything from source or use pre-built binary packages • Designed to support several Version Control Systems • Possible to register artefacts on the repository • Publishes run-time configuration information such as environment variables, runtime dependencies, etc … • Produce different package formats (rpms, tar, msi, debs) • Produces rich build and test reports • Ability to run custom plugins during build and test procedure INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 10

The ETICS partners Build system, software configuration, service infrastructure, dissemination, EGEE, g. Lite, project coord. The Condor batch system, distributed testing tools, service infrastructure, NMI Software configuration, service infrastructure, dissemination Web portals and tools, quality process, dissemination, DILIGENT Test methods and metrics, unit testing tools, EBIT INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 11

Service architecture Web Application Web Service Via browser Report DB Project Build/Test Artefacts DB NMI Execution Engine Clients WNs INFSOM-RI-026753 NMI Client Wrapper ETICS Infrastructure EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 12

Service Architecture • ETICS Service build as an N-tier architecture • All elements of ETICS are configured, built and tested with ETICS • ETICS Service is Open Source: Apache 2. 0 license INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 13

Data model - 1 • Project: container for components and subsystems • Subsystem: container for components • Component: a module configuration can also be made of sub-module’s configurations • Module: generic way to refer to project, subsystem, component • Configuration: different versions of the modules (e. g different CVS tags, dependencies, platforms) INFSOM-RI-026753 P S Conf C EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 14

Data model - 2 • Platform: represents an specific combination of architecture, operating system and C/C++ compiler – E. g. slc 3_ia 32_gcc 323 • Allows users to specify platform specific metadata – – – VCS commands (e. g. CVS) Build commands Test commands Properties Environment variables Dependencies INFSOM-RI-026753 Plat P S Conf C EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 15

Getting started INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 16

ETICS registration https: //etics. cern. ch: 8443/etics. Admin/ /public/registration/request. Registration. jsp INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 17

ETICS Web Application Web Service Via browser Report DB Project Build/Test Artefacts DB NMI Execution Engine Clients WNs INFSOM-RI-026753 NMI Client Wrapper ETICS Infrastructure EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 18

Web Applications • ETICS composed of following Web Applications: – – – Build and Test (WA) Repository Administration • The applications are being integrated under a single user experience INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 19

WA – Description (1) • Build and Test Web Application (WA) allows users to – – Browse metadata Edit metadata Submit build and test jobs Compatible with Firefox browser • State-full and secure application • WA written in Java, jsp, javascript and JDuck (Engineering client/server web framework) • Interfaces with the ETICS Build and Test Web Service INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 20

WA - Description (2) • Web Application is the simplest way to understand ETICS • We will now go through a few simple steps to explain how ETICS works using the Build and Test Web Application • Note: Much more details will be provided during the demo INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 21

WA – Project Metadata (3) • All ETICS registered projects are available from a central entry point: https: //etics. cern. ch: 8443/etics • Project details are shown upon selection in a dropdown list Project Details INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 22

WA - Project Metadata (4) Browse Panel View/Edit Panel Configuration List INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 23

WA- Configuration Metadata (5) • • • ETICS defines configuration similar to version in CVS Configurations can be defined differently for different platforms Configurations include what to do to checkout, build and test software modules: – Version control system (VCS) - e. g. CVS, SVN - commands Build commands Test commands Properties and environment Dependencies • • – – Properties and environment variables are inherited and propagated Each command includes clear set targets. e. g. build: clear, init, checkstyle, doc, compile, package, publish INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 24

WA - Configuration Metadata (6) • Here a few examples • Version Control System Commands: – Checkout: cvs –d ${vcsroot} co ${module. Name} • Build Commands: – – – Init: mkdir –p etc; cp ${security. location}/etc Configure: . /configure Compile: ant –f build. xml • Test Commands: – Init: . /scripts/install-script ${target-hostname} – Test: . /scripts/test-script ${report. Dir} INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 25

WA - Remote build (7) INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 26

ETICS Reports INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 27

ETICS Repository – (1) INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 28

ETICS Repository – (2) • Trees – – – AJAX Technology Easy browsing trees Organised by Project • Detailed description of files – Project, Package, Version and Platform – Type, Size and Last Modified – Download Link http: //etics-test. cern. ch: 8080/repository. Browser/ INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 29

ETICS Administration Tool – (1) INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 30

ETICS Administration Tool – (2) INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 31

Command-line Client Web Application Web Service Via browser Report DB Project Build/Test Artefacts DB NMI Execution Engine Clients WNs INFSOM-RI-026753 NMI Client Wrapper ETICS Infrastructure EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 32

Command-line Client – (1) • Build and Test Command-Line client (CLI) allows users to – – Build and test locally Browse metadata Edit metadata Submit build and test jobs • State-full and secure application • CLI written in Python, using ZSI (SOAP toolkit for Python) for web service interfacing and serialisation • Interfaces with the ETICS Build and Test Web Service INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 33

Command-line Client Local build – (2) • Download project metadata: etics-get-project <project-name> Action: download project metadata • Checkout and/or download software: etics-checkout –c <config-name> <module-name> Action: download binary packages or fetch code from source • Build the software: etics-build <module-name> Action : build and unit test code hieratically, and generate packages and reports • Test the software: etics-test <module-name> Action : test code hieratically, and generate packages and reports INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 34

Command-line Client Editing - (3) • Command-line Client can also be used for editing: etics-module [<options>] [<module-name>] etics-configuration [<options>] [<module-name> • Can be used for scripting or interactive sessions • Provides the same features as the Web Application INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 35

Command-line Client Plugin-Framework – (4) • Framework allows users with investments in existing build, test, certification and/or software quality assurance tools to leverage this investment • Users can select, from a rich set, which metrics are important to them and have ETICS automatically calculate them during build and test, and make them visible • Promote inclusion of contributions from ETICS user community (more in this in dissemination presentation) • Examples of already available plugins: – IPv 6 compliance tests – x. Unit tests automatic execution – Multi-language line of code count (SLOC) INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 36

Build and Test Web Service Web Application Web Service Via browser Report DB Project Build/Test Artefacts DB NMI Execution Engine Clients WNs INFSOM-RI-026753 NMI Client Wrapper ETICS Infrastructure EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 37

WS - (1) • Build and Test Web Service (WS) provides single secure interface to the ETICS backend system – Metadata (persisted in database) – Execution engine – Security (role based) • Heart of the ETICS System, it performs complex queries on behalf of the clients (WA and CLI) INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 38

WS – Security – (2) • All accesses to the WS must be performed using a secure port • Authentication is performed using digital certificates (x. 509) • Authorisation is role-based • Clients connecting without certificate are mapped to Guest, with minimal privileges (most likely just read access) • Service can be setup to leverage certification authority management efforts such as EUGrid. PMA INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 39

Execution Engine Web Application Web Service Via browser Report DB Project Build/Test Artefacts DB NMI Execution Engine Clients WNs INFSOM-RI-026753 NMI Client Wrapper ETICS Infrastructure EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 40

NMI - Execution engine – (1) • NMI is an execution engine internally used by ETICS to manage remote build and test jobs on different platforms • NMI is a layer built on the top of Condor • Condor is a middleware distribution that “provides mechanisms and policies supporting High Throughput Computing (HTC) on large collections of distributed computing resources” • Both NMI and Condor are developed and supported by our partner, the University of Wisconsin in Madison, USA INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 41

NMI - Web interface – (2) http: //etics. cern. ch/nmi INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 42

NMI- Additional features – (3) • Following features were added to NMI/Condor to better support the ETICS Execution Engine – Private resources: projects can provide specific resources for exclusive usage – Freeze: lock the node for further analysis in case of failure – Root-enabled environment: build or test jobs can request to run as root • Users can register private machines if special setup and/or specific licenses are required INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 43

NMI- Using the ETICS client – (4) • For users to trust the system, local and remote builds have to be executed in as close a condition as possible • Once a build or test job has landed on a Condor worker-node (WN), the ETICS client takes over • The exact same sequence is executed on the WN, as if on a local machine – Setup – Get project – Checkout – Build/Test • The difference is that the WN is controlled and maintained by the ETICS experts and guaranteed to be a clean standard installation • This approach guarantees that over time, builds and tests will be reproducible, a important tenet in software quality standards INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 44

NMI - ETICS sites – (5) • Server at CERN etics. cern. ch – SLC 3, SLC 4, RHES, Debian, FC (32 - and 64 -bits), Win. XP, Mac OSX • Server at Univ. of Wisconsin, Madison, USA – HP HPUX, IBM AIX, OSF 1, SGI Irix, Sun Solaris, Linux (several flavours) • Server at INFN, Bologna, Italy – Cent. OS, SLC 3, SLC 4 (32 - and 64 -bits) INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 45

ETICS Reports – (1) INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 46

ETICS Reports – (2) INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 47

ETICS Reports – (3) INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 48

ETICS Reports – (4) • Trees – Easy browsing trees – Organised by Project or by Build Date • Detailed reports – – Package Summary Build Configuration Build Logs http: //etics. cern. ch: 8080/report. Browser/ INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 49

ETICS Repository – (1) INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 50

Conclusions • ETICS defined, integrated and delivered a novel Configuration, Build and Test Service • The ETICS service was used in an incremental fashion to build and test the ETICS service, providing rapid feedback to the team • Currently 20 projects registered and 74 users • Always in contact with early adopters – Feedback really useful • Future functionality for next year will include – Handle complex testing scenarios (automatic deployment and test) – Virtual machines • If you need further information do not hesitate to contact us etics-discuss@cern. ch etics-support@cern. ch INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 51

Thanks http: //www. eu-etics. org INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6 -7 March 2007 52