Testing with CTest and CDash P MatoCERN Motivation

Testing with CTest and CDash P. Mato/CERN

Motivation � Work done in the context of Geant 4 ◦ Presented at last Collaboration meeting � Facilitate testing and packaging using the tools (CTest and CPack) developed by Kitware and distributed as part of CMake. ◦ No need to reinvent the wheel ◦ Perfect integration to CMake build system � Testing is absolutely essential for any “agile” software development activity � Developers should participate to the testing activity ◦ Adding, running, surveying tests should be extremely easy and effortless P. Mato/CERN 2

Adopting Kitware Process? P. Mato/CERN 3

Defining and Running Tests P. Mato/CERN

Defining Tests � With CTest/CMake tests are defined using the add_test(. . . ) command ◦ Developed a high-level function GEANT 4_ADD_TEST(…) to encapsulate the Geant 4 policies and an specialized command wrapper for the tests ◦ Test programs themselves can also be built as part of the test (BUILD option) # function GEANT 4_ADD_TEST( <name> # COMMAND cmd [arg 1. . . ] # [PRECMD cmd [arg 1. . . ]] [POSTCMD cmd [arg 1. . . ]] # [OUTPUT outfile] [ERROR errfile] # [ENVIRONMENT var 1=val 1 var 2=val 2. . . ] # [DEPENDS test 1. . . ] # [TIMEOUT seconds] # [DEBUG] # [SOURCE_DIR dir] [BINARY_DIR dir] # [BUILD target] ) P. Mato/CERN 5

Defining Tests: Tests cmake_minimum_required(VERSION 2. 6 FATAL_ERROR) project(test 21) find_package(Geant 4 REQUIRED) include(${Geant 4_USE_FILE}) Used a macro to simplify writing GEANT 4_EXECUTABLE(test 21. cc src/*. cc) GEANT 4_EXECUTABLE(test 21_geninput test 21_hadronic_exerciser. cc) #---Test definitions----------------------GEANT 4_ADD_TEST(test 21_geninput COMMAND test 21_geninput OUTPUT test 21. in BUILD test 21_geninput ) GEANT 4_ADD_TEST(test 21 COMMAND test 21. in BUILD test 21 ENVIRONMENT ${GEANT 4_TEST_ENVIRONMENT} DEPENDS test 21_geninput) P. Mato/CERN 6

Defining Tests: Examples � Tests are also defined for each example ◦ GEANT 4_ADD_TEST() specified outside the CMake. Lists. txt of the example (tests/CMake. Lists. txt) ◦ Full paths are required instead of relative ones GEANT 4_ADD_TEST(example-nov-n 01 COMMAND ${CMAKE_BINARY_DIR}/examples/novice/N 01/example. N 01 ${CMAKE_SOURCE_DIR}/examples/novice/N 01/example. N 01. in SOURCE_DIR ${CMAKE_SOURCE_DIR}/examples/novice/N 01 BINARY_DIR ${CMAKE_BINARY_DIR}/examples/novice/N 01 BUILD example. N 01) P. Mato/CERN 7

Running Tests � The ctest executable is provided and distributed as part of CMake ◦ Many options available to select what tests to run, configuration, verbosity, etc. ◦ Used by developers and the continuous integration testing >>>> ctest -R test 10 Test project /Users/mato/Development/G 4/make Start 90: test 10 1/2 Test #90: test 10. . . . Start 91: test 10 -large. N 2/2 Test #91: test 10 -large. N. . . . . Passed 15. 32 sec Passed 70. 76 sec 100% tests passed, 0 tests failed out of 2 Total Test time (real) = 86. 34 sec P. Mato/CERN 8

Scripting with CTest � Complex tests can be run using dedicated scripts ◦ CMake scripting �Platform independent ◦ Variables to customize CTest behavior ◦ Commands to drive the whole test procedure � Executing the script ◦ ctest –S <script> set(CTEST_CUSTOM_TESTS_IGNORE test 19 test 29 test 39 test 49) if(WIN 32). . . endif() set(CTEST_TIMEOUT 1200) ctest_empty_binary_directory( ${CTEST_BINARY_DIRECTORY}) ctest_start("Continuous") ctest_update() ctest_configure() ctest_build() ctest_test(PARALLEL_LEVEL ${ncpu}) ctest_submit() P. Mato/CERN 9

Uploading test results to CDash � The results of running a Nightly or Continuous integration tests as well as developer initiated (Experimental) test runs can be very easily uploaded to CDash ◦ Customizable views with different level of details ◦ Command wrapper output ◦ E-mail notification � Currently squatting the Desy service for testing https: //aidasoft. desy. de/CDash/index. php? project=Geant 4 % ctest –j 4 –D Experimental P. Mato/CERN 10

CDash Architecture cron Developer workstation builder upload results builder CDash server cron builder Dashboards P. Mato/CERN 11

CDashboard set(CTEST_PROJECT_NAME "Geant 4") set(CTEST_NIGHTLY_START_TIME "00: 00 CET") set(CTEST_DROP_METHOD "http") set(CTEST_DROP_SITE "aidasoft. desy. de") set(CTEST_DROP_LOCATION "/CDash/submit. php? project=Geant 4") set(CTEST_DROP_SITE_CDASH TRUE) P. Mato/CERN 12

Continuous Integration Testing �A pilot ‘continuous’ integration and testing has been setup (few times/day) P. Mato/CERN 13

Pros and Cons � Pros ◦ Loosely coupled set of build and test servers �No need for an overall workflow or scheduling ◦ Support for continuous, nightly and user triggered builds and tests �New modes can be defined ◦ Testing on special platforms can be provided by interested customers ◦ Extremely simple, almost nothing to configure. � Cons ◦ Limited customization ◦ Dashboard may became overcrowded P. Mato/CERN 14

Packaging P. Mato/CERN

Packaging with CPack � CPack (bundled with CMake) is able to create professional platform specific installed ◦ GZ and Self extract TGZ (STGZ), Null. Soft Scriptable Install System (NSIS), OSX Package. Maker, RPM, Deb � For Geant 4: ◦ Binary and Source installers (TGZ, NSIS, Package. Maker) ◦ User can select what components to install P. Mato/CERN 16

Building the Installers � To create the installers you need to enable a couple of CMake options to include the examples & data: ◦ GEANT 4_INSTALL_DATA ◦ GEANT 4_INSTALL_EXAMPLES � Make the 'package' target for the binary distribution and 'package_source' for the sources P. Mato/CERN 17

Conclusions � Provided the definition, building and running of tests using CMake/CTest/CDash tools ◦ The main goal has been to make it easy for developers to define, run, automate and monitor tests � CDash could offer a competitive replacement to the LCG nightlies � CPack is a nice utility to provide ‘professional’ packaging with very little effort P. Mato/CERN 18