TESTING SCREAM FOUCAR ALLHANDS 2019 SCREAM BASIC TESTING

TESTING SCREAM FOUCAR, ALL-HANDS 2019

SCREAM BASIC TESTING STACK • CMake : Configure tool for SCREAM • CTest : CMake module for running tests • Catch 2 : C++ unit-test framework • Individual unit tests

EXAMPLE: ADDING A UNIT TEST • CMake needs to be able to find the directory containing your unit test • • Some “connected” CMake. Lists. txt must call add_subdirectory(your_test_dir) Your_test_dir must have a CMake. Lists. txt that … • • • Defines an executable Sets up includes and libs for that executable Adds a test OR • Use our test macro! • You can skip all these steps if you’re just adding a test to an existing test source file

set(MY_TEST_SRCS ${SCREAM_SRC_DIR}/share/util/scream_catch_main. cpp my_tests. cpp ) link_directories(${SCREAM_LIBRARY_DIRS} ${SCREAM_TPL_LIBRARY_DIRS}) add_executable(my_tests ${MY_TEST_SRCS}) target_include_directories(my_tests PUBLIC ${SCREAM_INCLUDE_DIRS} ${SCREAM_TPL_INCLUDE_DIRS} ${CATCH_INCLUDE_DIR}) target_link_libraries(my_tests scream_control scream_dynamics p 3 rrtmgp shoc scream_share ${SCREAM_TPL_LIBRARIES}) set_target_properties(my_tests PROPERTIES LINK_FLAGS "${SCREAM_LINK_FLAGS}") add_test(run_my_tests)

INCLUDE (Scream. Utils) Create. Unit. Test(my_tests. cpp scream_control scream_dynamics p 3 rrtmgp shoc scream_share)

#include "catch 2/catch. hpp" #include "control/atmosphere_driver. hpp" #include "physics/rrtmgp. hpp" #include "physics/shoc. hpp” namespace { TEST_CASE("scream", ”my_tests") { int val_driver = scream: : control: : driver_stub(); int val_rrtmpg = scream: : rrtmgp_stub(); int val_shoc = scream: : shoc_stub(); REQUIRE(val_driver == 42); REQUIRE(val_rrtmpg == 42); REQUIRE(val_shoc == 42); } } // empty namespace

CREATEUNITTEST MACRO • Create. Unit. Test is more than just convenience • Allows customized ”sweeping” over both MPI ranks and threads! Create. Unit. Test(demo. cpp scream_share THREADS 1 ${SCREAM_TEST_MAX_THREADS} ${SCREAM_TEST_THREAD_INC} MPI_RANKS 1 ${SCREAM_TEST_MAX_RANKS} ${SCREAM_TEST_RANK_INC}) • If • • • SCREAM_TEST_MAX_THREADS=10 SCREAM_TEST_THREAD_INC=1 SCREAM_TEST_MAX_RANKS=10 SCREAM_TEST_RANK_INC=1 Then, 100 tests will be created! Can run specific combinations: % ctest -R demo_ut_np 15_omp 13

DEVELOPMENT TESTING • Why all this fuss? • To run unit tests: % cmake. . # Assumes build dir is right below scream % make –j 8 ; make baseline % ctest

ACCEPTANCE TESTING • In order to “accept” scream on current system, we need to: • Do baseline testing • Test multiple different configurations • test-all-scream : A custom python script to launch a full set of acceptance tests (scream-docs/perf-scripts/test-allscream) • CTest script : drives a full individual test cdash job (conf+build+test) (scream/cmake/ctest_script. cmake) • CMake : Configure tool for SCREAM • CTest : CMake module for running tests • Catch 2 : C++ unit-test framework • Individual unit tests

ACCEPTANCE TESTING EXAMPLE • Why all the additional fuss? • To run acceptance tests for current machine for current branch: % test-all-scream $(which mpicxx) -m melvin • As of today, you just: • • Ran a full DEBUG set of tests Did a baseline comparison against origin/master Ran a full DEBUG set of tests with packsize = 1 and FPE on Did a single-precision build • If we decide to change what it means to ”accept” scream, all we need to do is change test-all-scream. Travis and Jenkins will instantly see the changes.

DISTRIBUTED ACCEPTANCE TESTING • Things get even more complicated since SCREAM must be tested on multiple platforms • Gather_all_data : A custom python script for setting up the accepted environment and launching tests on various machines • test-all-scream : A custom python script to launch a full set of acceptance tests • CTest script : drives a full individual test “job” • CMake : Configure tool for SCREAM • CTest : CMake module for running tests • Catch 2 : C++ unit-test framework • Individual unit tests

DISTRIBUTED ACCEPTANCE TESTING EXAMPLE • Run acceptance tests (on current commit) on all machines known to gather_all_data (in parallel): % gather_all_data Completed test-all-scream analysis on melvin Completed test-all-scream analysis on waterman Completed test-all-scream analysis on blake Completed test-all-scream analysis on white Completed test-all-scream analysis on bowman

GITHUB TRAVIS CI TESTING • Defined by scream/. travis. yml • Runs on every PR, including new runs when PR gets modified • Basically, just runs test-all-scream • Some extra setup is needed to grab kokkos submodule and to set up coverage gathering

JENKINS NIGHTLY TESTING • Just runs: gather_all_data --local -m $machine --submit • Each job only differs by value of $machine • Making new jobs is extremely easy once gather_all_data has data for the target machine

SCREAM DASHBOARD

CONCLUSION • We have made the SE investments to ensure SCREAM has great testability • Our CMake/CTest/Macros stack make it easy to add and run individual tests • test-all-scream encapsulates our acceptance testing across all systems (Travis, Jenkins, etc) • Python scripts make it easy to launch an immense amount of testing