EPICS Automatic Regression Test Facility for EPICS Support

EPICS Automatic Regression Test Facility for EPICS Support Modules

Introduction EPICS • Diamond has approx 200 Support modules – Device/Driver modules and EPICS DB template – Many modules used across many projects – Need to have confidence that when a project is rebuilt these modules work – Issues of changing EPICS /Tool chain/OS/ version • Four major parts to software engineering: – – Design: Should come first! Code: We are good at this. Documentation: Doxygen etc. Test: Often manual and hard to repeat.

The Lessons EPICS • Testing is not an optional extra. – Carried out by the Software Developer – Carried out by the Software User – Automated • Tests must be properly repeated at least before every release and preferably more often. • Tests are easier to repeat if they are completely automatic.

Diamond is Constructing • A software framework that allows the creation of automatic test suites for modules. • A script that runs test suites and collects reports. • A hardware test bench containing ‘high value’ equipment. • Software simulations of equipment. EPICS

The Framework EPICS • A software framework that aids the creation of automatic test suites for modules. • Based on Py. Unit, the Python unit testing library. • Test reports conform to the TAP protocol and/or JUnit XML format. • Monitoring of EPICS database records for coverage reporting. • Hardware and simulation targets can use same test suite. 11/10/09 Mark Heron Automatic Regression Test Facility for EPICS Support Modules

The Run Script EPICS • Searches a directory tree for modules with automatic test support. • Runs tests suites it finds, logging results and output. • Can run multiple tests in parallel subject to resource constraints. • Level of diagnostic output can be specified. • Tests run can be restricted to single module, single target, single case. • “dls-run-tests -h” for command help. • The run script is designed to operate from the ‘support’ directory of the work tree. 11/10/09 Mark Heron Automatic Regression Test Facility for EPICS Support Modules

EPICS Directory Structure Changes support <module. Name> new directories etc test make. Documentation make. Iocs 11/10/09 Mark Heron iocs Automatic Regression Test Facility for EPICS Support Modules tests. py

Device Simulations EPICS • Based on existing DLS practice. • Written in Python. • Currently capable of simulating any device that communicates through serial or IP connection. • Support for instrumentation to allow protocol coverage reporting.

Framework Class Diagram 11/10/09 Mark Heron Automatic Regression Test Facility for EPICS Support Modules EPICS

Example – The Test Suite from pkg_resources import require('dls. autotestframework ') from dls. autotestframework import * class Fw 102 Test. Suite(Test. Suite): def create. Tests(self): # Define the targets for this test suite Target("simulation", self, ioc. Directory="example", ioc. Boot. Cmd="bin/linux-x 86/stfw 102 Ex_sim. boot", epics. Db. Files="db/fw 102 Ex. db", sim. Devices=[Sim. Device("controller 1", 9016)], gui. Cmds=['edm -m "P=FGZ 73762, M=: WHEEL 1" - eolc -x data/fw 102. edl']) Target("hardware", self, ioc. Directory="example", ioc. Boot. Cmd="bin/linux-x 86/stfw 102 Ex. boot", epics. Db. Files="db/fw 102 Ex. db", gui. Cmds=['edm -m "P=FGZ 73762, M=: WHEEL 1" - eolc -x data/fw 102. edl']) # The tests Case. Local. Increment. Switch(self ) Case. Local. Decrement. Switch(self ) Case. Power. Off. On(self) Case. Pv. Increment(self) Case. Pv. Decrement(self) Case. Pv. Multiple. Increment(self ) Case. Pv. Multiple. Decrement(self ) Case. Pv. Set. Position(self) Case. Pv. Trigger. Mode(self) Case. Pv. Polling(self) 11/10/09 Mark Heron Automatic Regression Test Facility for EPICS Support Modules EPICS

Example – Intermediate Class class Fw 102 Case(Test. Case): def cur. Device. Pos(self): ''' Get the current wheel position from the device simulation ''' result = 0 self. command("controller 1", " getpos") args = self. recv. Response("controller 1", "pos", 1) if args is not None: result = int(args[0]) return result def verify. Position(self, intended): ''' Verify that the wheel is in the intended position ''' if self. simulation. Device. Present("controller 1"): self. verify(self. cur. Device. Pos (), intended) self. verify. Pv("FGZ 73762: WHEEL 1: POSITION_RBV", intended) self. verify. Pv("FGZ 73762: WHEEL 1: POSITION", intended) self. verify. Pv("FGZ 73762: WHEEL 1: INPOS", 1) 11/10/09 Mark Heron Automatic Regression Test Facility for EPICS Support Modules EPICS

Example – A Test Case EPICS class Case. Local. Increment. Switch(Fw 102 Case): def run. Test(self): '''The local increment switch. ''' if self. simulation. Device. Present("controller 1"): # Check the current position of the wheel before = self. initial. Position() # Take the wheel round twice for i in range(12): # Now advance the wheel using the local button self. command("controller 1", "incr") after = before + 1 if after > 6: after = 1 # Check the final position of the wheel self. sleep(2) self. diagnostic("Before=%d, after=%d" % (before, after), 1) self. verify. Position(after) before = after 11/10/09 Mark Heron Automatic Regression Test Facility for EPICS Support Modules

Example - Test Report [1] 1. . 10 [fgz 73762@pc 0054 diamondtest]$. /runtests. py -t simulation -f default. config -i -g -b -p 3 -l tests. log -q -m fw 102 [1] ok 1 - The local increment switch. [1] ok 2 - The local decrement switch. [1] ok 3 - Power off and on. [1] ok 4 - The PV increment command. [1] ok 5 - The PV decrement command. [1] ok 6 - Fast increment sync correction. [1] ok 7 - Fast decrement sync correction. [1] ok 8 - Movement directly to a position. [1] ok 9 - Control of the trigger mode. [1] ok 10 - Control of the polling mechanism. [1] # =============== [1] # Passed 10/10 tests, 100. 00% okay, in 260. 57 s [1] # =============== [1] # Sim device controller 1 coverage report: [1] # setpos: ok [1] # settrigger: ok [1] # getpos: ok [1] # gettrigger: ok [1] # =============== [1] # EPICS database coverage report: [1] # FGZ 73762: WHEEL 1: TRIGGER(mbbo): ok [1] # FGZ 73762: WHEEL 1: TRIGGER_RBV(mbbi): ok [1] # FGZ 73762: WHEEL 1: RESTART(ai): not touched [1] # FGZ 73762: WHEEL 1: INPOS(calc): ok [1] # FGZ 73762: WHEEL 1: REINIT 1(fanout): ok [1] # FGZ 73762: WHEEL 1: POSITION(mbbo): ok [1] # FGZ 73762: WHEEL 1: COMMSFAIL(bi): ok [1] # FGZ 73762: WHEEL 1: POSITION_RBV(longin): ok [1] # FGZ 73762: WHEEL 1: CALCCOMMS(calcout): ok [1] # FGZ 73762: WHEEL 1: DISABLEPOLL(bo): ok [1] # FGZ 73762: WHEEL 1: STEPBK(calcout): ok 11/10/09 Mark Heron Automatic Regression Test Facility for EPICS Support Modules EPICS

Hudson • Use Hudson framework – For scheduling and reporting of test • A Continuous Integration Engine – Checkout from repository – Does some actions • Build, Post-build actions, (execute test script) – Determine Status • Run on build farm using defined environment – EPICS version, OS and tool chain • Invoked by period checks on SVN repository • Status by web and emails to developers • Success (all actions successful) • Unstable (some tests failed) • Failed (compile error) 11/10/09 Mark Heron Automatic Regression Test Facility for EPICS Support Modules EPICS

EPICS 11/10/09 Mark Heron Automatic Regression Test Facility for EPICS Support Modules

Progress Todate EPICS • Python Framework and documentation is in place • Ran a week long Testathon – Mixture of training and test development involving all Controls Group developers – Tests developed for a number of modules – Will run again Jan 2010 • Specifying and building hardware test stand

Acknowledgement EPICS • This is being lead by Jon Thomson 11/10/09 Mark Heron Automatic Regression Test Facility for EPICS Support Modules

EPICS 11/10/09 Mark Heron Automatic Regression Test Facility for EPICS Support Modules