EFFECTIVE TESTING OF HEALTHCARE SIMULATION SOFTWARE Presenter Shant

EFFECTIVE TESTING OF HEALTHCARE SIMULATION SOFTWARE Presenter: Shant Mandossian

SIMULATION SOFTWARE • Modeled after real world systems. • Allows users to observe an operation without actually performing it. • Used in initial system designs in order to optimize component selection.

SIMULATION SOFTWARE Two Different Types • Discrete Event Simulation • Continuous Simulation

SIMULATION SOFTWARE CONTINUOUS SIMULATION

SIMULATION SOFTWARE CONTINUOUS SIMULATION

SIMULATION SOFTWARE DISCRETE EVENT SIMULATION

SIMULATION SOFTWARE DISCRETE EVENT SIMULATION

SIMULATION SOFTWARE DISCRETE EVENT SIMULATION Significant Components • Input • Oracle • Agent Resources • Resource Manager • Non-Agent Resources • Output • Artifacts

SIMULATION SOFTWARE TESTING COMMON METHODS • Research in simulation software focuses on validation • Insufficient investigation into testing the software itself.

SIMULATION SOFTWARE TESTING WHY IS IT DIFFICULT? • Simulation software exists to find the answers in the first place. • No test oracle exists because exact simulation results are simply not known. • Subtle errors in implementation can greatly affect the accuracy of the simulation.

SIMULATION SOFTWARE TESTING WHY IS IT DIFFICULT? • Simulations consist of many interacting components • Each step or event can take a random amount of time to complete.

SIMULATION SOFTWARE TESTING N-VERSION PROGRAMMING • Early approach to testing software without a test oracle • Independent programming teams develop an application from the same specifications. • Identical sets of input are fed into each version of the program. • Results are compared in an effort to detect defects.

SIMULATION SOFTWARE TESTING METAMORPHIC TESTING •

SIMULATION SOFTWARE TESTING METAMORPHIC TESTING • The Bad • Like most tests, this method only shows us if defects exist. • Not that the software is defect-free. • The Good • Provides a way of revealing potential defects without the help of a test oracle

SIMULATION SOFTWARE HEALTHCARE • Healthcare professionals rely on simulation software

SIMULATION SOFTWARE HEALTHCARE • Software is used to simulate elements of the human body for purposes of training and decision making.

SIMULATION SOFTWARE HEALTHCARE • Simulations of a specific organ like the heart. • Simulations of an entire physiological system.

SIMULATION SOFTWARE HEALTHCARE • Emergency Department

SIMULATION SOFTWARE TESTING EMERGENCY DEPARTMENT SIMULATOR Process • Patient arrives • Patient is seen by a triage-nurse • Patient registers with the clerk • Clerk collects information puts in the patient record • If a bed is available, patient goes to treatment area • If all beds are occupied, patients waits in the waiting room until a bed is cleared • Patient is taken to a free bed • Patient is seen by a nurse • Doctor takes assessment of the patient • Assessment may result in tests • Once tests are done, doctor admits or discharges the patient • Nurse fills out paperwork.

SIMULATION SOFTWARE TESTING EMERGENCY DEPARTMENT SIMULATOR • Agent Resources • Clerk • Triage Nurse • Doctor • Non-Agent Resources • Beds

SIMULATION SOFTWARE TESTING EMERGENCY DEPARTMENT SIMULATOR Finding Metamorphic Properties • All resources used in simulation • Doctors, Nurses, beds, etc… • Input given to the simulator • Simulation instructors • Program source code

SIMULATION SOFTWARE TESTING EMERGENCY DEPARTMENT SIMULATOR Potential Bottleneck • Patients wait until a bed is available. • Run multiple simulations with varying numbers of beds, but keep all other resources constant. • Plot average Length of Stay (LOS) for each runthrough of the simulator to see if results are as expected.

SIMULATION SOFTWARE TESTING EMERGENCY DEPARTMENT SIMULATOR Results show that LOS improves with more beds, but the amount of improvement decays.

SIMULATION SOFTWARE TESTING EMERGENCY DEPARTMENT SIMULATOR Plotting the utilization of doctors over the number of beds shows that on average, utilization increases but also with a decay

SIMULATION SOFTWARE TESTING EMERGENCY DEPARTMENT SIMULATOR • Both Length of Stay (LOS) of patients vs. number of beds and Doctor utilization vs number of beds, displayed expected results. • New test case with unexpected results. • Five patients arrive, 6 minutes apart. • Resources Available • One nurse, doctor, clerk, and triage nurse. • Three beds. • Results • Average LOS for each patient = 217. 4

SIMULATION SOFTWARE TESTING EMERGENCY DEPARTMENT SIMULATOR

SIMULATION SOFTWARE TESTING EMERGENCY DEPARTMENT SIMULATOR

SIMULATION SOFTWARE TESTING EMERGENCY DEPARTMENT SIMULATOR

SIMULATION SOFTWARE TESTING EMERGENCY DEPARTMENT SIMULATOR • With One Nurse • Patient 4 was ready to be discharged by the doctor before patient 5 was ready to be seen by the same doctor. • With Two Nurses • Patient 5 was ready to be seen by the doctor before patient 4 was ready to be discharged. This is unexpected behavior.

SIMULATION SOFTWARE TESTING THE TEST METHOD • Testing the effectiveness of Metamorphic Testing • Mutation testing is used. • Defects purposely injected into the source code. • 25 out of 104 generated mutants were used in the test • Many mutants were excluded for various reasons. • Caused obvious errors and crashes • Were caught by exception handling.

SIMULATION SOFTWARE TESTING THE TEST METHOD Metamorphic testing successfully caught all mutation defects.

SIMULATION SOFTWARE TESTING OTHER METHODS • Other testing methods (instead of metamorphic) • Assertion checking.

SIMULATION SOFTWARE TESTING EMERGENCY DEPARTMENT SIMULATOR • Assertion checking • Wasn’t used here in comparison • Expected to perform less successfully. • Example • Off by 1 error for range [a, b-1] instead of [a, b] • Will never produce an out-of-range result, causing the assertion test to miss the defect. • Metamorphic testing can take on the statistical mean and variance over many run-throughs and find an inconsistency.

SIMULATION SOFTWARE TESTING CONCLUSION AND SUMMARY • Simulation Software • Discrete Event Simulation • Testing Simulation Software • Relation to healthcare software • Difficult to test • Metamorphic Testing • Mutation testing on Metamorphic testing • Assertion Checking