COTS testing Tor Stlhane Some used approaches Component

COTS testing Tor Stålhane

Some used approaches • • • Component meta-data approach. Retro-components approach. Built-in test (BIT) approach. The STECC strategy. COTS

What is Meta-data are any data related to a component that is not code. Commonly used meta-data are for instance: • State diagrams • Quality of Service information • Pseudo code and algorithms • Test logs – what has been tested? • Usage patterns – how has the component been used up till now?

Comments on Meta-data can take up a considerable amount of storage. Thus, they are • An integrated part of the component • Stored separately and have to be downloaded when needed.

Component meta-data – 1 Component Binary code Metadata Call graphs, Testing info. done by provider

Component meta-data – 2 Component functionality Metadata req Metadata server Meta DB

Assessment based on meta-data • Round trip path tests based on state diagrams • Functional tests based on algorithms or pseudo code • Tests relevance assessment based on test logs

Retrospectors A retrospector is a tool that records the testing and execution history of a component. The information is stored as meta-data. A retro-component is a software component with a retrospector.

Retro-components approach Component functionality Metadata req and test data Metadata server Meta DB

Using the retrospectors • By collecting usage info we have a better basis for defect removal and testing • For COTS component users, the retrospector will give important info on – How the component was testes – e. g. instead of a test log – How the component has been used by others. This will tell us whether we are going to use the componet in a new way => high risk

Built-In Test – BIT We will focus on BIT – RTT (Run Time Testability). We need two sets of tests: • In the component, to test that its environment behaves as expected • In the component’s clients, to test that the component implements the semantics that its clients have been developed to expect

Testing the component The test consists of the following steps: • Bring the component to the starting state for the test • Run the test • Check that the – results are as expected – final state is as expected

BIT-description for a gearbox

Selecting tests We need to consider two points: • The quality of the test – the more comprehensive the better. Unfortunately, more comprehensive => larger • The size of the test – the faster the better. Unfortunately, faster => smaller The solution is to have several sets of tests that are run at different occasions.

Test weight configuration

BIT architecture – 1 We have the following components: • The component, with one or more interfaces and implementation of functionality • BIT-RTT provides support for the testing • External tester – runs the tests needed • Handler – takes care of errors, exceptions, failsafe behavior etc. • Constructor – initialization of components such as external testers and handlers

BIT architecture – 2

BIT dead-lock testing

Disadvantages of BIT • Static nature. • Generally do not ensure that tests are conducted as required by the component user • The component provider makes some assumptions concerning the requirements of the component user, which again might be wrong or inaccurate.

What is STECC stands for “Self TEsting Cots Components”. The method has much in common with BIT. The main difference is that • BIT is static – we rerun one or more already defined tests. • STECC is dynamic – we generate new tests based on a description. We may also interact with the tester.

STECC strategy query functionality Tester Metadata Req. Metadata Test generator Server Meta DB

Benefits of STECC • The STECC addresses the needs of the component provider and the component user • The COTS provider might not want to disclose information, in particular source code • The COTS user might require this information for performing adequate and necessary testing • Existing approaches, such as BIT, do not appropriately tackle such a situation

Assessing COTS – 1 When considering a candidate component, developers need to ask three key questions: • Does the component meet the developer’s needs? • Is the quality of the component high enough? • What impact will the component have on the system? It is practical to consider the answers to these questions for several relevant scenarios

Assessing COTS – 2

Black box test reduction using Input-output Analysis • Random Testing is not complete. • To perform complete functional testing, the number of test cases can be reduced by Input-output Analysis. We can indentify I/O relationships by using static analysis or execution analysis of program.

Test reduction using Orthogonal Array Testing Used when the number of input parameters is small and the values that each of the parameters may take are clearly bounded

L 9 orthogonal array • One input at a time: (1, 1, 1, 1), (2, 1, 1, 1), …(1, 1, 1, 3). Finds ”single mode faults”, but not faults when there exist logical relations between the variables • Exhaustive testing: 34 = 81 tests • L 9 orthogonal: see table to the right. • Test strategy: o Detect and isolate all single mode errors o Detect all double mode errors (specific levels of two parameters occur together) o Multimode errors: some multimode errors can be found by L 9 orthogonal array testing

Final test set