NETW 703 Dr Eng Amr T AbdelHamid Protocol

NETW 703 Dr. Eng. Amr T. Abdel-Hamid Protocol Design Testing Basics Spring 2020

Program Testing or, Program Simulation Protocol Engineering Ø What is software testing? ØRunning a program ØIn order to find faults Dr. Amr Talaat Øa. k. a. defects Øa. k. a. errors Øa. k. a. flaws Øa. k. a. faults Øa. k. a. BUGS Ø This enables us to ØIncrease our confidence that the program has high quality and low risk Ø Because we can never be sure we caught all bugs

Testing Protocol Engineering Ø What isn’t software testing? ØPurely static analysis: examining a program’s source code or binary in order to find bugs, but not executing the program ØVerification & Validation: Based on building a model and checking it. Dr. Amr Talaat

Why Testing? Protocol Engineering Ø Ideally: we prove code correct, using formal mathematical techniques (with a computer, not chalk) Ø Extremely difficult: for some trivial programs (100 lines) and many small (5 K lines) programs Ø Simply not practical to prove correctness in most cases – often not even for safety or mission critical code Dr. Amr Talaat

Unit, Integration, System Testing Protocol Engineering Ø Stages of testing Ø Unit testing is the first phase, done by developers of modules Ø Integration testing combines unit tested modules and tests how they interact Ø System testing tests a whole program to make sure it meets requirements Dr. Amr Talaat Ø “Design testing” is testing prototypes or very abstract models before implementation – seldom mentioned, but when possible it can save time. Ø Model checking may be possible at this stage Ø Functional Testing tests a program from a “user’s” perspective – does it do what it should? Opposed to unit testing, which often proceeds from the perspective of other parts of the program

Faults, Errors, and Failures Protocol Engineering Ø Fault: a static flaw in a program Ø What we usually think of as “a bug” Ø Error: a bad program state that results from a fault Ø Not every fault always produces an error Ø Failure: an observable incorrect behavior of a program as a result of an error Ø Not every error ever becomes visible Dr. Amr Talaat

To Expose a Fault with a Test Protocol Engineering Ø Reachability: the test much actually reach and execute the location of the fault Ø Infection: the fault must actually corrupt the program state (produce an error) Ø Propagation: the error must persist and cause an incorrect output – a failure Dr. Amr Talaat

Example 1 Protocol Engineering Dr. Amr Talaat int find. Last (int a[], int n, int x) { // Returns index of last element in a // equal to x, or -1 if no such. // n is length of a int i; for (i = n-1; i > 0; i--) { Find if (a[i] = x) return i; } return -1; } the fault

Example 1 (cont. ) Protocol Engineering Dr. Amr Talaat int find. Last (int a[], int n, int x) { // Returns index of last element in a // equal to x, or -1 if no such. // n is length of a Here’s a test case: int i; a = {} for (i = n-1; i > 0; i--) { n=0 if (a[i] = x) x=2 return i; } Everything is ok, as return -1; we did not reach the } bug

Example 1 (cont. ) Protocol Engineering Dr. Amr Talaat int find. Last (int a[], int n, int x) { // Returns index of last element in a // equal to x, or -1 if no such. // n is length of a int i; Here’s another: for (i = n-1; i > 0; i--) { a = {3, 9, 4} if (a[i] = x) n=3 return i; x=2 } return -1; } Reaches the fault Infects state with error But no failure

Example 1 (cont. ) Protocol Engineering int find. Last (int a[], int n, int x) { // Returns index of last element in a // equal to x, or -1 if no such. // n is length of a int i; for (i = n-1; i > 0; i--) { And finally: if (a[i] = x) a = {2, 9, 4} return i; n=3 } x=2 return -1; } Reaches the fault Dr. Amr Talaat Infects state with error And fails – returns -1 instead of 0

Controllability and Observability Protocol Engineering Ø Goals for a test case: Ø Reach a fault Ø Produce an error Ø Make the error visible as a failure Dr. Amr Talaat Ø In order to make this easy the program must be controllable and observable Ø Controllability: ØHow easy it is to drive the program where we want to go Ø Observability: ØHow easy it is to tell what the program is doing

Black Box Testing Protocol Engineering Ø Black box testing Ø Treats a program or system as a black box Ø i. e. testing that does not look at source code or internal structure of the system Ø Send a program a stream of inputs, observe the outputs, decide if the system passed or failed the test Dr. Amr Talaat

Conformance Testing Protocol Engineering Ø How do we test finite state machines? Ø Let’s say we have ØKnown FSM A ØKnow all states and transitions ØUnknown FSM B (same alphabet) ØCan only perform experiments a a b d a c ØHow do we tell if A = B? Dr. Amr Talaat Ø Known as the conformance testing or equivalence testing problem

Conformance Testing Protocol Engineering Ø Exhaustive tests can be very expensive Ø In general, we cannot computationally afford to perform complete testing Ø We will always face the risk of missing errors Ø Even when we reduce our problem to the simplest model ØThe complexity of testing full equivalence to a reference model is simply too high ØExhaustion is exhausting Dr. Amr Talaat

Protocol Engineering Protocol Conformance Testing Dr. Amr Talaat Ø To confirm if an implementation conforms to its standard Ø Issue 1: preparation of conformance tests in coverage of all design aspects Ø Issue 2: time required to run test should not be unacceptably long Ø Two main limitations Ø Controllability: the IUT cannot be directly put into a desired state, usually requiring several additional state transitions Ø Observability: prevents the external tester from directly observing the state of the IUT, which is critical for a test to detect errors Ø Formal conformance testing techniques based on FSM Ø Generate a set of input sequences that will force the FSM implementation to undergo all specified transitions Ø Black box approach: only the outputs generated by the IUT (upon receipt of inputs) are observable to the external tester

Protocol Engineering Fault Model for FSM Dr. Amr Talaat Ø Output fault: the machine provides an output different from the one specified by the output function Ø Transfer fault: the machine enters a different state than that specified by the transfer function Ø Transfer faults with additional states: number of states of the system is increased by the presence of faults, additional states is used to model certain types of errors Ø Additional or missing transitions: one basic assumption is that the FSM is deterministic and completely defined (fully specified). So the faults occur when it turns out to be nondeterministic and/or incompletely (partially) specified

FIFO input queues Fault Models Protocol Engineering Dr. Amr Talaat Ø Ordering fault: FIFO ordering is not preserved, or in case of multiple input queues, some input event enters a wrong input queue. Ø Maximum length fault: the maximum length implemented is less than the one specified, or if an input event gets lost while queue is not overflow. Ø Flow control fault: errors of ordering or of loss occur, in case the number of submitted input events overflows the maximum queue length specified Ø Some other definitions & assumptions Ø Deterministic FSM: predictable behavior in a given state for a given input Ø Strongly connected: for each state pair (si, sj) there is a transition path going from si to sj , I. e. each state can be reached from any other state Ø Fully specified: form each state it has a transition for each input symbol. Otherwise partially specified Ø Minimal: the number of states of M is less than or equal to the number of states of any equivalent machine

Transition Level Approach Protocol Engineering Dr. Amr Talaat Ø The methods for protocol conformance test sequence generation Ø Produce a test sequence which checks the correctness of each transition of the FSM implementation Ø By no means exhaustive, I. e. no guarantee to exhibit correct behavior given every possible input sequence. Ø The intent is to design a test sequence which guarantees “beyond a reasonable doubt” Ø Three basic steps for checking a transition: Ø Step 1: The FSM implementation is put into state si (e. g. reset+transfer) Ø Difficulty in realizing this is due to the limited controllability of the implementation Ø Step 2: Input ak is applied and the output is checked to verify that it is ol, as expected; Ø Step 3: The new state of the FSM implementation is checked to verify that it is sj, as expected Ø Difficulty in verifying this is due to the limited observability of the implementation 19/25

Protocol Engineering T-Method: Transition Tour Method Ø For a given FSM S, a transition tour is a sequence which takes Ø Ø Ø Dr. Amr Talaat Ø the FSM S from the initial state s 0, traverses every transition at least once, and returns to the initial state s 0. Straightforward and simple scheme New state of the FSM is not checked Detects all output errors There is no guarantee that all transfer errors can be detected The problem of generating a minimum-cost test sequence using the transition tour method is equivalent to the so-called “Chinese Postman” problem in graph theory First studied by Chinese mathematician Kuan Mei-Ko (管梅谷) in 1962 20/25

T-Method EX 1 Protocol Engineering Imp. 1 Specificatio n Dr. Amr Talaat Transition Tour: b b b aaa yyxxyx Imp. 2

Ex. 2 Protocol Engineering Transition Tour Sequence: r, a, r, c, a, b, b, r, c, b, a, a r, c, a, c, b, a, c, a, Dr. Amr Talaat

Outline Protocol Engineering Ø Generate a Transition Tour Sequence for the specification below: Dr. Amr Talaat

D-Method: Distinguishing Sequences Protocol Engineering Dr. Amr Talaat Ø A sequence of inputs is a distinguishing sequence (DS) for an FSM S, if the output sequence produced by the FSM S in response to the input sequence is distinct for each initial state Ø A DS is used as a state identification sequence Ø A DS is a useful tool for achieving Step 3 in checking the new state Ø Fault detection power Ø Detects all output errors Ø Detects all transfer errors Ø Two severe drawbacks Ø In practice, very few FSMs actually possess a DS Ø Even if an FSM does have a DS, the upper bound on the length of the DS will be too large to be useful in general Ø The requirement is too strong (leading to W- & U- methods…)

Protocol Engineering D-Method Ex. 1 Dr. Amr Talaat

Ex. 1 Protocol Engineering Ø The test cases ( -sequences) are: Ø state 1: Ø r, a, b, b Ø r, b, b, b Ø state 2: Ø r, b, a, b, b Ø r, b, b n state 3: ¨ r, a, a, b, b ¨ r, a, b, b, b Test case structure corresponding to 3 -steps: preamble, tested transition, state identification Dr. Amr Talaat Transfer sequence (Preamble): the minimum cost (shortest path) input sequence taking FSM from one state to another.

Protocol Engineering Ex. 3 Dr. Amr Talaat

Ø Find DS and -sequences Protocol Engineering Ex. 4 Dr. Amr Talaat

W-Method: Characterizing Sequences Protocol Engineering Dr. Amr Talaat Ø For FSMs that do not possess a DS, W-Method defines partial DS each of which distinguishes a state si from a subset of the remaining states instead of from every state of the FSM Ø The states of the FSM are first partitioned into blocks which can be distinguished by observing the sequence of outputs produced by a sequence of inputs Ø Each block is subsequently partitioned into distinguishable subblocks, and so on, until each block consists of exactly one state Ø Main idea is to iteratively find a DS for each subset Ø To identify a state (for step 3) Ø Applying an input sequence Ø Returning to the state via a transfer sequence Ø Applying a second input sequence, and so on Ø The complete set of such input sequences for an FSM is called the characterizing set Ø Attach each CS in the set to the end of each transfer sequence

Protocol Engineering Ex. 1 Dr. Amr Talaat Ø For the input sequence Acs 1 = A, A, the response is identical for states 2 and 3 (01), but is distinct from that for states 0(00), 1(11), 4(10). Ø Another input sequence Acs 2 = B is distinct for states 2(0) and 3(1). Ø Acs 1 is required to identify states 0, 1, 4, and two input sequences Ø Acs 1 and Acs 2, along with appropriate transfer sequences, are required to identify states 2 and 3. W = {AA, B}

Ø Find W Protocol Engineering Ex 2 Dr. Amr Talaat

U-Method: Unique Input/Output Sequences Protocol Engineering Dr. Amr Talaat Ø An I/O behavior that is not exhibited by any other state of the FSM Ø Answer the question of “is the implementation currently in state x? ” Ø Advantages against DS & CS Ø Cost is never more than DS and in practice is usually much less (shorter) Ø Nearly all FSMs have UIO sequences for each state Ø DS – same for all states; UIO sequence – normally different for each state Ø To check state s by using UIO sequence of s Ø Apply input part of UIO, compare output sequence with the expected one Ø If the same, then the FSM is in the state s; otherwise, not in the state s Ø If not in state s, no information about the identity of the actual state s’

Ex. 1 Protocol Engineering Dr. Amr Talaat

Protocol Engineering Dr. Amr Talaat

Analysis Protocol Engineering Dr. Amr Talaat Ø Fault Testing Coverage Ø Fault coverage for D-, W-, and U-methods is better than of Tmethod Ø Fault coverage for D-, W-, and U-methods are the same Ø All of these four methods assume minimal, strongly connected and fully specified Mealy FSM model of protocol entities Ø On average, T-method produces the shortest sequence, Wmethod the longest. D- and U- methods generate test sequence of comparable lengths Ø T-method test sequences are able to detect output faults but not transition Ø D-, W-, and U-methods are capable of detecting all kinds of faults and give the same performance. Ø U-method attracts more and more attentions and there are several approaches based on the basic idea with some improvements

White Box Testing Protocol Engineering Ø Opens up the box! Ø (also known as glass box, clear box, or structural testing) Ø Use source code (or other structure beyond the input/output spec. ) to design test cases Ø Brings us to the idea of coverage Dr. Amr Talaat

Coverage Measures Protocol Engineering Ø In general, used to measure the quality of a test suite Ø Even in cases where the suite was designed for some other purpose (such as testing lots of different use scenarios) Ø Not always a very good measure of suite quality, but “better than nothing” Ø We “open the box” in white box testing partly in order to look at (and design tests to achieve) coverage Dr. Amr Talaat

Coverage Protocol Engineering Ø Literature of software testing is primarily concerned with various notions of coverage Ø Ammann and Offutt identify four basic kinds of coverage: Ø Graph coverage Ø Logic coverage Ø Input space partitioning Ø Syntax-based coverage Dr. Amr Talaat

Graph Coverage Protocol Engineering Dr. Amr Talaat l Cover all the nodes, edges, or paths of some graph related to the program l Examples: • Statement coverage • Branch coverage • Path coverage • Data flow coverage • Model-based testing coverage • Many more – most common kind of coverage, by far

Statement/Basic Block Coverage Protocol Engineering if (x < y) { y = 0; x = x + 1; } else { x = y; } Statement coverage: Cover every node of these graphs 1 x<y y=0 x=x+1 x >= y 2 3 x=y 4 Dr. Amr Talaat Treat as one node because if one statement executes the other must also execute (code is a basic block) if (x < y) { y = 0; x = x + 1; } 1 x<y y=0 x=x+1 x >= y 2 3

Branch Coverage Protocol Engineering if (x < y) { y = 0; x = x + 1; } else { x = y; } 1 x<y y=0 x=x+1 x >= y 2 3 Branch coverage vs. statement coverage: Same for if-then-else x=y 4 Dr. Amr Talaat But consider this if-then structure. For branch coverage can’t just cover all nodes, but must cover all edges – get to node 3 both after 2 and without executing 2! if (x < y) { y = 0; x = x + 1; } 1 x<y y=0 x=x+1 x >= y 2 3

Path Coverage Protocol Engineering if (x < y) { y = 0; x = x + 1; } else { x = y; } Dr. Amr Talaat if (x < y) { y = 0; x = x + 1; } 1 x<y y=0 x=x+1 How many paths through this code are there? Need one test case for each to get path coverage x >= y 2 3 4 x<y y=0 x=x+1 x >= y 5 x=y To get statement and branch coverage, we only need two test cases: 1 2 4 5 6 and 1 3 4 6 Path coverage needs two more: 12456 1346 1246 13456 6 In general: exponential in the number of conditional branches!

Logic Coverage Protocol Engineering What if, instead of: if (x < y) { y = 0; x = x + 1; } we have: Dr. Amr Talaat if (((a>b) || G)) && (x < y)) { y = 0; x = x + 1; } 1 ((a>b) || G)) && (x < y) y=0 x=x+1 2 ((a <= b) && !G) || (x >= y) 3 Now, branch coverage will guarantee that we cover all the edges, but does not guarantee we will do so for all the different logical reasons We want to test the logic of the guard of the if statement

Testing “for” Coverage Protocol Engineering Ø Never seek to improve coverage just for the sake of increasing coverage Ø Coverage is not the goal ØFinding failures that expose faults is the goal ØNo amount of coverage will prove that the program cannot fail Dr. Amr Talaat “Program testing can be used to show the presence of bugs, but never to show their absence!” – E. Dijkstra, Notes On Structured Programming