When to Test Less Presented by Lan Guo

Introduction (1) • Methods of software testing: functional, coverage, and user-oriented • Phases of software testing: unit, integration, product, or regression • White-box (coverage) testing uses the structure of the software to measure the quality of testing.

Introduction (2) – Statement coverage – Decision coverage – Data flow coverage – Mutation testing • Black-box testing aims at the specific features of the software. – Generate an operational profile, which is a list of occurrence probabilities of each element in the input domain of the program

Introduction (3) – The operational profile is dependent on the input from the customer. • Difficulties in black-box testing – new software, new feature, no customer base – inadequate test set – coarse features, interacting features – multiple and unknown user groups

Introduction (4) • There is a saturation effect associated with ALL testing methods. Reliability mutation decision functional Data flow Testing effort

Introduction (5) • Elaborate and expensive testing does not tend to find more faults than inexpensive manual or simple automatic testing schemes.

Theory of Testing (1) • In black-box testing, the probability to find a fault in N random tests is y = 1 - (1 - x)N x: the probability of finding a fault for a random input • Black-box testing can be expensive. If x = 0. 0001, 46, 000 tests are needed to be 99% sure to reveal a fault.

Theory of Testing (2) • Formal-methods testing is even more expensive compared to black-box testing. • While-box testing only slightly reduces the number of tests required by black-box testing. In addition, the partition cost is very high.

Reality of Testing (1) • Software-in-the-small projects produce software without rigorous testing. • The software does not crash more often. • A program’s shape is the shape of the pathways within the program. The more tangled the pathways, the more complex the program shape.

Reality of Testing (2) • The simple shapes are common, which is demonstrated by du-pathways, control-flow diagrams and saturation effect. • Saturation effect is consistent with programs containing either many portions with simple shapes--so the portions are easily reached-- or many portions that are so twisted in shape that we will never reach them.

Average Shape of Software • Right shape: the program is either too complex to test or too simple to test elaborately. • In either case, there is no point to conduct a lengthy and expensive test. • Most programs are indeed the right shape for simple testing. Proof: A mathematical model of building a pathway across a program from randomly selected inputs to some randomly selected state.

Assumption 1 • Programs are networks connecting system concepts. (NAYO) • Optimizing compiler builds a network (control and data-flow graphs) from the code. – An execution trace of a procedural program shows what statements are executed, which subroutines are called, and in what order. – A verification condition generator creates a proof tree (execution tree), including preconditions and postconditions

Assumption 2 • A fault explanation tree is a tree whose leaves are inputs and whose root is a fault. • If we arrive at a state which should not happen in this tree, it is a bug in our system.

Assumption 3 • Testing is a process of trying to generate a fault explanation tree from the program network. • If we can’t generate such tree, we are confident that there is no fault in our program.

Assumption 4 • NAYO tree (explanation tree) N: no edge A: and node Y: yes edge O: or node • No edge only connects or nodes • Yes edge connects both and nodes and or nodes

Assumption 5 • Testing is the process of extracting NAYO trees (the explanations) from NAYO networks (the program).

Mathematics Equations (1) • Given in inputs to a NAYO network with V nodes, the probability to find a fault is: xo = in/V (A) • The probability of reaching an and node with andp parents is the probability of reaching all its parents: xand = xiandp (B)

Mathematics Equations (2) • The probability of reaching an or node with orp parents is the probability of not missing all of its parents: xor = 1 - (1 - xo)orp (C) • If the ratio of and nodes in a NAYO tree is andf, then the ratio of or nodes in the same tree is orf = 1 - andf. The probability to reach some random node xj is: xj = andf * xand + orf * xor (D)

Mathematics Equations (3) • Rearrange Equation A to isolate the number of tests required to be 99% sure of finding a fault with probability xj is: y = 0. 99 = 1 - ((1 - xj)N) Therefore, N = log(1 - 0. 99) log(1 -xj)

Implementation • We can approximately categorize the tree’s shape by the number of tests N required to reach that tree’s root to find a fault. • Results from the Simulation Runs Classification Threshold Simple 0 < N < 102 Moderate 102 <=N<104 Hard 104<=N<106 N >= 106 Overly complex Percent 36 19 25 20

Conclusion • Black-box mathematics is blind to the internal structures of the program. If we include internal structure in the analysis, the result is more optimistic. • The average case analysis of the shape of programs indicates that simple and RANDOM testing can be very valuable.

Related Research • Simple and random testing can probe nondeterminate systems. • Limited testing is adequate for knowledge based systems.

Potential Future Research • Extending static code analysis tools (optimizer and verification condition generator) to extract the parameters in the mathematical model from real-world programs, in order to monitor the change of program’s shape. • Offering design guideline to avoid hardshaped programs.

References (1) • T. Menzies and B. Cukic, “When to Test Less”, IEEE Software, September/October 2000, pp. 107112 • D. Hamlet and R. Taylor, “Partition Testing Does not Inspire Confidence”, IEEE Trans. Software Eng. , Vol. 16, No. 12, Dec. 1990, pp. 1402 -1411 • J. Horgan and A. Mathur, “Software Testing and Reliability”, The Handbook of Software Reliability Eng. , M. R. Lyu, ed. , Mc. Graw-Hill, New York, 1996, pp. 531 -565.

References (2) • T. Menzies and B. Cukic, “Adequacy of limited testing for knowledge based systems”, International Journal on Artificial Intelligence Tools, Vol. 9, No. 1, 2000, pp. 153 -172 • T. Menzies, B. Cukic, H. Singh, and J. Powell, “Testing Nondeterminate Systems”