Test case prioritization Paolo Tonella Fondazione Bruno Kessler

Test case prioritization Paolo Tonella Fondazione Bruno Kessler tonella@fbk. eu

Motivation The order in which test cases are executed affects: • The rate of fault detection: good orderings reveal faults earlier than bad ones, or (alternatively) they – reveal high risk faults earlier; – reveal faults related to critical code sections earlier; – give satisfactory confidence in the system’s reliability earlier. • The rate of code coverage: good orderings meet the required coverage level earlier than bad ones.

The need for a heuristic approach The faults revealed by a test case are unknown until the test case is executed and its output is evaluated against the oracle. Heuristics are required to estimate the ability of a test case to reveal faults.

Main prioritization heuristics Available heuristics take advantage of execution information, assuming this can be obtained at a relatively low cost (if the output is not validated) or can be estimated from previous runs: • Coverage-based prioritization. Coverage • Mutation-based prioritization. Mutation

Coverage-based heuristics • Total statement coverage: test cases are prioritized by number of statements covered. • Additional statement coverage: test cases are prioritized by incremental number of statements covered. • Total branch coverage: test cases are prioritized by number of branches covered. • Additional branch coverage: test cases are prioritized by incremental number of branches covered.

Statement coverage Statement Test case 1 Test case 2 Test case 3 1 X X X 2 X X X 3 X X 4 X 5 X 6 X 7 X X 8 X X 9 X X Total statement coverage: Additional statement coverage: (TC 3, TC 1, TC 2) (TC 3, TC 2, TC 1)

Mutation-based heuristics • Total Fault-Exposing-Potential (FEP): test cases are prioritized by FEP. • Additional Fault-Exposing-Potential (FEP): test Underlying assumption: very lowconfidence execution cost. cases are prioritized by incremental ΔC. A mutant of a statement is generated (automatically) by randomly changing the statement (e. g. , “x++” → “x--”). A mutant is killed by a test case if the original and the mutated program give different outputs when the test case is executed.

Mutation-based heuristics • Additional Fault-Exposing-Potential (FEP): test cases are prioritized by incremental confidence ΔC. C estimates the confidence in the correctness of a statement. Initially, C is zero for every statement. When the test case with highest ΔC is selected, the confidence is updated as follows:

FEP-based prioritization Statement Test case 1 Test case 2 Test case 3 1 0. 4 0. 5 0. 3 2 0. 5 0. 9 0. 4 3 0. 01 0. 4 4 1. 0 5 0. 4 6 0. 1 7 0. 5 0. 2 8 0. 6 0. 3 9 0. 3 0. 1 Total FEP 2. 3 Total FEP: 2. 41 (TC 2, TC 1, TC 3) 2. 2

ΔC-based prioritization Statement C Test case 1 Test case 2 Test case 3 1 0. 0 0. 4 0. 5 0. 3 2 0. 0 0. 5 0. 9 0. 4 3 0. 01 0. 4 4 0. 0 1. 0 5 0. 0 0. 4 6 0. 0 0. 1 7 0. 0 0. 5 0. 2 8 0. 0 0. 6 0. 3 9 0. 0 0. 3 0. 1 ΔC 2. 3 2. 41 ΔC: (TC 2) 2. 2

ΔC-based prioritization Statement C Test case 1 Test case 2 Test case 3 1 0. 5 0. 4 0. 5 0. 3 2 0. 9 0. 5 0. 9 0. 4 3 0. 01 0. 4 4 1. 0 5 0. 0 0. 4 6 0. 0 0. 1 7 0. 0 0. 5 0. 2 8 0. 0 0. 6 0. 3 9 0. 0 0. 3 0. 1 ΔC 2. 3 2. 41 2. 2

ΔC-based prioritization Statement C Test case 1 Test case 3 1 0. 5 0. 2 0. 15 2 0. 9 0. 05 0. 04 3 0. 01 4 1. 0 5 0. 0 0. 4 6 0. 0 0. 1 7 0. 0 0. 5 0. 2 8 0. 0 0. 6 0. 3 9 0. 0 0. 3 0. 1 1. 65 1. 69 ΔC 0. 4 ΔC: (TC 2, TC 3)

ΔC-based prioritization Statement C Test case 1 Test case 3 1 0. 65 0. 2 0. 15 2 0. 94 0. 05 0. 04 3 0. 41 4 1. 0 5 0. 4 6 0. 1 7 0. 2 0. 5 0. 2 8 0. 3 0. 6 0. 3 9 0. 1 0. 3 0. 1 1. 65 1. 69 ΔC 0. 4

ΔC-based prioritization Statement C Test case 1 1 0. 65 0. 14 2 0. 94 0. 03 3 0. 41 4 1. 0 5 0. 4 6 0. 1 7 0. 2 0. 4 8 0. 3 0. 42 9 0. 1 0. 27 ΔC 1. 26 ΔC: (TC 2, TC 3, TC 1)

Empirical validation APFD: weighted Average of the Percentage of Faults Detected. test 100% fault A X X B X X C X X X X D E 10 faults 1 2 3 4 5 6 7 8 9 X APFD = 50% X X X A B C D Prioritized test cases E

Empirical validation APFD: weighted Average of the Percentage of Faults Detected. test 100% fault A X X B X X C X X X X D E 10 faults 1 2 3 4 5 6 7 8 9 APFD = 84% X X C E B A Prioritized test cases D

Experimental results • All the heuristics outperform the untreated or randomly prioritized test suites. • In almost every case, additional FEP prioritization outperforms coverage-based prioritization, but with only a marginal gain in APFD. – Given their expense, FEP-based prioritization may not be as cost effective as coverage based techniques More details in: Gregg Rothermel, Roland Untch, Chengyun Chu, and Mary Jean Harrold, Test Case Prioritization. IEEE Transactions on Software Engineering, vol. 27, no. 10, pp. 929 -948, October, 2001.

Taking advantage of user knowledge Pair-wise comparisons Intensity of importance TCi < TCj 1 3 5 7 9 (TCi, TCj) AHP Prioritized TS 1: TCi 1 2: TCi 2 n: TCin TS = {TC 1, …, TCn} AHP (Analytic Hierarchy Process) determines the target ranking of the test cases in TS from user input (pair-wise comparisons).

Analytic Hierarchy Process 1 Equal 3 Moderate 5 Strong 7 Very strong 9 Extreme TC 1 TC 2 TC 3 TC 4 A= TC 1 TC 2 TC 3 TC 4 1/1 1/3 5/1 9/1 3/1 1/3 1/5 3/1 1/5 1/9 3/1 5/1 1/1 N * (N-1) / 2 pair-wise comparisons elicited vmax A vmax = λmax vmax Principal eigenvector TC 1 TC 2 TC 3 TC 4 0. 289 0. 544 0. 634 0. 466 λmax = 7. 248 TC 3 TC 2 TC 4 TC 1