Balancing TradeOffs in TestSuite Reduction August Shi Alex

Balancing Trade-Offs in Test-Suite Reduction August Shi, Alex Gyori, Milos Gligoric, Andrey Zaytsev, and Darko Marinov 11/18/2014 FSE 22 Hong Kong NSF Grant Nos. CNS-0958199, CCF-1012759, CCF-1439957

Testing Can Be Slow test 0 test 1 test 2 test 3 … test. N-1 test. N Code Under Test 2

Speed Up by Removing Tests test 0 test 1 test 2 test 3 … test. N-1 test. N Code Under Test 3

Test-Suite Reduction test 1 test 3 … test. N Code Under Test Reduced test suite has fewer tests than full test suite but representative of full test suite on this code version 4

Test-Suite Reduction and Changes test 1 test 3 … test. N Versioni changes Versioni+1 test 3 … test. N 5

Test-Suite Reduction and Changes test 1 test 3 … test. N Versioni changes Versioni+1 test 3 … test. N Is reduced test suite still representative of full test suite on future versions? 6

How does evolution affect reduced test suites?

Test-Suite Reduction S 1 T 1 X S 2 S 3 S 4 X X T 3 X X X T 5 X S 5 X T 2 T 4 T = Tests S = Statements X 8

Test-Suite Reduction S 1 T 1 X S 2 S 3 S 4 X X T 3 X X X T 5 X S 5 X T 2 T 4 T = Tests S = Statements X Reduced Test Suite R 1 = {T 3, T 5} 9

Test-Suite Reduction S 1 T 1 X S 2 S 3 S 4 X X T 3 X X X T 5 X S 5 X T 2 T 4 T = Tests S = Statements X Reduced Test Suite R 1 = {T 3, T 5} Statement Adequate Reduction (SAR) 10

Test-Suite Reduction S 1 T 1 X S 2 S 3 S 4 X X T 3 X X X T 5 X S 5 X T 2 T 4 T = Tests S = Statements X 11

Test-Suite Reduction S 1 T 1 X S 2 S 3 S 4 X X T 3 X X X T 5 X S 5 X T 2 T 4 T = Tests S = Statements X 12

Test-Suite Reduction S 1 T 1 X S 2 S 3 S 4 X X T 3 X X X T 5 X S 5 X T 2 T 4 T = Tests S = Statements X 13

Test-Suite Reduction S 1 T 1 X S 2 S 3 S 4 X X T 3 X X X T 5 X S 5 X T 2 T 4 T = Tests S = Statements X 14

T = Tests S = Statements M = Mutants Test-Suite Reduction S 1 T 1 X S 2 S 3 S 4 X X X T 3 X X X T 5 X M 1 M 2 M 3 M 4 X T 2 T 4 S 5 M 6 X X X M 5 X X X X 15

T = Tests S = Statements M = Mutants Test-Suite Reduction S 1 T 1 X S 2 S 3 S 4 X X X T 3 X X X T 5 X M 1 M 2 M 3 M 4 X T 2 T 4 S 5 M 6 X X X M 5 X X X X 16

T = Tests S = Statements M = Mutants Test-Suite Reduction S 1 T 1 X S 2 S 3 S 4 X X X T 3 X X X T 5 X M 1 M 2 M 3 M 4 X T 2 T 4 S 5 M 6 X X X M 5 X X X X 17

T = Tests S = Statements M = Mutants Test-Suite Reduction S 1 T 1 X S 2 S 3 S 4 X X X T 3 X X X T 5 X M 1 M 2 M 3 M 4 X T 2 T 4 S 5 M 6 X X X M 5 X X X X 18

T = Tests S = Statements M = Mutants Test-Suite Reduction S 1 T 1 X S 2 S 3 S 4 X X X T 3 X X X T 5 X M 1 M 2 M 3 M 4 X T 2 T 4 S 5 M 6 X X X M 5 X X X X Reduced Test Suite R 2 = {T 1, T 4, T 5} 19

T = Tests S = Statements M = Mutants Test-Suite Reduction S 1 T 1 X S 2 S 3 S 4 X X X T 3 X X X T 5 X M 1 M 2 M 3 M 4 X T 2 T 4 S 5 M 6 X X X M 5 X X X X Reduced Test Suite R 2 = {T 1, T 4, T 5} Mutant Adequate Reduction (MAR) 20

T = Tests S = Statements M = Mutants Test-Suite Reduction S 1 T 1 X S 2 S 3 S 4 X X X T 3 X X X T 5 X M 1 M 2 M 3 M 4 X T 2 T 4 S 5 M 6 X X X M 5 X X X X 21

T = Tests S = Statements M = Mutants Test-Suite Reduction S 1 T 1 X S 2 S 3 S 4 X X X T 3 X X X T 5 X M 1 M 2 M 3 M 4 X T 2 T 4 S 5 M 6 X X X M 5 X X X X 22

T = Tests S = Statements M = Mutants Test-Suite Reduction S 1 T 1 X S 2 S 3 S 4 X X X T 3 X X X T 5 X M 1 M 2 M 3 M 4 X T 2 T 4 S 5 M 6 X X X M 5 X X X X 23

T = Tests S = Statements M = Mutants Test-Suite Reduction S 1 T 1 X S 2 S 3 S 4 X X X T 3 X X X T 5 X M 1 M 2 M 3 M 4 X T 2 T 4 S 5 M 6 X X X M 5 X X X X 24

Choosing a Test Suite Siz. Red Stmt. Loss Mut. Loss R 1 60% 0% 33% R 2 40% 0% 0% T 0% 0% 0% 25

Choosing a Test Suite Siz. Red Stmt. Loss Mut. Loss R 1 60% 0% 33% R 2 40% 0% 0% T 0% 0% 0% Single Version Evaluation 26

How does evolution affect reduced test suites? (Example)

How does evolution affect reduced test suites? T 1 S 2 X X S 3 S 4 T 2 X X T 3 X X X T 4 T 5 X S 5 X Versioni 28

How does evolution affect reduced test suites? T 1 S 2 X X S 3 S 4 S 1’ S 5 T 1 X X T 3 X X T 4 T 5 X X Versioni T 5 S 3’ S 4’ S 5’ X T 2 S 2’ S 6’ X X X X X Versioni+1 29

How does evolution affect reduced test suites? T 1 S 2 X X S 3 S 4 S 1’ S 5 T 1 X X T 3 X X T 4 T 5 X X Versioni T 5 S 3’ S 4’ S 5’ X T 2 S 2’ S 6’ X X X X X Versioni+1 30

How does evolution affect reduced test suites? T 1 S 2 X X S 3 S 4 S 1’ S 5 T 1 X X T 3 X X T 4 T 5 X X Versioni T 5 S 3’ S 4’ S 5’ X T 2 S 2’ S 6’ X X X X X Versioni+1 31

How does evolution affect reduced test suites? T 1 S 2 X X S 3 S 4 S 1’ S 5 T 1 X X T 3 X X T 4 T 5 X X Versioni T 5 S 3’ S 4’ S 5’ X T 2 S 2’ S 6’ X X X X X Versioni+1 32

How does evolution affect reduced test suites? T 1 S 2 X X S 3 S 4 S 1’ S 5 T 1 X X T 3 X X T 4 T 5 X X Versioni T 5 S 3’ S 4’ S 5’ X T 2 S 2’ S 6’ X X X X X Versioni+1 How do we measure change in loss? 33

Evolution-Aware Metrics T 1 S 2 X X S 3 S 4 T 2 X X T 3 X X X T 4 T 5 X S 5 X Versioni 34

Evolution-Aware Metrics T 1 S 2 X X S 3 S 4 T 2 X X T 3 X X X T 4 T 5 X S 5 X Versioni 35

Evolution-Aware Metrics T 1 S 2 X X S 3 S 4 S 1’ S 5 T 1 X X T 3 X X T 4 T 5 X X Versioni T 5 S 3’ S 4’ S 5’ X T 2 S 2’ S 6’ X X X X X Versioni+1 36

Evolution-Aware Metrics T 1 S 2 X X S 3 S 4 S 1’ S 5 T 1 X X T 3 X X T 4 T 5 X X Versioni T 5 S 3’ S 4’ S 5’ X T 2 S 2’ S 6’ X X X X X Versioni+1 37

Evolution-Aware Metrics T 1 S 2 X X S 3 S 4 S 1’ S 5 T 1 X X T 3 X X T 4 T 5 X X Versioni T 5 S 3’ S 4’ S 5’ X T 2 S 2’ S 6’ X X X X X Versioni+1 38

Evolution-Aware Metrics T 1 S 2 X X S 3 S 4 S 1’ S 5 T 1 X X T 3 X X T 4 T 5 X X Versioni T 5 S 3’ S 4’ S 5’ X T 2 S 2’ S 6’ X X X X X Versioni+1 39

Evolution-Aware Metrics T 1 S 2 X X S 3 S 4 S 1’ S 5 T 1 X X T 3 X X T 4 T 5 X X Versioni T 5 S 3’ S 4’ S 5’ X T 2 S 2’ S 6’ X X X X X Versioni+1 40

Evolution-Aware Metrics T 1 S 2 X X S 3 S 4 S 1’ S 5 T 1 X X T 3 X X T 4 T 5 X X T 5 S 4’ S 5’ S 6’ X X X X X Versioni+1 Versioni S 3’ X T 2 S 2’ 41

Evolution-Aware Metrics T 1 S 2 X X S 3 S 4 S 1’ S 5 T 1 X X T 3 X X T 4 T 5 X X T 5 S 4’ S 5’ S 6’ X X X X X Versioni+1 Versioni S 3’ X T 2 S 2’ 42

Evolution-Aware Metrics M 1 T 1 M 2 M 4 X T 2 X T 4 M 5 M 6 X T 3 T 5 M 3 X M 1’ T 1 X X T 3 X X T 4 X X Versioni T 5 M 3’ M 4’ X T 2 X M 2’ M 5’ M 6’ X X X X Versioni+1 43

Evolution-Aware Metrics M 1 T 1 M 2 M 4 X T 2 X T 4 M 5 M 6 X T 3 T 5 M 3 X M 1’ T 1 X X T 3 X X T 4 X X Versioni T 5 M 3’ M 4’ X T 2 X M 2’ M 5’ M 6’ X X X X Versioni+1 44

Evolution-Aware Metrics M 1 T 1 M 2 M 4 X T 2 X T 4 M 5 M 6 X T 3 T 5 M 3 X M 1’ T 1 X X T 3 X X T 4 X X Versioni T 5 M 3’ M 4’ X T 2 X M 2’ M 5’ M 6’ X X X X Versioni+1 45

Evolution-Aware Metrics M 1 T 1 M 2 M 4 X T 2 X T 4 M 5 M 6 X T 3 T 5 M 3 X M 1’ T 1 X X T 3 X X T 4 X X T 5 M 4’ M 5’ M 6’ X X X X Versioni+1 Versioni M 3’ X T 2 X M 2’ 46

Evolution-Aware Metrics Versioni T 1 S 2 X X S 3 Versioni+1 S 4 S 5 S 1’ T 1 T 2 X X T 3 X X T 4 T 5 X T 5 S 3’ S 4’ X T 2 X S 2’ S 5’ X X Versioni+2 X S 6’ S 1’’ T 1 S 2’’ S 3’’ X T 2 S 4’’ X X X T 3 X X T 4 X X T 5 X 47

Evolution-Aware Metrics Versioni T 1 S 2 X X S 3 Versioni+1 S 4 S 5 S 1’ T 1 T 2 X X T 3 X X T 4 T 5 X T 5 S 3’ S 4’ X T 2 X S 2’ S 5’ X X Versioni+2 X S 6’ S 1’’ T 1 S 2’’ S 3’’ X T 2 S 4’’ X X X T 3 X X T 4 X X T 5 X 48

Evolution-Aware Metrics Versioni T 1 S 2 X X S 4 S 5 S 1’ T 1 T 2 X X T 3 X X T 4 T 5 S 3 Versioni+1 X T 5 S 3’ S 4’ X T 2 X S 2’ S 5’ X X Versioni+2 X S 6’ S 1’’ T 1 S 2’’ S 3’’ X T 2 S 4’’ X X X T 3 X X T 4 X X T 5 X Compute REC 1 49

Evolution-Aware Metrics Versioni T 1 S 2 X X S 4 S 5 S 1’ T 1 T 2 X X T 3 X X T 4 T 5 S 3 Versioni+1 X T 5 S 3’ S 4’ X T 2 X S 2’ Versioni+2 S 5’ X X S 6’ S 1’’ T 1 S 2’’ S 3’’ X T 2 S 4’’ X X X T 3 X X T 4 X X X T 5 X Compute REC 2 50

Evolution-Aware Metrics Versioni T 1 S 2 X X S 3 Versioni+1 S 4 S 5 S 1’ T 1 T 2 X X T 3 X X T 4 T 5 X T 5 X S 5’ S 6’ X T 3 X X S 4’’ X X T 3 X X T 4 X X S 3’’ S 4’’ S 3’ S 4’ S 5’ X X S 3’’ X T 5 S 2’’ X T 2 X T 4 S 1’’ T 1 X T 2 T 5 S 4’ X X S 1’ T 1 S 3’ X T 2 X S 2’ Versioni+2 X S 6’ X S 1’’ T 1 S 2’’ X T 2 X X X T 3 X X T 4 X X T 5 X 51

Evolution-Aware Metrics Versioni T 1 S 2 X X S 3 Versioni+1 S 4 S 5 S 1’ T 1 T 2 X X T 3 X X T 4 T 5 X T 5 X S 5’ S 6’ X T 3 X X S 4’’ X X T 3 X X T 4 X X S 3’’ S 4’’ S 3’ S 4’ S 5’ X X S 3’’ X T 5 S 2’’ X T 2 X T 4 S 1’’ T 1 X T 2 T 5 S 4’ X X S 1’ T 1 S 3’ X T 2 X S 2’ Versioni+2 X S 6’ X S 1’’ T 1 S 2’’ X T 2 X X X T 3 X X T 4 X X T 5 X Compute REC 1 52

How does evolution affect reduced test suites? (Evaluation)

Evaluation: Implementation • Use PIT to collect statement coverage and mutants killed by tests on each version • Use Greedy heuristic to perform test-suite reduction • Use Statement Adequate Reduction and Mutant Adequate Reduction 54

Evaluation: Projects 55

Evaluation: Projects 56

Evaluation: Projects. Distance between versions = 30 Git commits 57

Evaluation: Projects 58

Evaluation: Projects 59

Evaluation: Projects 60

Evaluation: Projects 61

Statement Adequate Reduction (SAR) 62

Statement Adequate Reduction (SAR) Median REC for SAR is around 0, does not drop much 63

Mutant Adequate Reduction (MAR) 64

Mutant Adequate Reduction (MAR) Median REC for MAR is around 0, does not drop much 65

How does evolution affect reduced test suites? (Answer)

The quality of the reduced test suite relative to the quality of the full test suite remains about the same during evolution

We further evaluated… • Different test-suite reduction algorithms • Four other algorithms in addition to Greedy • Inadequate test-suite reduction • Reduction that does not preserve all statements covered or mutants killed 68

Threats to Validity • Reduced test suites evaluated with same metrics (statement coverage and killed mutants) used for reduction • Tests tracked by name • 30 commits between versions 69

Related Work • Improving test-suite reduction • Hao et al. [ICSE 2012] • Lin and Huang [IST 2009] • Yoo and Harman [ISSTA 2007] • Jeffrey and Gupta [TSE 2007] • Black et al. [ICSE 2004] • Offutt et al. [ICTCS 1995] • Effect of software evolution on coverage • Elbaum et al. [ICSM 2001] 70

Conclusions • Q: How does software evolution affect test-suite reduction? 71

Conclusions • Q: How does software evolution affect test-suite reduction? • Introduced new evolution-aware metrics (REC) • Performed the largest evaluation of test-suite reduction • Different metrics/algorithms, inadequate test-suite reduction 72

Conclusions • Q: How does software evolution affect test-suite reduction? • Introduced new evolution-aware metrics (REC) • Performed the largest evaluation of test-suite reduction • Different metrics/algorithms, inadequate test-suite reduction • A: Reduced test suites do not reduce in quality relative to the full test suite over time • If reduced test suite is acceptable for this version, then it is likely acceptable in future versions 73

Conclusions • Q: How does software evolution affect test-suite reduction? • Introduced new evolution-aware metrics (REC) • Performed the largest evaluation of test-suite reduction • Different metrics/algorithms, inadequate test-suite reduction • A: Reduced test suites do not reduce in quality relative to the full test suite over time • If reduced test suite is acceptable for this version, then it is likely acceptable in future versions August Shi: awshi 2@illinois. edu http: //mir. cs. illinois. edu/evolred/ 74

BACKUP

Project Filters • Uses Maven build system • Has > 100 commits in history • HEAD at time of experiments could be successfully run through PIT • Has > 4 versions working through PIT in history 76

Evaluation: Basic Test-Suite Reduction 77

Different Algorithms? T 1 S 2 X X S 3 S 4 T 2 X X T 3 X X X T 4 T 5 X S 5 X Versioni 78

Different Algorithms? T 1 S 2 X X S 3 S 4 T 2 X X T 3 X X X T 4 T 5 X S 5 X Versioni 79

Different Algorithms? T 1 S 2 X X S 3 S 4 S 1 S 5 T 1 X X T 3 X X T 4 T 5 X X Versioni T 5 S 3 S 4 X T 2 S 2 S 5 S 6 X X X X X Versionj 80

Different Algorithms? T 1 S 2 X X S 3 S 4 S 1 S 5 T 1 X X T 3 X X T 4 T 5 X X T 5 S 4 S 5 S 6 X X X X X Versionj Versioni • Greedy • GRE • GE S 3 X T 2 S 2 • HGS • ILP 81

Evaluation: Other Algorithms • Other 4 algorithms produce very similar results compared to Greedy • Difference in size reduction at most 5. 26 pp across all algorithms • Difference in Mut. Loss for SAR algorithms at most 7. 15 pp • Difference in Stmt. Loss for MAR algorithms at most 4. 15 pp • Difference in RECd for any distance d at most 0. 33 pp for Mut. Loss, 0. 67 pp for Stmt. Loss 82

Inadequate Reduction? T 1 S 2 X X S 3 S 4 T 2 X X T 3 X X X T 4 T 5 X S 5 X Versioni 83

Inadequate Reduction? T 1 S 2 X X S 3 S 4 T 2 X X T 3 X X X T 4 T 5 X S 5 X Versioni 84

Inadequate Reduction? T 1 S 2 X X S 3 S 4 S 1 S 5 T 1 X X T 3 X X T 4 T 5 X X Versioni T 5 S 3 S 4 X T 2 S 2 S 5 S 6 X X X X X Versionj 85

Evaluation: Inadequate Reduction Statement Inadequate Reduction (SIR) Mutant Inadequate Reduction (MIR) 86

Evaluation: SIR Evolution Results for Greedy 87

Evaluation: MIR Evolution Results for Greedy 88

Code Change 89