Covrig A Framework for the Analysis of Code

Covrig: A Framework for the Analysis of Code, Test, and Coverage Evolution in Real Software Paul Marinescu, Petr Hosek, Cristian Cadar Imperial College London 1

Goal • Answer questions about software evolution – Code quality – Test quality – Development model – Testing improvement opportunities …using software development historical data 2

Target Audience • Researchers – Hypothesis validation (e. g. are software patches poorly tested? ) • Programmers/Project Managers – Assess development quality 3

Software Metrics • Static – Measured by parsing the software artifacts • Dynamic – Require running the evolving software – More challenging – Very few studies 4

Example questions 1. 2. 3. 4. 5. 6. 7. Do executable and test code evolve in sync? How many patches touch only code/test/none/both? What is the distribution of patch sizes? How spread out is each patch through the code? Is test suite execution deterministic? How does the overall coverage evolve? What is the distribution of patch coverage across revisions? 8. What is the latent patch coverage? 9. Are bug fixes better covered than other patches? 10. Is the coverage of buggy code less than average? 5

Data mining infrastructure Empirical case study 6

Covrig Overview 2 1 3 7

Docker Containers • Lightweight, OS-level virtualization – Guest shares kernel with host – Namespace isolation • • PID Network IPC Filesystem – Resource limiting • cgroups + Linux Containers + Docker 8

Docker Containers Features • • • Isolation Consistency Reproducibility Easy cloud deployment Performance 9

Covrig 10

Static Metric Granularity Test size Lines Executable code size Lines Patch executable size Dynamic Overall coverage Patch coverage Hunks Files Lines Branches Latent patch coverage Lines Test result FAIL/PASS 11

Challenges Evolving dependencies Evolving containers Custom compile flags (-Wno-error) 12

Challenges Branching development structure Consider only the ‘main’ branch Alice Bob r 1 r 2 r 3 r 4 m 1 r 2+r 4 13

Challenges Revisions that fail to compile Accumulate until reaching a compilable revision r 1 r 2 r 3 r 1+r 2+r 3 14

Data mining infrastructure Empirical case study 15

Case Study Subjects App ELOC Tests Lang Binutils 27, 029 Deja. Gnu Git Lighttpd Memcached Redis Zero. MQ Period (mo) LOC 5, 186 35 79, 760 C/shell 108, 464 5 23, 884 Python 2, 440 36 4, 426 C/Perl 4, 605 47 7, 589 6 3, 460 17 18, 203 Tcl 7, 276 C++ 1500 revisions and 12 years of development in total 16

Patch type 17

Is test suite execution deterministic? FAIL/PASS determinism Nondeterministic Revisions Binutils Git Lighttpd Memcached Redis Zero. MQ 0 1 1 21 16 32 18

Is test suite execution deterministic? Coverage determinism Nondeterministic Lines (median) Binutils Git Lighttpd Memcached Redis Zero. MQ 0 13 10 8. 5 23 27 19

Test Suite Nondeterminism Causes • Bugs – Race conditions – Hardcoded wall clock timeouts – Incorrect resource consumption expectations • Random test data • Benign race conditions 20

Are patches properly tested? Sometimes 21

Patch coverage 22

Patch coverage 0% 0% 0% 23

Does covered code contain fewer bugs that not covered code? Not really 24

Does covered code contain fewer bugs that not covered code? Patch Coverage (median) Buggy Memcached Patches Fully Covered All Buggy All 100% 89% 67% 45% Redis 94% 0% 47% 25% Zero. MQ 71% 76% 37% 33% 85 total bugs 25

Conclusions Dynamic software metrics mining Case study on 6 systems/1500 revisions/12 years of development Open source extensible infrastructure http: //srg. doc. ic. ac. uk/projects/covrig/ 26