Software Analysis via Data Analysis Matthias F Stallmann

Software Analysis via Data Analysis Matthias F. Stallmann, 2006/04/04, DIMACS EAA: ISC planning meeting Based on joint work with: 2006/04/04 Franc Brglez Raleigh, NC, USA DIMACS EAA: ICS Xiao Yu Li Seattle, WA, USA 1

Software versus Algorithms o o Problems are NP-hard; require heuristics or (worst case) exponential algorithms. Simple algorithms must be compared with • • o cplex and other ILP solvers metaheuristics (SA, GA, “particle swarm”, “ants…”, etc. ) with lots of adjustable parameters Want “black box” comparison with (in many cases) no prior understanding of (some of) the algorithms 2006/04/04 DIMACS EAA: ICS 2

Data Presentations based on CPLEX runs for (permutations of) a single instance o Descriptive Statistics • mean / median / stdev - 600. 4 / 25. 3 / 1767. 3 o Histogram o Percent solved 2006/04/04 DIMACS EAA: ICS 3

Stretching the Truth or making it clearer? • more information here, but we need to look carefully 2006/04/04 DIMACS EAA: ICS 4

A more “normal” distribution CPLEX with different settings under the same conditions 2006/04/04 DIMACS EAA: ICS 5

uf 250. . 87: QT 2/QT 1 vs UW 2/UW 1 (1) What about random 3 -SAT instances? solvability exp. d. 16. 7/17. 2 runtime (seconds) 2006/04/04 DIMACS EAA: ICS 6

uf 250. . 87: QT 2/QT 1 vs UW 2/UW 1 (2) UW 1 performs the same as QT 1 (t-test: t = 1. 88 > 1. 97) exp. d. 16. 7/17. 2 solvability exp. d. 12. 3/12. 5 runtime (seconds) 2006/04/04 DIMACS EAA: ICS 7

uf 250. . 87: QT 2/QT 1 vs UW 2/UW 1 (3) UW 1 performs the same as QT 1 (t-test: t = 1. 88 > 1. 97) UW 2 outperforms UW 1 by a factor of 31. . . solvability exp. d. 16. 7/17. 2 exp. d. 12. 3/12. 5 exp. d. 0. 39/0. 29 runtime (seconds) 2006/04/04 DIMACS EAA: ICS 8

uf 250. . 87: QT 2/QT 1 vs UW 2/UW 1 (4) UW 1 performs the same as QT 1 (t-test: t = 1. 88 > 1. 97) UW 2 outperforms UW 1 by a factor of 31. . . QT 2 outperforms UW 2 slightly (t-test: t = 2. 24 > 1. 97) solvability exp. d. 16. 7/17. 2 exp. d. 12. 3/12. 5 0. 31/0. 28 exp. d. 0. 21/0. 19 exp. d. 0. 39/0. 29 runtime (seconds) 2006/04/04 DIMACS EAA: ICS 9

Sources of Data Distributions first, 100 random instances (SAT) median 4. 8 mean 21. 2 stdev 42. 9 Heavy tail 2006/04/04 DIMACS EAA: ICS 10

Not all instances are equal: here’s an easy one (128 permutations + original) median mean stdev 4. 4 6. 7 6. 5 Exponential 2006/04/04 DIMACS EAA: ICS 11

…and here’s a harder one median 84. 8 mean 126. 7 stdev 117. 6 Exponential 2006/04/04 DIMACS EAA: ICS 12

Another wrinkle: stochastic search first, same seed and 32 permuted instances + original number of flips median mean stdev 42484 62457 86551 slightly worse than Exponential 2006/04/04 DIMACS EAA: ICS 13

…versus 33 different seeds; same distribution? number of flips median mean stdev 36637 50185 83226 slightly worse than Exponential 2006/04/04 DIMACS EAA: ICS 14

Things get strange when the solver is not completely stochastic (e. g. B&B with stochastic search) Bi-modal: Stochastic search either finds optimum at root, or at first branch. Lower bound “finds” optimum at root. No randomness in LB method. 2006/04/04 DIMACS EAA: ICS 15

Lower bound method is extremely sensitive to input ordering Heavy tail (and one instance times out): Lower bound method finds optimum at root or in an early branch only if input order is “friendly”. 2006/04/04 DIMACS EAA: ICS 16

Another Data Analysis Application: Instance Profiling o o sao 2. b is very easy to solve (variables easy to distinguish) e 64. b is very difficult (lots of variables occur equally often) 2006/04/04 DIMACS EAA: ICS 17