CS 527 Topics in Software Engineering Software Testing

CS 527 Topics in Software Engineering (Software Testing and Analysis) Darko Marinov September 7, 2010

Schedule • First few lectures to help you select projects – Previously: Intro, Re. Assert, UDITA, Pex – Today: Randoop (test generation) – Sep 9: JPF (model checking), note: journal paper – Sep 14: CHESS (multithreaded testing) – Sep 16: Regression testing, note: survey paper – Sep 21: Static (code) analysis? – Sep 23: GUI testing? – Sep 28: Analysis of code comments? – Your suggestions?

Reports, Project, Presentations • Paper reports due for every lecture – 4 items for now (fewer after proposals) – Did you get feedback for first report? • No more feedback but for projects (and non-ASCII : ) • Project proposals due September 30 – It’s hard to have a good proposal in a day/week – Start discussion with Sandro and me early • Your presentations start from October 7 – Need to choose papers more in advance – Can sign up for slots even now – Bonus given to those who sign up for first lecture

Paper Today • Feedback-directed random test generation by Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball (ICSE 2007) • Slides courtesy of Carlos Pacheco

Paper Overview • Problem (Question) – Generate unit tests (with high coverage? ) • Solution (Result) – Generate sequences of method calls – Random choice of methods and parameters – Publicly available tool for Java (Randoop) • Evaluation (Validation) – Data structures (JPF is next lecture) – Checking API contracts – Regression testing (lecture next week)

Questions for Discussion (1) • Is there a limit on the length of the sequence generated by Randoop? [ST] • Is sequence extension effective when/if different violation results? (e. g. nondeterminism) [MK] • Can this method also effective for other type of classes such as UI or business logic? [HY] • How can we combine systematic testing and random testing? [QL] • Why is BFS deemed more preferable to DFS in systematic testing? [DM]

Questions for Discussion (2) • Can we deploy symbolic execution when coverage of some methods is small to prioritize some values? [MG] • What are some other options to add repetition to the generator? [JN] • Tradeoff in random and systematic? [AL] • Other applications/domains? How do we know what coverage is good? • What are tool limitations? • What about methods that change arguments?

Questions for Discussion (3) • What is the relation between test coverage and correct code? High coverage and bugs? • What is Randoop well suited for? • What is Randoop not well suited for? • Why is Randoop not used more often? • What is the number of false positives? • Combining Randoop and Pex (or JPF or X)? – Do they find complementary bugs? • Can we parallelize Randoop? • Can we use application-specific filters?

Past Questions for Discussion (1) • Evaluation considered mostly good, but – Why evaluate on data structures? – Why not compare with more other tools? • What kind of bugs were found? – Any feedback from developers on bugs? • ISSTA 2008 paper (see optional paper) – When is this technique good/bad? – What to use to complement RANDOOP? • Why is this solution better than others?

Past Questions for Discussion (2) • What about non-primitive inputs? – Can RANDOOP generate input programs? • • What about testing business systems? How to reduce # of illegal sequences? How many seeds to use in runs? What are general pros/cons of random testing? • Is time a good stopping condition?

Sample Project Ideas • First: would Randoop by itself be a good class project (is it too big or too small? ) • Apply Randoop on some software • Extend Randoop, e. g. , directed generation – Improve effectiveness or performance • Apply random testing in another domain • Compare with more tools and techniques • Evaluate for regression testing – Assertion failures? False alarms? Repair? • Check Randoop page Project. Ideas