Simplifying and Isolating FailureInducing Input Andreas Zeller and

Simplifying and Isolating Failure-Inducing Input Andreas Zeller and Ralf Hildebrandt IEEE Transactions on Software Engineering (TSE) 2002

The Targeted Problem Test Case (Input) Program Under Test Failure … Ugh Which circumstances of the test case are responsible for this failure? … help the debugger!!

Overall Strategy Diff? Test Case (Input) Program Under Test Minimal Test Case Passing Test Case Program Under Test Failure … Ugh Pass! Simplify Failing test case AND Isolate Difference of Pass/Failing Tests

Basic ddmin Algorithm - Simplify INPUT: Failing test case Simplify failing test and run simplified test Until minimal test case, where deleting 1 more input entity causes failure to vanish. OUTPUT: Only relevant part of failing test case repetitive execution of successively smaller test cases Key idea – execution behavior is determined by a number of circumstances… input = set of circumstances

A test case = circumstances set Note: Some of these slides are taken from Zeller’s ISSTA 2000 talk

Input differences = set of changes

Representing Test and Outcomes

Simplifying to Minimal Tests

Example of Minimizing Process Partition into delta 1 and delta 2. If delta 1 fails, delta 1 is a smaller test case. Done If delta 1 does not fail, delta 2 fails, delta 2 is smaller test case. Done. Both tests pass or unresolved. Need to test larger subset of original.

Observation and Strategy Larger subsets of original higher chance of test failing Smaller subsets of original faster progression in case test fails, but less chance that test fails Strategy: - Partition into larger number n of subsets - Test each small set AND its large complement Outcomes: - small set fails reduce to subset with n=2 - large complement fails reduce to complement, n-1 - no test fails try with 2 n subsets - granularity is n > size of original set. Minimal. done

Summary of 2 nd minimizing Process

Example: GCC Dumps Core Original Test Case (input is program for GCC to compile)

Sample of Tests

Another example: Minimizing Fuzz

A Third Example: Mozilla print issue

Simplify = minimize user actions

Simplify = minimize HTML

Isolating Failure-Inducing Differences • The motivating issue: – The larger size of the simplified input -> The higher the number of required tests. Ex. Simplified FLEX input of 2121 characters: number of tests 11, 589 – 17, 960. With high precision up to 37, 454 tests. If individual tests fast, no problem. If complicated tests, unacceptable!!!

The basic idea of Isolation • Initially empty passing case. • As a test case fails: – Smaller test case is failing case • When a test case passes: – Larger test case is used as new passing test case minimizes diff between pass and fail test

Simplification vs Isolation • Simplification – Make each part of simplified test case relevant (ie, remove parts making failure go away) • Isolation – Find one relevant part of test case (ie removing this particular part makes failure go away) • Isolating diff will pinpoint failure cause much faster than minimizing test case

dd: extending ddmin • Compute subsets as subsets of failing test – passing test • Change rule “reduce to subset” so passing test U subset tested • Add 2 rules: – Increase to complement: If fail-subset passes, reduce diff between fail-subset and fail. – Increase to subset: If pass U subset passes, make pass U subset the new passing test.

Comparing ddmin vs dd • Isolating GCC input: – Ddmin – 731 tests to minimize program – Dd – 59 tests. Pinpoints relevant diff of 2 chars • Isolating fuzz input: – Ddmin number of tests significantly lower – Minimal failure-inducing diff always 1 char

Summary and Future Work • Delta debugging has become well known, with a plugin in Eclipse • Isolating failure-inducing inputs is key contribution • Since this paper: – – – Domain-specific simplification methods Optimization Undoing changes Combine with other techniques Other applications – security flaws