AnomalyBased Bug Prediction Isolation and Validation An Automated

Anomaly-Based Bug Prediction, Isolation, and Validation: An Automated Approach for Software Debugging Martin Dimitrov Huiyang Zhou School of Electrical Engineering and Computer Science University of Central Florida

Background: Terminology • Defect: Erroneous piece of code (a bug) • Infection: The defect is triggered => the program state differs from what the programmer intended. • Failure: An observable error (crash, hang, wrong results) in program behavior. Terminologies are based on the book “Why Programs Fail” by Andreas Zeller. University of Central Florida 1

Background: From Defects to Failures Variable and input values Erroneous code Observer sees failure Sane state Figure from the book “Why Programs Fail” by A. Zeller Infected state Program execution University of Central Florida 2

Motivation • The typical process of software debugging involves: 1) Examine the point of program failure and reason backwards about the possible causes. 2) Create a hypothesis of what could be the root cause. 3) Modify the program to verify the hypothesis. 4) If the failure is still there, the search resumes. • Software debugging is tedious and time consuming ! • In this work we propose an approach to automate the debugging effort and pinpoint the failure root cause. University of Central Florida 3

Presentation Outline • • Motivation Proposed approach Detecting anomalies (step 1) Isolating relevant anomalies (step 2) Validating anomalies (step 3) Experimental methodology Experimental results Conclusions University of Central Florida 4

Proposed Approach Dynamic Instruction Stream mov cmp jge mov lea movl inc cmp jl movl inc cmp . . Current instruction time University of Central Florida 5

Proposed Approach Dynamic Instruction Stream cmp jge mov lea movl inc cmp jl movl inc cmp jr . . Current instruction time University of Central Florida 5

Proposed Approach Dynamic Instruction Stream jge mov lea movl inc cmp jl movl inc cmp . . . . jr. . . movl. . . Current instruction time University of Central Florida 5

Proposed Approach Dynamic Instruction Stream mov lea movl inc cmp jl movl inc cmp . . . . jr. . . movl. . . inc. . . Current instruction time University of Central Florida 5

Proposed Approach Dynamic Instruction Stream mov lea movl inc cmp jl movl inc cmp . . . . jr. . . movl. . . inc. . . cmp. . . Current instruction time University of Central Florida 5

Proposed Approach Dynamic Instruction Stream lea movl inc cmp jl movl inc cmp . . . jr. . . movl. . . inc. . . cmp. . . jl. . . Current instruction time University of Central Florida 5

Proposed Approach Dynamic Instruction Stream movl inc cmp jl movl inc cmp . . jr. . . movl. . . inc. . . cmp. . . jl. . . test. . . Current instruction time University of Central Florida 5

Proposed Approach Dynamic Instruction Stream inc cmp jl movl inc cmp . . . . jr. . . movl. . . inc. . . cmp. . . jl. . . test. . . jne. . . Current instruction time University of Central Florida 5

Proposed Approach Dynamic Instruction Stream cmp jl movl inc cmp . . . . jr. . . movl. . . inc. . . cmp. . . jl. . . test. . . jne. . . mov. . . Current instruction time University of Central Florida 5

Proposed Approach Dynamic Instruction Stream jl movl inc cmp . . . jr. . . movl. . . inc. . . cmp. . . jl. . . test. . . jne. . . mov. . . call. . . Current instruction time University of Central Florida 5

Proposed Approach Dynamic Instruction Stream movl inc cmp . . jr. . . movl. . . inc. . . cmp. . . jl. . . test. . . jne. . . mov. . . call. . . mov. . . time • A program failure is observed: – – – • • Failure point Crash Hang Incorrect results, etc. Start the automated debugging process The output of our approach is a ranked list of instructions (the possible root-cause of failure) Failure University of Central Florida 5

Proposed Approach Step 1: Detect anomalies in program execution Dynamic Instruction Stream mov cmp jge mov lea movl inc cmp jl movl inc cmp . . anomaly Current instruction time University of Central Florida 6

Proposed Approach Step 1: Detect anomalies in program execution Dynamic Instruction Stream cmp jge mov lea movl inc cmp jl movl inc cmp . . jr . . . anomaly Current instruction time University of Central Florida 6

Proposed Approach Step 1: Detect anomalies in program execution Dynamic Instruction Stream jge mov lea movl inc cmp jl movl inc cmp jr . . . . anomaly . . . movl . . . Current instruction time University of Central Florida 6

Proposed Approach Step 1: Detect anomalies in program execution Dynamic Instruction Stream mov lea movl inc cmp jl movl inc cmp . . . . jr. . . movl. . . inc . . . anomaly Current instruction time University of Central Florida 6

Proposed Approach Step 1: Detect anomalies in program execution Dynamic Instruction Stream mov lea movl inc cmp jl movl inc cmp . . . . jr. . . movl. . . inc. . . cmp . . . anomaly Current instruction time University of Central Florida 6

Proposed Approach Step 1: Detect anomalies in program execution Dynamic Instruction Stream lea movl inc cmp jl movl inc cmp . . . jr. . . movl. . . inc. . . cmp. . . jl . . . anomaly Current instruction time University of Central Florida 6

Proposed Approach Step 1: Detect anomalies in program execution Dynamic Instruction Stream movl inc cmp jl movl inc cmp . . jr. . . movl. . . inc. . . cmp. . . jl. . . test . . . anomaly Current instruction time University of Central Florida 6

Proposed Approach Step 1: Detect anomalies in program execution Dynamic Instruction Stream inc cmp jl movl inc cmp . . . . jr. . . movl. . . inc. . . cmp. . . jl. . . test. . . jne . . . anomaly Current instruction time University of Central Florida 6

Proposed Approach Step 1: Detect anomalies in program execution Dynamic Instruction Stream cmp jl movl inc cmp . . . . jr. . . movl. . . inc. . . cmp. . . jl. . . test. . . jne. . . mov . . . anomaly Current instruction time University of Central Florida 6

Proposed Approach Step 1: Detect anomalies in program execution Dynamic Instruction Stream jl movl inc cmp . . . jr. . . movl. . . inc. . . cmp. . . jl. . . test. . . jne. . . mov. . . call . . . anomaly Current instruction time University of Central Florida 6

Proposed Approach Step 1: Detect anomalies in program execution Dynamic Instruction Stream movl inc cmp . . jr. . . movl. . . inc. . . cmp. . . jl. . . test. . . jne. . . mov. . . call. . . mov. . . time anomaly • Each anomaly constitutes a hypothesis for the root cause of program failure. anomaly Failure point Failure University of Central Florida 6

Proposed Approach Step 2: Isolate the relevant anomalies Dynamic Instruction Stream movl inc cmp . . jr. . . movl. . . inc. . . cmp. . . jl. . . test. . . jne. . . mov. . . call. . . mov. . . time anomaly • Create dynamic forward slices from the anomalies to the failure point. • Discard anomalies which do not lead to the failure point. anomaly Failure point Failure University of Central Florida 7

Proposed Approach Step 3: Validate the isolated anomalies Dynamic Instruction Stream movl inc cmp . . jr. . . movl. . . inc. . . cmp. . . jl. . . test. . . jne. . . mov. . . call. . . mov. . . time • Automatically “fix” the anomaly and observe if the program still fails. anomaly Failure point Failure University of Central Florida 8

Proposed Approach Step 3: Validate the isolated anomalies Dynamic Instruction Stream movl inc cmp . . jr. . . movl. . . inc. . . cmp. . . jl. . . test. . . jne. . . mov. . . call. . . mov. . . time • If the failure disappears we have a high confidence the root cause have been pinpointed. No failure Success University of Central Florida 8

Detecting Program Anomalies (Step 1) • When infected by a software bug the program is likely to misbehave: – – Out-of-bounds addresses and values Unusual control paths Page faults Redundant computations, etc. • Anomaly detection: Infer program specifications from passing runs and turn them into ‘soft’ assertions. – Learn program invariants during passing runs (e. g. variable “i” is always between 0 and 100) – Flag violated invariants during the failing run (e. g. Report anomaly if variable “i” is 101) University of Central Florida 9

Detecting Program Anomalies • We use several anomaly detectors to monitor a large spectrum of program invariants and catch more bugs. • DIDUCE [S. Hangal et al. , ICSE 2002] – Instructions tent to produce values/addresses within a certain range (e. g. 0 <= i <= 100). Detect violations of these invariants. • Acc. Mon [P. Zhou et al. , MICRO-37 2004] – Only a few static instructions access a given memory location (load/store set locality). Signal an anomaly if memory access does not belong to the load/store set. • Loop. Count – Detect abnormal number of loop iterations. University of Central Florida 10

Detecting Program Anomalies void more_arrays () {. . . a_count += STORE_INCR; Heap Memory /* Copy the old arrays. */ for (indx = 1; indx < old_count; indx++) arrays[indx] = old_ary[indx]; /* Initialize the new elements. */ for (; indx < v_count; indx++){/* defect */ arrays[indx] = NULL; /* infection: */ /* Free the old elements. */ if (old_count != 0){ free (old_ary); /* crash */ } a_count v_count data 0 x 80 bfe 6 c data size data 0 x 80 bfe 7 c 0 x 80 bfe 70 0 x 80 bfe 74 0 x 80 bfe 78 0 x 80 bfe 80 } University of Central Florida 11

Detecting Program Anomalies void more_arrays () {. . . a_count += STORE_INCR; /* Copy the old arrays. */ for (indx = 1; indx < old_count; indx++) arrays[indx] = old_ary[indx]; /* Initialize the new elements. */ for (; indx < v_count; indx++){/* defect */ arrays[indx] = NULL; /* infection: */ /* Free the old elements. */ if (old_count != 0){ free (old_ary); /* crash */ } DIDUCE 24 anomalies Acc. Mon 68 anomalies Loop. Count: Loop. Count 36 anomalies Loop iterates more times than Acc. Mon: store usual. instruction (false –is not in store set of this positive) memory location. this memory location. (false – DIDUCE: positive) Address of store instruction is out of normal range. (false – positive) } University of Central Florida 12

Detecting Program Anomalies: Architectural Support • DIDUCE and Acc. Mon capture invariants using limited size caches structures, as proposed in previous work • Loop. Count utilizes the existing loop-branch predictor to detect anomalies. • Advantages and disadvantages of hardware support: + Performance efficiency + Portability + Efficient ways to change or invalidate dynamic instructions - Limited hardware resource may become a concern University of Central Florida 13

Isolating Relevant Anomalies (Step 2) • Anomaly detectors alone are NOT effective for debugging: – May signal too many anomalies / false positives – Tradeoff between bug coverage and number of false positives • Our solution: – Allow aggressive anomaly detection for maximum bug coverage – Automatically isolate only the relevant anomalies by constructing dynamic forwards slices from the anomaly to the failure point University of Central Florida 14

Isolating Relevant Anomalies: Architectural Support • Add token(s) to each register and memory word. • Detected anomalies set a token associated with the destination memory word or register. • Tokens propagate based data dependencies. • When the program fails, examine the point of failure for token. University of Central Florida 15

Isolating Relevant Anomalies: Architectural Support void more_arrays () {. . . Token Memory 0 x 80 bfdf 8 /* Copy the old arrays. */ for (indx = 1; indx < old_count; indx++) arrays[indx] = old_ary[indx]; 0 x 80 bfdfc 0 x 80 bfe 00 0 x 80 bfe 04 /* Initialize the new elements. */ for (; indx < v_count; indx++){/* defect */ arrays[indx] = NULL; /* infection */ } /* Free the old elements. */ if (old_count != 0){ free (old_ary); /* crash */ Failure } mov %ebx, 0 xc(%edx) University of Central Florida . . . 0 x 80 bfe 08 0 x 80 bfe 7 c 0 x 80 bfe 80 0 x 80 bfe 84 0 x 80 bfe 88 16

Isolating Relevant Anomalies: Architectural Support • Problem: Many tokens for each memory location/ register • Solution: – We leverage tagged architectures for information flow tracking. – Use only one token (1 bit) (i. e. , shared by all anomalies ) – We leverage delta debugging [A. Zeller, FSE 1999] to isolate the relevant anomalies automatically. Number of Initial Anomalies Number of Isolated Anomalies DIDUCE 24 anomalies DIDUCE 3 anomalies Acc. Mon 68 anomalies Acc. Mon 2 anomalies Loop. Count 36 anomalies Loop. Count 4 anomalies University of Central Florida 17

Delta-Debugging University of Central Florida 18

Validating Isolated Anomalies (Step 3) • Validate the remaining anomalies by applying a “fix” and observing if the program failure disappears. • Our “fix” is to nullify the anomalous instruction (turn it into no-op) • If the program succeeds, we have a high confidence we have found the root cause (or at least broken the infection chain) University of Central Florida 19

Validating Isolated Anomalies void more_arrays () {. . . /* Initialize the new elements. */ for (; indx < v_count; indx++){/* defect */ arrays[indx] = NULL; /* infection */ Token Memory 0 x 0 /* Free the old elements. */ if (old_count != 0){ free (old_ary); /* crash */ } } Success • The “size” information is not corrupted and the program terminates successfully. University of Central Florida data data size data 0 x 80 bfe 6 c 0 x 80 bfe 70 0 x 80 bfe 74 0 x 80 bfe 78 0 x 80 bfe 7 c 0 x 80 bfe 80 20

Validating Isolated Anomalies • Four possible outcomes of our validation step: > Hello World ! Success Program produces expected output. > Hello Martin No crash Program does not crash, but produces unexpected results. >. . . (hang) > Segmentation Fault Failure Unknown Program fails in an unexpected manner. Program fails the same way as before. • Rank isolated anomalies based on the outcome: succeed (highest) , no crash, unknown, failure (lowest) • In our running example the root-cause is ranked #1. University of Central Florida 21

Experimental Methodology • Implemented a working debugging tool using binary instrumentation (PIN). • Evaluated applications from Bug. Bench [S. Lu et al. , Bugs 2005] and gcc compiler. Application Lines of Defect Location Code Defect Description bc-1. 06 17, 042 storage. c: 176 Incorrect bounds checking causes heap buffer overflow gzip-1. 2. 4 8, 163 gzip. c: 1009 Buffer overflow due to misuse of library call strcpy ncompress-4. 2. 4 2 defects polymorph-0. 4. 0 1, 922 716 compress 42. c: 886 and 1740 polymorph. c: 200 Buffer overflow due to misuse of library call strcpy Incorrect bounds checking causes stack buffer underflow Incorrect bounds checking causes stack buffer overflow man-1. 5 h 1 4675 man. c: 998 Incorrect loop exit condition causes stack buffer overflow gcc-2. 95. 2 338, 000 combine. c: 4013 Incorrect call to apply_distributive_law causes a loop in the RTL tree University of Central Florida 22

Experimental Results Application Initial Anomalies Isolated Anomalies Validated (Application Succeeds) Defect rank D A L bc-1. 06 24 68 36 3 2 4 1 1 gzip-1. 2. 4 21 40 19 1 1 1 2 ncompress-4. 2. 4 (strcpy defect) 6 7 6 2 2 1 0 1 1 1 ncompress-4. 2. 4 (stack underflow) 2 4 n/a 1 1 0 0 1 polymorph-0. 4. 0 21 10 20 3 1 0 0 3 man-1. 5 h 1 15 114 46 2 2 0 1 1 0 1 gcc-2. 95. 2 768 1062 666 84 130 47 2 4 3 9 University of Central Florida 23

Case Study: GCC University of Central Florida 24

GCC Defect University of Central Florida 25

GCC Fix University of Central Florida 26

Experimental Results Compared to Failure-Inducing Chops Application Failure-Inducing Proposed Chops Approach bc-1. 06 167 1 gzip-1. 2. 4 6 2 ncompress-4. 2. 4 (strcpy defect) ncompress-4. 2. 4 (stack underflow) polymorph-0. 4. 0 4 1 11 1 8 3 man-1. 5 h 1 n/a 1 gcc-2. 95. 2 1335 9 University of Central Florida 27

Limitations • No failure, no bug detection – Un-triggered bugs – Bugs are triggered but output is correct • Target at bugs in sequential programs University of Central Florida 28

Conclusions • We present a novel automated approach to pinpoint the root causes of software failures: – Detect anomalies during program execution. – Isolate only the relevant anomalies. – Validate isolated anomalies by nullifying execution results • Our experimental results demonstrate that we accurately pinpoint the defect even for large programs such as gcc. University of Central Florida 29

Questions • The tool is available for download at: http: //csl. cs. ucf. edu/debugging University of Central Florida 30