Backstop A Tool for Debugging Runtime Errors Chris

Backstop: A Tool for Debugging Runtime Errors Chris Murphy, Eunhee Kim, Gail Kaiser, Adam Cannon Columbia University

Observation n Novice/CS 1 Java programmers have difficulty dealing with uncaught exceptions and the resulting stack traces java. lang. Number. Format. Exception: For input string: "4. 5" at java. lang. Number. Format. Exception. for. Input. String (Number. Format. Exception. java: 48) at java. lang. Integer. parse. Int(Integer. java: 456) at java. lang. Integer. parse. Int(Integer. java: 497) at Test. main(Test. java: 124)

Problems n Students are often confused by runtime errors in Java n Stack traces were not developed with the introductory-level CS student in mind! n Few, if any, tools exist to help novice programmers interpret and debug runtime errors

Our Solution: Backstop n A command-line tool to produce detailed and helpful error messages when an uncaught runtime error (exception) occurs n Also provides debugging support by displaying the lines of code that are executed as a program runs, as well as the values of any variables on those lines

Hypothesis n “Backstop allows students to understand debug runtime errors more easily than just the Java stacktrace” n Backstop is part of Columbia University’s ongoing efforts to develop tools to help CS 1 students work with Java

Overview Related Work n Handling of Runtime Errors n Usability Study n Debugging Support n Conclusions and Future Work n

Related Work n Programming environments with easier-tounderstand compiler errors ¨ Blue. J, n Professor. J, Dr. Java, … Identify most common compiler errors ¨ Gauntlet, n Identifying logical errors ¨ In. STEP, n Expresso, Hi. C, … DEBUG, … Debugging ¨ CMe. Run

Handling of Runtime Errors n Designed to catch any uncaught exceptions in the student’s program n Once the exception (or any other Java runtime error) is caught, interpret its meaning and provide a user-friendly message that also seeks to provide some enlightenment as to what caused the error

Handling of Runtime Errors n Backstop. Runtime “wraps” the student’s (compiled) Java program ¨ Uses reflection to invoke the “main” method ¨ Catches any uncaught exception and dispatches it to the appropriate handler n Stack. Trace. Elements and source code are used to produce a friendlier, more helpful error message that is appropriate for the type of exception

***************************** ERROR ****************************************** Refer. Antoerror “the occurred at line 124 of Test. java in the method "main". I'm sorry, the program cannot continue. code”, not “you” The line of code that caused the error is int my. Num = Integer. parse. Int(arg); Reinforce what is already known It seems that the code wants to convert a String to an integer. However, the String "4. 5" appears to be a floating point value, not an integer. Don’t forget, an integer can’t contain a decimal point. You may want to use a different datatype to store the value, like float. This is called a Number. Format. Exception. Make a suggestion The full error is as follows: Alsojava. lang. Number. Format. Exception: use Java For input string: "4. 5" at java. lang. Number. Format. Exception. for. Input. String terms (Number. Format. Exception. java: 48) at java. lang. Integer. parse. Int(Integer. java: 456) at java. lang. Integer. parse. Int(Integer. java: 497) at Test. main(Test. java: 124)

Usability Study n Research Question: “Is the information provided by Backstop helpful for understanding runtime errors? ” 17 students (8 male, 9 female) who had just completed a CS 1 summer course n Students were asked to find/fix a runtime error in code they did not write n After completion, were asked subjective questions about the experience n

Usability Study 1. 2. 3. 4. 5. Subject was shown the code for a program that reads a String and displays the number of occurrences of each letter Subject was then told how the algorithm works It was then suggested that the code would produce an error for some inputs Subject was suggested to try a sentence like “hello world” Backstop error appeared

Problem Code // loop through and increment the values for each char for (int i = 0; i < s. length(); i++) { // get the character at index i char c = s. char. At(i); // increment the value in the array of occurrences // figure out its index by subtracting 'a' from it occur[(int)(c - 'a')]++; }

************************** ERROR ************************************** An error occurred at line 30 of String. Count. java in the method "main". I'm sorry, the program cannot continue. The line of code that caused the error is occur[(int)(c - 'a')]++; It seems that the code tried to use an illegal value as an index in an array. The code was trying to access an element at index -65 of the array called "occur". The expression "(int)(c - 'a')" had the value -65 when the error occurred. Remember, you cannot have a negative index. Be sure that the index is always positive. This error is called an Array. Index. Out. Of. Bounds. Exception. The full error is as follows: java. lang. Array. Index. Out. Of. Bounds. Exception: -65 at String. Count. main(String. Count. java: 30)

Questions n Was it more useful than the stacktrace? ¨ All n said “yes” Did the output mislead you? ¨ Two n said “yes”; one admitted he didn’t read it Describe the tone (vs. stack trace) ¨ Six said it was “friendly” ¨ Four said “too long” ¨ Two said “too pedantic for a general audience” ¨ Three did notice it

Observations n Most common complaint was that the output is too long ¨ One possibility is giving the option to see more details or user-adjusted verbosity levels n Average time to complete the task with Backstop was around 5. 8 minutes ¨ Understanding the cause took 1 -2 minutes

General Debugging Support n Display each line of code as it gets executed ¨ File name ¨ Line number ¨ Source code n Also display the values of any variables

Original source code sum = x + y; y = x; Line number printed before code is executed, in case of exception Modified source code System. out. print("n. Fib. java: 27: sum = x + y; "); System. out. print(" (x was " + x); System. out. print(" , y was " + y ); System. out. print(")"); sum = x + y; System. out. print(" (sum is now " + sum + ")"); System. out. print("n. Fib. java: 29: y = x; "); System. out. print(" (x was " + x); System. out. print(")"); y = x; Everything on one line Verb tense indicates System. out. print(" (y is now " + y + ")"); to retain original line number Output at runtime change of variable values Fib. java: 27: sum = x + y; (x was 2 , y was 1) (sum is now 3) Fib. java: 29: y = x; (x was 2) (y is now 2)

String. Count. java: 27: char c = s. char. At(i); (i was 0) (c is now h) String. Count. java: 30: occur[(int)(c - 'a')]++; (occur[(int)(c - 'a')] is now String. Count. java: 27: char c = s. char. At(i); (i was 1) (c is now e) String. Count. java: 30: occur[(int)(c - 'a')]++; (occur[(int)(c - 'a')] is now String. Count. java: 27: char c = s. char. At(i); (i was 2) (c is now l) String. Count. java: 30: occur[(int)(c - 'a')]++; (occur[(int)(c - 'a')] is now String. Count. java: 27: char c = s. char. At(i); (i was 3) (c is now l) String. Count. java: 30: occur[(int)(c - 'a')]++; (occur[(int)(c - 'a')] is now String. Count. java: 27: char c = s. char. At(i); (i was 4) (c is now o) String. Count. java: 30: occur[(int)(c - 'a')]++; (occur[(int)(c - 'a')] is now String. Count. java: 27: char c = s. char. At(i); (i was 5) (c is now !) String. Count. java: 30: occur[(int)(c - 'a')]++; ************************************* ERROR ***************************************************** An error occurred at line 30 of String. Count. java in the method "main". I'm sorry, the program cannot continue. 1) 1) 1) 2) 1) The line of code that caused the error is occur[(int)(c - 'a')]++; It seems that the code tried to use an illegal value as an index to an array. The code was trying to access an element at index -64 of the array called "occur". The variable "(int)(c - 'a')" had the value -64 when the error occured. Remember, you cannot have a negative index. Be sure that the index is always positive. This error is called an Array. Index. Out. Of. Bounds. Exception. The full error is as follows: java. lang. Array. Index. Out. Of. Bounds. Exception : -64 at String. Count. main(String. Count. java: 30) at sun. reflect. Native. Method. Accessor. Impl. invoke 0(Native Method)

Questions n Was it helpful? ¨ All n but one said “yes” Did the output mislead you? ¨ Five n said “yes”, but four said “only at first” Describe the appearance ¨ Five said more spacing would make it easier to read ¨ Four said more spacing would make it harder to read ¨ Three said the spacing would make no difference

Evaluation n Backstop helped students identify and fix errors in code they did not write n Backstop automates/mimics the way that many CS 1 students debug a Java program on the command line n Still need a study involving the use of both parts of Backstop to debug a runtime error

Conclusion and Future Work n Our contribution: Backstop ¨ Makes the error messages produced by uncaught Java exceptions easier to understand (compared to the Java stacktrace) ¨ Provides friendlier and more useful information about how to fix the error ¨ Automates simple debugging approach n Future work could include integration with an IDE, and further studies to measure Backstop’s effectiveness

Backstop: A Tool for Debugging Runtime Errors Chris Murphy, Eunhee Kim, Gail Kaiser, Adam Cannon Columbia University http: //www. cs. columbia. edu/~cmurphy/backstop cmurphy@cs. columbia. edu