Testing Concurrent Programs Rice Computer Science Club Mathias

Testing Concurrent Programs Rice Computer Science Club Mathias Ricken Rice University October 4, 2007

Moore’s Law *

Timeliness CPU clock frequencies stagnate Multi-Core CPUs provide additional processing power – Multiple threads needed to use multiple cores Writing concurrent programs is difficult!

Programming Examples

Unit Testing Unit tests… – Test a part, not the whole program – Occur earlier – Automate testing – Serve as documentation – Prevent bugs from reoccurring – Help keep the shared repository clean Effective with a single thread of control

Foundation of Unit Testing Unit tests depend on deterministic behavior Known input, expected output… Success correct behavior Failure flawed code Outcome of test is meaningful

Problems Due to Concurrency Thread scheduling is nondeterministic and machine-dependent – Code may be executed under different schedules – Different schedules may produce different results Known input, expected output… Success Failure correct behavior in this schedule, may be flawed in other schedule flawed code Success of unit test is meaningless

Possible Solutions Programming Language Features – Ensuring that bad things cannot happen – May restrict programmers Lock-Free Algorithms – Ensuring that if bad things happen, it’s ok – May limit data structures available Comprehensive Testing – Testing if bad things happen in any schedule – Does not prevent problems, but does not limit solutions either

Contributions Improvements to JUnit – Detect exceptions and failed assertions in threads other than the main thread Annotations for Concurrency Invariants – Express complicated requirements about locks and threads Tools for Schedule-Based Execution – Record, deadlock monitor – Random delays, random yields

Improvements to JUnit Uncaught exceptions and failed assertions – Not caught in child threads

Sample JUnit Tests public class Test extends Test. Case { public void test. Exception() { throw new Runtime. Exception("booh!"); } } public void test. Assertion() { Both tests assert. Equals(0, 1); fail. } } if (0!=1) throw new Assertion. Failed. Error();

Problematic JUnit Tests Main thread public class Test extends Test. Case { public void test. Exception() { new Thread(new Runnable() { public void run() { throw new Runtime. Exception("booh!"); } }). start(); Child thread } } end of Main thread spawns Child thread test uncaught! success!

Problematic JUnit Tests Main thread public class Test extends Test. Case { public void test. Exception() { new Thread(new Runnable() { public void run() { throw new Runtime. Exception("booh!"); } }). start(); Child Uncaught exception, thread } test should fail but } does not!

Improvements to JUnit Uncaught exceptions and failed assertions – Not caught in child threads Thread group with exception handler – JUnit test runs in a separate thread, not main thread – Child threads are created in same thread group – When test ends, check if handler was invoked

Thread Group for JUnit Tests Test thread public class Test extends Test. Case { public void test. Exception() { new Thread(new Runnable() { public void run() { throw new Runtime. Exception("booh!"); } }). start(); Child thread } invokes } checks Test. Group’s Uncaught Exception Handler

Thread Group for JUnit Tests Test thread public class Test extends Test. Case { public void test. Exception() { new Thread(new Runnable() { public void run() { throw new Runtime. Exception("booh!"); } }). start(); Child thread } spawns and waits resumes } Main failure! thread Test thread spawns Child thread end of test uncaught! invokes group’s handler check group’s handler

Improvements to JUnit Uncaught exceptions and failed assertions – Not caught in child threads Thread group with exception handler – JUnit test runs in a separate thread, not main thread – Child threads are created in same thread group – When test ends, check if handler was invoked Detection of uncaught exceptions and failed assertions in child threads that occurred before test’s end Past tense: occurred!

Child Thread Outlives Parent Test thread public class Test extends Test. Case { public void test. Exception() { new Thread(new Runnable() { public void run() { throw new Runtime. Exception("booh!"); } }). start(); Child thread } spawns and waits resumes } Main failure! thread Test thread spawns Child thread end of test uncaught! invokes group’s handler check group’s handler

Child Thread Outlives Parent Test thread public class Test extends Test. Case { public void test. Exception() { new Thread(new Runnable() { public void run() { throw new Runtime. Exception("booh!"); } }). start(); Child thread } check group’s resumes handler } Main spawns and waits success! thread Test thread spawns Too late! end of test Child thread uncaught! invokes group’s handler

Improvements to JUnit Child threads are not required to terminate – A test may pass before an error is reached Detect if any child threads are still alive – Declare failure if test thread has not waited – Ignore daemon threads, system threads (AWT, RMI, garbage collection, etc. ) Previous schedule is a test failure – Should be prevented by using Thread. join()

Enforced Join Test thread public class Test extends Test. Case { public void test. Exception() { Thread t = new Thread(new () { new Thread(new Runnable() Runnable { public void run() { { throw new Runtime. Exception("booh !"); throw new Runtime. Exception("booh!"); } } }); t. start(); … … t. join(); … Child } thread }

Improvements to JUnit Child threads are not required to terminate – A test may pass before an error is reached Detect if any child threads are still alive – Declare failure if test thread has not waited – Ignore daemon threads, system threads (AWT, RMI, garbage collection, etc. ) Previous schedule is a test failure – Should be prevented by using Thread. join()

Testing Conc. JUnit Replacement for junit. jar or as plugin JAR for JUnit 4. 2 Available as binary and source at http: //www. concutest. org/ Results from Dr. Java’s unit tests – Child thread for communication with slave VM still alive in test – Several reader and writer threads still alive in low level test (calls to join() missing)

Conclusion Improved JUnit now detects problems in other threads – Only in chosen schedule – Needs schedule-based execution Annotations ease documentation and checking of concurrency invariants – Open-source library of Java API invariants Support programs for schedule-based execution

Future Work Schedule-Based Execution – Replay given schedule – Generate possible schedules – Dynamic race detection – Probabilities/durations for random yields/sleeps Extend annotations to Floyd-Hoare logic – Preconditions, postconditions – Representation invariants

Many Thanks To… My advisor – Corky Cartwright My committee members – Walid Taha – Bill Scherer NFS and Texas ATP – For partially providing funding Rice Computer Science Club

Extra Slides

Tractability of Comprehensive Testing Test all possible schedules – Concurrent unit tests meaningful again Number of schedules (N) – t: # of threads, s: # of slices per thread detail

Extra: Number of Schedules Product of s-combinations For thread 1: choose s out of ts time slices For thread 2: choose s out of ts-s time slices … For thread t-1: choose s out of 2 s time slices For thread t-1: choose s out of s time slices Writing s-combinations using factorial Cancel out terms in denominator and next numerator Left with (ts)! in numerator and t numerators with s! back

Tractability of Comprehensive Testing If program is race-free, we do not have to simulate all thread switches – Threads interfere only at “critical points”: lock operations, shared or volatile variables, etc. – Code between critical points cannot affect outcome – Simulate all possible arrangements of blocks delimited by critical points Run dynamic race detection in parallel – Lockset algorithm (e. g. Eraser by Savage et al)

Critical Points Example Local Var 1 Thread 1 lock access unlock All lock accesses access unlock protected by lock Shared Var Lock Thread 2 All accesses protected by lock Local Var 1 access unlock All accesses protected by lock Local variables don’t need locking

Fewer Schedules Fewer critical points than thread switches – Reduces number of schedules – Example: Two threads, but no communication N=1 Unit tests are small – Reduces number of schedules Hopefully comprehensive simulation is tractable – If not, heuristics are still better than nothing

Limitations Improvements only check chosen schedule – A different schedule may still fail – Requires comprehensive testing to be meaningful May still miss uncaught exceptions – Specify absolute parent thread group, not relative – Cannot detect uncaught exceptions in a program’s uncaught exception handler (JLS limitation) details

Extra: Limitations May still miss uncaught exceptions – Specify absolute parent thread group, not relative (rare) Koders. com: 913 matches Thread. Group vs. 49, 329 matches for Thread – Cannot detect uncaught exceptions in a program’s uncaught exception handler (JLS limitation) Koders. com: 32 method definitions for uncaught. Exception method back

Extra: Dr. Java Statistics 2004 Unit tests passed failed not run Invariants met failed % failed KLOC “event thread” 736 610 36 90 5116 4161 965 18. 83% 107 1 2006 881 0 0 34412 30616 3796 11. 03 129 99 back

Concurrency Invariants Has to be called in event thread – Table. Model, Tree. Model May not be called in event thread – invoke. And. Wait() Have to acquire readers/writers lock – Abstract. Document – Dr. Java’s documents

Invariants Difficult to Determine May be found in – Javadoc comments – Only in internal comments – Whitepapers Often not documented at all Errors not immediately evident Impossible to check automatically

Java Annotations Add invariants as annotations @Not. Event. Thread public static void invoke. And. Wait( Runnable r) { … } Process class files – Find uses of annotations – Insert bytecode to check invariants at method beginning

Advantages of Annotations Java Language constructs – Syntax checked by compiler Easy to apply to part of the program – e. g. when compared to a type system change Light-weight – Negligible runtime impact if not debugging (slightly bigger class files) Automatic Checking

Predicate Annotations In annotation definition, specify static boolean Java method – Method must be callable from every context completely static and public Data in annotation, method arguments and value of this passed when method invoked

Predicate Annotation Example @Predicate. Link(value=Predicates. class, Refers to method="example", Predicates arguments=true). example public @interface Example. Annotation { String foo; } Definition

Predicate Annotation Example public class Test. Code { @Example. Annotation(foo="test") public void test(int param) { … } } … Test. Code t = new Test. Code(); t. test(5); Usage Call

Predicate Annotation Example public class Predicates { public static boolean example( Object this 0, int param, String foo) { return (foo. length()<param); }

Predicate Annotation Example @Predicate. Link(value=Predicates. class, method="example", arguments=true) public @interface Example. Annotation { String foo; } public class Test. Code { @Example. Annotation(foo="test") public void test(int param){…} } … Test. Code t = new Test. Code(); t. test(5); public class Predicates { public static boolean example( Object this 0, int param, String foo) { return (foo. length()<param); // this 0==t, param==5, foo=="test" }

Invariant Annotation Library @Only. Event. Thread, @Not. Event. Thread @Only. Thread. With. Name @Not. Null. Argument @Distinct. Arguments, @Same. Arguments @Only. Synchronized. This, @Not. Synchronized. This @Only. Synchronized. Argument, @Not. Synchronized. Argument etc. (ca. 80 annotations)

Problem: Multiple Annotations Java does not allow the same annotation class multiple times @Only. Thread. With. Name("foo") @Only. Thread. With. Name("bar") // error void test. Method() { … } Conjunctions, disjunctions and negations?

Annotation Subclasses? Let annotation extend a supertype? public @interface Invariant { } public @interface Only. Thread. With. Name extends Invariant { String name(); } public @interface And extends Invariant { Invariant[] terms(); } Subtyping not allowed for annotations

Generic Annotations? Write @And as generic annotation? public @interface And<T> { T[] terms(); } public @interface Only. Thread. With. Name { String name(); } Generics not allowed in annotations

Work-Around Different meta-annotation, @Combine(Combine. Mode. AND) public @interface Several. Names { Only. Thread. With. Name[] value(); } @Several. Names({@Only. Thread. With. Name("foo"), @Only. Thread. With. Name("bar")}) void test. Method() { … }

Combine Annotations May only contain invariant annotations – Predicate annotations – Combine annotations – Arrays of the above Predicate method automatically generated – Calls member predicate methods – Accumulates using AND, OR or NOT first negates, then uses AND Default mode is OR De Morgan’s Law: NOT (a OR b) = (NOT a) AND (NOT b)

Invariant Inheritance Invariants on a method – Apply to the method and all overriding methods in subclasses Invariants on a class – Apply to all methods introduced in that class or subclasses Methods can have invariants defined elsewhere All annotations describe requirements for the client (and, due to subclassing, for subclasses) – Allows frameworks to describe requirements – Description “thread-safe” is often wrong

Invariant Subtyping To maintain substitutability, subclasses may not strengthen invariants Invariants can be modeled as special input parameter – Tuple of invariants (“record” in λ calculus [Pierce]) – Subtyping rules for records declare the “wider” record as subtype – In function types, parameter types are contravariant I 0 = {}, I 1 = {inv 1}, I 2 = {inv 1, inv 2}, I 2 <: I 1 <: I 0 F 0 = I 0 → ·, F 1 = I 1 → ·, F 2 = I 2 → ·, F 0 <: F 1 <: F 2

Invariant Subtyping Analyze methods with invariants as parameter class A { void f() { … }; } class B extends A { @Inv 1 void f() { … }; } Invariants subtyping: class C extends B { @Inv 2 void f() { … }; } A <@ B <@ C IA = {}, IB = {inv 1}, IC = {inv 1, inv 2}; IC <: IB <: IA FA = IA → ·, FB = IB → ·, FC = IC → ·; FA <: FB <: FC Java subtyping: C <: B <: A

Detection of Subtyping Problems If Java subtyping and invariant subtyping disagree (A <: B but B <@ A) – Substitutability not maintained – Statically emit warning Detect if client subclasses do not use framework classes as prescribed – Safer multithreaded frameworks

Java API Annotations Started to annotate methods in Java API – 30 whole classes, 44 individual methods Community project at http: //community. concutest. org/ – – – Anyone can suggest annotations Vote on suggested annotations Browse by class or annotation type Annotations can be extracted into XML – Share annotations – Add checks without needing source code

Testing Invariant Checker Annotated two Dr. Java versions – 3/26/2004 – 9/2/2006 Ran test suite, logged invariant violations – 2004: 18. 83% failed – 2006: 11. 03% failed 2006 version easier to annotate – Better documentation of invariants