A Library for Testing Student Assignments on Concurrent

A Library for Testing Student Assignments on Concurrent Programming by Mikael Mayër Ravichandhran Madhavan Laboratory for Automated Reasoning EPFL

Overview Concurrent Program var state = new A() … def foo(x) =synchronized { … wait() … notify() } Unit test def test 1 = { val (x, y) = parallel(foo(1), foo(1)) assert(x == y) } Unit Test Engine Tries multiple interleavings Buggy interleaving • Assertion failure • Exception • Deadlock • Timeout Passes all interleavings subject to • Context depth • Max # of interleavings • Resource limits

Testing concurrent software is well researched Half-a-million results!

Search query: “Model Checking” 37 k results

Search query: “Unit testing” 8 k results

A broad classification Bounded model checkers Race Detection Assertions Runtime Crashes NPE, Out-ofbounds Automatically discovers threads, schedules & inputs Unit testing/ model checking hybrids Unit test engines • User defines threads, schedules & inputs • Less automation • More flexibility

Where does our tool fit in? Bounded model checkers Race Detection Assertions Runtime Crashes NPE, Out-ofbounds Unit testing/ model checking hybrids Unit test engines In our tool: • user defines threads & inputs • schedules are explored automatically

Driving application Solution for an assignment A failed test case & an interleaved trace A student trying to learn Scala Server running test suites designed by the instructor

What does our library offer? Student’s Perspective Instructor’s Perspective • No code modifications / annotations Choose shared atomic blocks • Use standard synchronization Check functional properties • Run the test engine locally Test specific code portions

A Concurrent bounded buffer • Elements can be enqueued or dequeued • Enqueue waits if the queue is full • Dequeue waits if the queue is empty

A Concurrent bounded buffer class Queue[T](val size: Int) { var _buffer = new Array[Option[T]](size) var _head = 0; Mutable State def def … def def } update(index: Int, elem: T) apply(index: Int): T head: Int is. Full: Boolean is. Empty: Boolean enqueue(e: T): Unit dequeue(): T Accessor / mutators Queue APIs (Students task to implement)

Enqueuing Dequeuing Thread 1: enqueue 1 Thread 3: dequeue synchronized { while (is. Full()) wait() add(1) notify() } synchronized { while (is. Empty()) wait() remove() notify() } Thread 2: enqueue 2 synchronized { while (is. Full()) wait() add(2) notify() } Buffer 1 Thread 4: dequeue synchronized { while (is. Empty()) wait() remove() notify() } Buggy solution

Marking shared atomic operations class Queue[T](val size: Int, schedr: Scheduler) extends Schedulable. Monitor { def update(i: Int, e: T) = exec { … }{s“a($i) = $e”} def apply(index: Int): T = exec { … } def head: Int = exec { … }

Redefines synchronization primitives class Queue[T](val size: Int, schedr: Scheduler) extends Schedulable. Monitor { def update(i: Int, e: T) = exec { … }{s“a($i) = $e”} def apply(index: Int): T = exec { … } def head: Int = exec { … }

Overriding synchronization primitives trait def } trait Monitor { wait()(implicit d: Dummy) = … synchronized[T](e: => T) = … Schedulable. Monitor extends Monitor { } wait, synchronized, notify are not overridable Implicit parameters and call-by-name saves the day! A use case for supporting overridable Monitors in future versions

Writing Testcases Thread code

Error trace printed on failure 1: synchronized check 3: synchronized check 4: synchronized check 2: synchronized check 3: synchronized -> enter 3: Read count -> 0 3: wait 4: synchronized -> enter 4: Read count -> 0 … 3: wait 3: Deadlock: Threads 2, 3 are waiting

Are these assertions enough?

Library Implementation

Schedules T 2 T 1 T 2 T 3 T 1 … Thread 1 ………… ………… a b c e ……… Thread 2 Thread 3 ………… ………… ………… d Shared operation Local operation

Enforcing Schedules def exec[T](op: => T)(msg: T => String) { wait. For. Turn() op … } All threads wait at the beginning of shared/synchronization operations Scheduled thread continues while others wait for turn An issue: scheduled thread may not own the locks it needs Solution: track the locks held & the execution state of each thread

Thread State Transitions Handles re-entrance of synchronized blocks

Schedule: 3 4 1 4 2 3 3 3 Running Sync Enqueuing Dequeuing Thread 1: enqueue 1 Thread 3: dequeue synchronized { while (is. Full()) wait() add(1) notify() } synchronized { while (is. Empty()) wait() remove() notify() } Thread 2: enqueue 2 Wait Running Sync Buggy solution synchronized { while (is. Full()) wait() add(2) notify() } Buffer 1 Wait Running Sync Thread 4: dequeue synchronized { while (is. Empty()) wait() remove() notify() } Wait Running Sync

Assignment 2: Lock-free sorted list Concurrent linked list of sorted numbers with • Find • Insert • Delete – find an element in the list – Insert an element in the sorted order – delete one occurrence of an element Using only compare. And. Set operation

Making CAS a shared operation A Wrapper for CAS class Atomic. Var(init: Int){ val a = new java. util. concurrent. Atomic. Integer(init) def get = exec{ a. get }(r => s"get $r") def compare. And. Set(x: Int, y: Int): Boolean = exec{ a. compare. And. Set(x, y) }{r => s"Set $x to $y Success: $r”} }

Sorted list test case Thread code Correctness check

Sorted list error trace Inserting (3, 4) in parallel into List(1, 2, 5) Found: List(1, 2, 4, 5) Expected: List(1, 2, 3, 4, 5) 1: HEAD: get next = Node(1) 2: Node(1): get next = Node(2) 1: Node(2): get next = Node(5) 2: Node(1): get next = Node(2) 2: Node(2): get next = Node(5) 1: Node(2): set next = Node(3) 2: Node(2): set next = Node(4)

Buggy interleaving Thread 1 HEAD 1 3 2 Thread 2 5 4

Experimental Evaluation

Assignment 1: Bounded buffer • 138 students submitted • 12 unit tests • Discovered 46 errors for 24 students • 9 deadlocks • 37 wrong outputs • Common errors: • Using notify instead of notify. All • Not enclosing wait inside a while • Underlocking

Assignment 2: Lock-free sorted list • 121 students submitted • 14 unit tests • Discovered 663 errors • Common errors: • Not handling all races using CAS and retries

Current Limitations • User marks shared atomic operations • Only threads created through the library API are scheduled • Reordering due to weak memory are not explored

Conclusion and Take away • A unit test engine for Scala, written in Scala, as a Library • Can handle standard synchronization primitives • Easy to import into any Scala application like JUnit, Scala. Test • Demonstrated on testing student assignments • FOSS implementation: github. com/epfl-lara/muscat

Related Work • M. Musuvathi, S. Qadeer, and T. Ball. Chess: A systematic testing tool for concurrent software. 2007. • W. Visser, K. Havelund, G. Brat, S. Park, and F. Lerda. Model checking programs. ASE, 2003. • T. Elmas, S. Qadeer, and S. Tasiran. Goldilocks: A race-aware java runtime. ACM, Nov. 2010. • B. Fischer, O. Inverso, and G. Parlato. Cseq: a sequentialization tool for c. In TACAS, 2013. • I. Rabinovitz and O. Grumberg. Bounded model checking of concurrent programs. CAV, 2005. • T. Andrews, S. Qadeer, S. K. Rajamani, J. Rehof, and Y. Xie. Zing: Exploiting program structure for model checking concurrent software. CONCUR, 2004. • J. Burnim, T. Elmas, G. Necula, and K. Sen. Concurrit: testing concurrent programs with programmable state-space exploration. USENIX Workshop, 2012. • V. Jagannath, M. Gligoric, D. Jin, Q. Luo, G. Rosu, and D. Marinov. Improved multithreaded unit testing. In FSE’ 11.

Related Work • M. Emmi, B. K. Ozkan, and S. Tasiran. Exploiting synchronization in the analysis of shared-memory asynchronous pro- grams. In SPIN, 2014. • J. Hamza. Algorithmic Verification of Concurrent and Distributed Data Structures. Ph. D thesis, 2015. • A. Bouajjani, M. Emmi, C. Enea, and J. Hamza. On reducing linearizability to state reachability. In ICALP, 2015. • M. Kebrt and O. Sery. Unitcheck: Unit testing and model checking combined. In ATVA, 2009. • D. Kroening and M. Tautschnig. CBMC – C bounded model checker. In TACAS, 2014. • A. Lal, T. Touili, N. Kidd, and T. Reps. Interprocedural analysis of concurrent programs under a context bound. In TACAS, 2008. • S. Qadeer. Poirot – a concurrency sleuth. In ICFEM, 2011. • K. Sen and G. Agha. Cute and jcute: Concolic unit testing and explicit path modelchecking tools. In CAV, 2006.

Related Work • O. Shacham, N. Bronson, A. Aiken, M. Sagiv, M. Vechev, and E. Yahav. Testing atomicity of composed concurrent operations. In OOPSLA, 2011. • William Pugh, Nathaniel Ayewah. Unit testing concurrent software. In ASE, 2007. • Klaus Havelund, Thomas Pressburger. Model checking JAVA programs using JAVA Path. Finder. In Journal on Software Tools for Technology Transfer, 2000