Setac A Phased Deterministic Testing Framework for Scala

Setac: A Phased Deterministic Testing Framework for Scala Actors Samira Tasharofi, Milos Gligoric, Darko Marinov, and Ralph Johnson

Background • Two approaches for parallel programming o Shared-memory • Do not scale with the number of cores • Cannot be applied for distributed systems o Message-passing • Best solution for distributed Systems • Can scale with the number of cores • Actor model o A model for message-passing programming o Supported by many libraries and languages • Scala, Erlang, C#, C++, Groovy(java-based), etc. Setac Testing Framework 1/25/2022 2

Actor Model • Actor is a computational entity with a mail box and local state o Deliver a message • Append the message to the mail box o Process a message • Extract a message from the mail box and execute that • Determined by message handler • In response to a message it processes: o Changes its local state o Sends messages to other actors o Creates new actors Setac Testing Framework 1/25/2022 3

Actors in Scala • Package scala. actors in Scala library • Developed by Philipp Haller • Features o o o Synchronous and asynchronous communication Dynamic creation/destroying of actors Exception handling Remote actors Customization of the thread pool executing actors Setac Testing Framework 1/25/2022 4

Question • How to test actor systems? Setac Testing Framework 1/25/2022 5

Problems • Schedule is a source of non-determinism in concurrent programs o Shared memory: order of accesses o Message-passing: order of messages • The output of the program might be different for a given input depending on the schedule o The bugs may only show up during some specific schedules o Systematic exploration can lead to state space explosion • Asynchronous events don not have instant effects Setac Testing Framework 1/25/2022 6

Example: Bounded. Buffer class Bounded. Buffer(max. Size: Int) extends Actor { var content = new Array[Int](size) var head, tail, cur. Size = 0 start override def act() = loop { react { case Put(x) if (cur. Size < max. Size) => { content(tail) = x tail = (tail + 1) % size cur. Size += 1 } case Get if (cur. Size > 0) => { val r = content(head) head = (head + 1) % size cur. Size -= 1 reply(r) } } class Producer(buf: Actor) extends Actor { start override def act() = loop { react { case Produce(values) => values. foreach(v => buf ! Put(v)) } } } class Consumer(buf: Actor) extends Actor { var token = -1 start override def act() = loop { react { case Consume(count) => for (i <- 0 to count-1) token = (buf !? Get). as. Instance. Of[Int] } } } Setac Testing Framework 1/25/2022 7

Buffer. Test producer class Buffer. Test { @Test def test. Buffer() { val buf = new Bounded. Buffer(1) val consumer = new Consumer(buf) val producer = new Producer(buf) Put(4), Put(5) buffer(1) Get, Get consumer 4, 5 producer ! Produce(List(4, 5)) consumer ! Consume(2) // consumer should receive 4 and 5 assert(… ) } Setac Testing Framework 1/25/2022 8

Buggy Bounded. Buffer class Bounded. Buffer(max. Size: Int) extends Actor { var content = new Array[Int](size) var head, tail, cur. Size = 0 start override def act() = loop { Bug react { case Put(x) if (cur. Size < <= max. Size) => { content(tail) = x tail = (tail + 1) % size cur. Size += 1 } case Get if (cur. Size > 0) => { val r = content(head) head = (head + 1) % size cur. Size -= 1 reply(r) } } Setac Testing Framework 1/25/2022 9

Buggy Bounded. Buffer: Schedules in Buffer. Test class Bounded. Buffer(max. Size: Int) extends Actor { var content = new Array[Int](size) var head, tail, cur. Size = 0 start override def act() = loop { react { case Put(x) if (cur. Size <= max. Size) => { content(tail) = x tail = (tail + 1) % size cur. Size += 1 } case Get if (cur. Size > 0) => { val r = content(head) head = (head + 1) % size cur. Size -= 1 reply(r) } } Setac Testing Framework consumer producer buffer(1) Put(4) Get 4 5 Put(5) Get Error! 5 1/25/2022 10

Correct Bounded. Buffer: Checking Assertions in Test class Bounded. Buffer(max. Size: Int) extends Actor { var content = new Array[Int](size) var head, tail, cur. Size = 0 start override def act() = loop { react { case Put(x) if (cur. Size < max. Size) => { content(tail) = x tail = (tail + 1) % size cur. Size += 1 } case Get if (cur. Size > 0) => { val r = content(head) head = (head + 1) % size cur. Size -= 1 reply(r) } } class Simple. Buffer. Test { @Test def test. Buffer() { val buf = new Bounded. Buffer(1) buf ! Put(4) assert(buf. cur. Size == 1) Fails!! } Setac Testing Framework buffer Put(4) local state mailbox … content, head, tail, cur. Size assert Fails!! 1/25/2022 11

Testing Actor Programs • Problems: o How to write unit tests for Actor programs while controlling the schedule? o How to check assertions at appropriate times? • Current Solutions: o Using synchronization constructs, e. g. Latches and Barriers o Using Thread. sleep Setac Testing Framework 1/25/2022 12

Buffer. Test: A Test Schedule Consumer Phase 1 4 Put(4) Put(5) Check assertions Phase 3 Boundede. Buffer(1) Get Check assertions Phase 2 Producer Get Check assertions 5 Setac Testing Framework 1/25/2022 13

Traditional: Buffer. Test class Buffer. Test { @Test def test. Buffer() { val put. Latch = new Count. Down. Latch(2) val get. Latch = new Count. Down. Latch(2) val buf = new Bounded. Buffer(1, put. Latch, get. Latch) val consume. Latch = new Count. Down. Latch(1) val consumer = new Consumer(buf, consume. Latch) val producer = new Producer(buf) //Phase 1 consumer ! Consume(1) consume. Latch. await() Thread. sleep(1000) assert(consumer. get. State == State. Blocked && buf. cur. Size == 0) // … } } consumer producer buffer(1) Phase 1 Get class Consumer(buf: Actor, consume. Latch: Count. Down. Latch) extends Actor { var token = -1 start override def act() = loop { react { case Consume(count) => consume. Latch. count. Down() for (i <- 0 to count-1) token = (buf !? Get). as. Instance. Of[Int] } } } Setac Testing Framework 1/25/2022 14

Traditional: Buffer. Test (cont) consumer Phase 2 4 producer buffer(1) Put(4) Put(5) Get Phase 3 5 class Buffer. Test { @Test def test. Buffer() { // … //Phase 2 producer ! Produce(List(4, 5)) put. Latch. await() assert(consumer. token == 4 && buf. cur. Size == 1) // Phase 3 consumer ! Consume(1) get. Latch. await() assert(consumer. token == 5 && buf. cur. Size == 0) } } class Bounded. Buffer(max. Size: Int, put. Latch: Count. Down. Latch, get. Latch: Count. Down. Latch) extends Actor { var content = new Array[Int](size) var head, tail, cur. Size = 0 start override def act() = loop { react { case Put(x) if (cur. Size < max. Size) => { content(tail) = x tail = (tail + 1) % size cur. Size += 1 put. Latch. count. Down() } case Get if (cur. Size > 0) => { val r = content(head) head = (head + 1) % size cur. Size -= 1 reply(r) get. Latch. count. Down() } } Setac Testing Framework 1/25/2022 15

Traditional Buffer. Test Problems • Unreliable o Thread. sleep • Complexity and Deadlock possibility o Latches and Barriers • Costly o Changing the program under test Setac Testing Framework 1/25/2022 16

Setac • Specify tests with some constraints on the schedule o Partial order of schedule messages o Centralized schedule, less complexity • Checking assertions when the system is stable o There is no message that can be processed o No progress in the system • No change in the run time environment • Minimal changes in the program under test Setac Testing Framework 1/25/2022 17

Setac: Buffer. Test class Buffer. Test extends Setac. Test { @Test def test. Buffer() { val buf = new Bounded. Buffer(1) val consumer = new Consumer(buf) val producer = new Producer(buf) val put 4 = create. Schedule. Message(producer, buf, Put(4)) val put 5 = create. Schedule. Message(producer, buf, Put(5)) val gets = create. Multiple. Schedule. Message(2, consumer, buf, Get) producer ! Produce(List(4, 5)) consumer ! Consume(2) //Phase 1 set. Schedule(gets(0)) assert. When. Stable(consumer. is. Blocked && buf. cur. Size == 0) //Phase 2 set. Schedule(put 4 -> put 5) assert. When. Stable(consumer. token == 4 && buf. cur. Size == 1 && put 5. is. Processed) // Phase 3 set. Schedule(gets(1)) assert. When. Stable(consumer. token == 5 && buf. cur. Size == 0) } } Setac Testing Framework 1/25/2022 18

Setac: Buffer. Test (cont) Test. Actor { class Bounded. Buffer(size: Int) extends Actor { // … } class Consumer(buf: Actor) extends Actor { Test. Actor { // … } class Producer(buf: Actor) extends Actor { Test. Actor { // … } Setac Testing Framework 1/25/2022 19

Setac: Buffer. Test • Removed Thread. sleep o More reliable • Removed synchronization constructs, e. g. Latches o Reduced complexity • Minimized changes in the program under test Setac Testing Framework 1/25/2022 20

Setac APIs • Follows widely used style for unit testing o Integrated with Junit o Refined over time based on examples • Checking assertions when system is stable • Total and partial order of schedule messages • Status of Test messages o Processed, delivered, etc. • Status of actors o Mail box: Number of messages, content of mail box, etc. o Execution Status: Blocked, Running, Suspended Setac Testing Framework 1/25/2022 21

Conclusions • Large number of schedules in actor programs o Only some schedules might be important to test • It is non-trivial to force specific schedule and check assertions at appropriate times o Sleeps - unreliable o Latches – hard to write/read o Changes in the program under test- high cost • Setac o Reliable o Easy to write o Very minimal changes in the program under test • Open for collaboration o Please try it: http: //mir. cs. illinois. edu/setac/ Setac Testing Framework 1/25/2022 23

Project Proposals • Akka o A better designed actor library for Scala users o Supported by Typesafe and has been used for large real projects • http: //akka. io/docs/akka/1. 1. 3/additional/companies-using-akka. html o Users always suffer from the lack of a testing framework o Users don’t know how to test their actors • Porting Setac for Akka o o A great contribution to Akka project Bigger user community Evaluating Setac Introducing a new approach/philosophy for testing actors Setac Testing Framework 1/25/2022 24

Project Proposals • Identifying the challenges and solutions for testing actor systems o The most common bugs in actor systems (bug patterns) o How to detect them and remove them o How to prevent them from happening • Facilitate Testing o Trace simplification for debugging o Record and replay traces o Propose interesting schedules Setac Testing Framework 1/25/2022 25

Let’s Have a Big Contribution Setac Testing Framework 1/25/2022 26