Systematic Testing of Reactive Software with Nondeterministic Events

Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven Yongbase Park, Shin Hong, Moonzoo Kim, Dongju Lee, Junhee Cho Software Testing and Verification Group (SWTV) http: //swtv. kaist. ac. kr S. Hong @ KAIST / 18

Safety Problems due to Poor Quality of SW 2021 -06 -03 Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven S. Hong @ KAIST 2 / 18

Strong IT Industry in South Korea Time-to. Market? 2021 -06 -03 SW Quality? Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven S. Hong @ KAIST 3 / 18

Overview • Reactive programs can have subtle concurrency errors as their behaviors depend on timing of input events • We developed an automated testing framework that generates event timing/sequences systematically to detect concurrency bugs • We found 3 new concurrency bugs in the controller S/W of the LG electric oven product 2021 -06 -03 Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven S. Hong @ KAIST 4 / 18

Reactive Programs in Controller Devices • A reactive program repeats: (1) receiving an input (2) updating the internal state for the new input (3) generating output w. r. t the new state • A reactive program utilizes non-deterministic interrupt handlers to receive a given input event immediately • Embedded software on 8 -bit MCU does not use standard synchronization since OS is not supported 2021 -06 -03 Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven S. Hong @ KAIST 5 / 18

Hard to Detect Corner-Case Bugs in Reactive Software • Reactive software may have various execution scenarios because of various event ordering and event timing – A corner-case bug only occurs on specific event ordering and event timing • Reactive software may suffer from corner-case bugs, which are hard to detect 1 2 3 4 5 6 7 8 9 10 11 int cnt=1; void main() { while(1) { if(cnt>0) { int x=10/cnt; print(x); } cnt++; }} void handler() { cnt=0; } 2021 -06 -03 Scenario #1 11: cnt=0 4: if(cnt>0) 8: cnt++ Scenario #N … Buggy Scenario 4: if(cnt>0) 5: x=10/cnt 6: print(x) 4: if(cnt>0) 11: cnt=0 5: x=10/cnt Error! 8: cnt++ 11: cnt=0 Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven S. Hong @ KAIST 6 / 18

Challenges in Testing Reactive Programs • Tests should cover all event sequence, together with all timing of events for each sequence – Infeasible for human testers – No proper automated testing techniques • Random noise injection technique is not effective and not efficient for finding concurrency bugs • Model checkers for multithreaded programs are not scalable to verify real-world controller software 2021 -06 -03 Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven S. Hong @ KAIST 7 / 18

Systematic Event Generation Framework 1. Instrument a target reactive program to invoke an event handler in a middle of the main loop depending on symbolic values 2. Use conclic testing to systematically generate symbolic values to cover all event scenarios Target program Code transform. Transformed program Program inputs Concolic testing CREST-BV 2021 -06 -03 Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven Event scenarios S. Hong @ KAIST 88 / 18

Code Transformation for Concolic Testing • Insert a probe for every statement the main loop – A probe invokes an event handler if its activation condition is satisfied in an execution • Define the activation condition of each probe at each main loop iteration as symbolic values main() { sym. Event. Scenario(); while (1) { if(act(2)) ev. Hdr(); int t = 0 ; if(act(3)) ev. Hdr(); 11 ev. Hdr(){…} data = 10 ; if(act(4)) ev. Hdr(); f(data) ; 2021 -06 -03 Systematic Testing of Reactive Software } } with Non-deterministic Events: main() { 1 while (1) { 2 int t = 0 ; 3 data = 10 ; 4 f(data) ; 5} } A Case Study on LG Electric Oven Event. Scenario Iter Active probe 1 st 2 0 2 nd 3 0 3 rd 2 4 … … … S. Hong @ KAIST 9 / 18

Contributions • Reported the real-world challenges to test reactive programs with non-deterministic events • Developed a systematic event generation (SEG) framework to generate various event scenarios • Demonstrated the effectiveness of the SEG framework by detecting 3 new bugs from LG electric oven S/W 2021 -06 -03 Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven S. Hong @ KAIST 10 / 18

Project Background • Target: embedded S/W on a high-end electric oven – 19, 650 lines of C code (180 files, 658 functions) • The project takes 5 months in 2014 • Team – 1 research engineer in LG CTO – 1 field developer in LGE H/A – 2 graduate students + 1 professor in KAIST • The source code, test cases, state-transition diagram, and emulator of the target system is given to the KAIST members. 2021 -06 -03 Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven S. Hong @ KAIST 11 / 18

Oven Controller Software Key input Key/door Event Handler Shared memory Main Loop Shared memory Door input Timed Event Handler LED command Cooking command Input buffer Circular. Queue LED output Cook Command Handler Cooking device control output Timer Test drivers Unit Testing of Circular. Queue Integration testing 2021 -06 -03 LED Handler Test targets Test oracle 145 LOC Invariants of Queue data structure 12, 691 LOC System state transitions diagram spec Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven S. Hong @ KAIST 12 / 18

Unit Testing Overview • Circular. Queue – enqueue() inserts an element if the queue is not full – dequeue() returns and removes the oldest member in the queue if the queue is not empty Circular. Queue • Test scenario dequeue() enqueue() Shared memory – Key theinput main loop performs dequeue(), and Main Loop Key/door perform enqueue() concurrently – the event handlers Shared memory Event Handler Input buffer • Two multi-variable atomicity violations are detected – Overwriting bug: unintentionally overwrite the oldest value with a new value – Inconsistency bug: unintentionally remove all values 2021 -06 -03 Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven S. Hong @ KAIST 13 / 18

Inconsistency Bug 1 void* dequeue() { 2 void* r = NULL; 3 if (!is. Empty()) { 4 r = head; 5 head=get. Next. Idx(…); 6 is. Full = FALSE; } 7 return r; } Circular. Queue status is. Full Arr ==false 1 2021 -06 -03 - tail head is. Full Arr ==false 1 2 - head tail is. Full Arr ==true 4 is. Empty()==true as is. Full==false && head==tail 2 Main loop is. Full ==false 2 3 event enqueue(3 ) event enqueue(4 ) head tail Arr 4 2 3 head tail Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven S. Hong @ KAIST 14 / 18

Testing Performance Technique SEG Random Average time to detect fault Fault detection 100% 15 sec 20% 1653 sec 23% 1578 sec 0% - • SEG is more effective to find the inconsistency bug than then random noise injection-based testing techniques 2021 -06 -03 Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven S. Hong @ KAIST 15 / 18

Integration Testing • We tested all code of the main loop and the key event handlers • SEG found illegal transition bug – An error occurs when the oven control software retrieves multiple input events from the input buffer Main loop auto-cook dequeue Input buffer Main loop dequeueＸ 2 auto-cook function-R dequeue Input buffer Success 2021 -06 -03 Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven Input buffer Error S. Hong @ KAIST 16 / 18

Lessons Learned on Effective Technology Transfer • The developers were happy to adopt the SEG framework because – Our group has 10 years’ experience in industrial collaboration with Samsung, LG, and Hyundai – The developers are well-trained about the technique (i. e. , concolic testing) so that they can adapt the tool for their needs • In 2012, Prof. Kim made 8 weeks of seminars on automated testing techniques to the field engineers of LGE • In 2013, a pilot study was made to apply concolic testing techniques to LGE software • In 2015, a new target domain (i. e. , washing machines) is being explored and plan to test robot vacuum cleaners next – Use success stories w/ other industries as a reference 2021 -06 -03 Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven S. Hong @ KAIST 17 / 18

Conclusion and Future Work • SEG effectively detects corner-case concurrency bugs in real-world embedded software – SEG generates event timing and sequences systematically – SEG found 3 new bugs and improved the target S/W quality • SEG found similar concurrency bugs from clocksource device drivers in Linux 3. 19 – We had reported 3 new bugs, and the developers made patches for these bugs 2021 -06 -03 Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven S. Hong @ KAIST 18 / 18

Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven Yongbase Park, Shin Hong, Moonzoo Kim, Dongju Lee, Junhee Cho KAIST LG Electronics S. Hong @ KAIST / 18

Reactive Program • The main loop repeats: (1) receive an input from an event handler (2) update the internal state w. r. t the new input (3) generate output w. r. t the new state • If an input event is given, the main loop is suspended immediately and the event handler is executed instead main(){ while(1){ m 1; … m. L ; } } 2021 -06 -03 ev. Hdr 1(){ e 11; … e 1 N ; } ev. Hdr 2(){ e 21; … e 2 K; } MCU e 1 ; … e. N ; m 1 ; … m. L ; User CPU m 1 e 1 ; … e. N ; m 2 ; … m. L ; Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven … User m 1 ; … m. L ; e 1 ; … e. N ; S. Hong @ KAIST 20 / 18

Challenges in Reactive Software Testing • Reactive software may suffer from concurrency errors, which are hard to detect • In manual testing – Human testers cannot precisely control all possible timing of events – Human testers may not design complex event scenarios as test cases • In automated testing by conventional techniques – Conventional concolic testing does not generate various order/timing of events – Model checking technique [Chandra 02] [Fidge 05] [Holzmann 99] is not effective for checking reactive software in industry due to high cost of abstract model creation and state explosion problems – Noise injection random testing [Lei 05] [Stoller 02] may not find corner-case bugs with low possibility of detection 2021 -06 -03 Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven S. Hong @ KAIST 21 / 18

Model Checking • Model checking techniques exhaustively and automatically check whether a given model meets a given specification – Previous works created models of reactive software with human efforts • [Fidge, 05] and [Chandra, 02] create the target model manually • [Holzmann, 99] semi-automatically generates a model using a usergiven translation table • Limitations – Developers in industry cannot afford time to create abstract models for reactive software due to hard time-to-market pressure – Model checking techniques may not generate results within limited time and memory space due to state explosion problem • The number of states of given model increases exponentially with the number of events 2021 -06 -03 Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven S. Hong @ KAIST 22 / 18

Concolic Testing • Concolic (CONCrete + symb. OLIC) testing (dynamic symbolic execution) aims to explore all paths of the given program 1. run a given program with concrete inputs 2. create a symbolic path constraint from the concrete execution 3. negate a branch of the symbolic path constraint 4. generate a new test input that satisfies the negated constraint 1 void main() { if(ev. Id==0)handler(); 2 stmt 1; 3 if(ev. Id==1)handler(); 4 stmt 2; 5 6 } 2021 -06 -03 No. ev. Id Sym. path constraint 1 1 ev. Id==0&&ev. Id!=1 0 0 1 ev. Id!=0&&ev. Id==1 2 Systematic Testing of Reactive Software withev. Id!=0&&ev. Id==1 Non-deterministic Events: 1 2 2 ev. Id!=0&&ev. Id!=1 3 A Case Study on LG Electric Oven Negated path constraint ev. Id==0&&ev. Id==1 ev. Id!=0&&ev. Id!=1 S. ev. Id!=0&&ev. Id!=1 Hong @ KAIST 23 / 18

Noise Injection Random Testing • To generate divers execution scenario, [Lei, 05] presents a technique that inserts random noise before message generation code in a message-passing program • Limitation: noise injection random testing may not find corner-case bugs with low possibility of detection 1 handler() call between mi and mi+1 1 void main() { while(…) { 2 CPU m 1 ; 3 m 2 ; m 1 ; 4 handler() … … 5 mi ; m 1 ; m. L ; 6 … 7 } } handler() mi ; 8 void handler() {…} mi+1; handler() 9 void ev. Generation() { … sleep(rand()); gen. Ev(); 10 m. L ; mi+1; sleep(rand()); gen. Ev(); … 11 Systematic Testing of Reactive Software with Non-deterministic handler() 2021 -06 -03 m. L; Events: } 12 A Case Study on LG Electric Oven 2 handler() calls between mi and mi+1 CPU m 1 ; … mi ; handler() mi+1; … S. Hong @m. KAIST L; 24 / 18

Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven Symbolic Variable & Probe Setup • A function init() declares each element of ev. Loc as a symbolic variable • init() is called before execution of main loop • is. In. Handler is a flag to prevent nested event handler executions • In the case study subject, nested event handler executions are not allowed • An event handler may call probe() because of functions that both main loop and the event handler call • If ev. Loc[iter] contains loc. Id, an event handler is called 2021 -06 -03 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 int ev. Loc[MAX_IT][MAX_EV]; void init() { for(i=0; i<MAX_IT; i++) for(j=0; j<MAX_EV; j++) CREST_int(ev. Loc[i][j]); } void probe(int loc. Id) { static int is. In. Handler=FALSE; if(!is. In. Handler) { for(j=0; j<MAX_EV; j++) { if(ev. Loc[iter][j]==loc. Id){ is. In. Handler = TRUE; event. Handler(); ev. Loc[iter][j] = DONE; is. In. Handler = FALSE; } } S. Hong @ KAIST 25 / 18

Target Electric Oven 8 -bit MCU 2021 -06 -03 Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven S. Hong @ KAIST 26 / 18

Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven Overwriting Bug CQ status queue. Full q. Array ==true 1 2 Main loop execution 3 1: dequeue() head. Idx tail. Idx queue. Full q. Array ==false 1 2 3 result tail. Idx head. Idx queue. Full q. Array ==true 4 2 3 result tail. Idx head. Idx 2021 -06 -03 6: queue. Full =false; event enqueue(4 ) 8: return result; S. Hong @ KAIST 27 / 18

Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven Illegal Transition Bug • The illegal transition occurs when the main loop handles multiple key events at the same time - The oven freezes due to the illegal state auto-cook & function-R cooking, manual-cook 2021 -06 -03 Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven S. Hong @ KAIST 28 / 18

Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven Bug Detection Time of SEG • The following table shows the error detection results of the illegal transition bug of random search strategy • DFS search strategy failed to detect the bug because of large search space • SVA detects the illegal transition bug faster than the other 2 strategy because SVA reduce search space by inserting less probes • SVA achieves higher coverage than the others because SVA negates more input-related branches than the other technique Technique MAX_EVENT=2 SEG 2021 -06 -03 MAX_EVENT=4 Detection Time # of executions Branch Coverage STMT 195. 10 3991. 40 50. 2% BB 172. 63 3800. 33 50. 0% SVA 113. 63 2932. 10 50. 9% STMT 208. 60 2920. 60 51. 8% BB 249. 43 3845. 10 51. 7% SVA 150. 47 2858. 57 52. 1% STMT 280. 03 3210. 60 52. 6% BB 249. 63 SVA 197. 93 3307. 43 52. 8% S. Hong @ KAIST 29 / 18 2594. 03 53. 4%

Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven State Transition Example 2021 -06 -03 S. Hong @ KAIST 30 / 18

Unit Testing Overview • Test scenario: the main loop performs dequeue(), and event handlers can perform enqueue() concurrently • Bug detection – Overwriting bug: unintentionally overwrite the oldest value in the queue with a new value – Inconsistency bug: unintentionally remove all values in the queue Circular. Queue status 1 void* dequeue() { queue. Full q. Array 2 void* result = NULL; ==false 1 2 3 if (!is. Empty()) { result = head. Idx; 4 head. Idx tail. Idx head. Idx=get. Next. Idx(…); 5 queue. Full = FALSE; 6 queue. Full q. Array 7 } 8 return result; ==false 1 2 3 9 } 2021 -06 -03 Systematic Testing of Reactive Software with Non-deterministic Events: tail. Idx head. Idx A Case Study on LG Electric Oven Main loop execution 5: head. Idx= get. Next. Idx(… ) event enqueue(3 ) 6: queue. Full =false S. Hong @ KAIST 31 / 18

Result with SEG Framework • Fully automated testing technique – The SEG approach requires than model checking and adhoc random testing • Effective in fault finding – Control event timing in a fine-grained manner – Generate all event scenarios systematically – Found 3 new bugs in LG electric-oven S/W and improved software quality • 2 bugs in unit-level testing of Circular. Queue • 1 bugs in integration-level testing of the main loop and the key event handlers 2021 -06 -03 Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven S. Hong @ KAIST 32 / 18