Testing the OPN Language Rule Coverage and Fuzz

Testing the OPN Language: Rule Coverage and Fuzz Testing Wujie Zheng 1

Outline n n The OPN Language Task 1: Rule Coverage Collection q q n Task 2: Fuzz Testing q n Verifying the OPN definitions A measurement of the test suite for the OPN compiler Test the type checker Conclusions and Future Work 2

The OPN Language n OPN (Open Protocol Notation) is a notation for describing architecture, messages, and behavior of network protocols. 3

The OPN Language n Components q q q OPN definitions The OPN compiler n Parser (and lexer) n Type checker n Code generator Runtime 4

The OPN Language n Components q q q OPN definitions The OPN compiler n Parser (and lexer) n Type checker n Code generator Runtime 5

The OPN Language Examples For Statements n n n Embedded. Statement : : = for ( For. Initializer ; Expression ; Statement. List , ? ) Embedded. Statement n For. Initializer : : = Variable. Declaration ( , Variable. Declaration )* | Statement. List , ? n Statement. List : : = Statement. Expression ( , Statement. Expression )* Endpoint Declarations n Endpoint. Declaration : : = n Visibility. Modifier* endpoint Identifier Endpoint. Address? Endpoint. Transports Aspects? n Contract. Element+ n ( { Endpoint. Member* } Aspects. After. Block? | Aspects? ; ) 6

Task 1: Rule Coverage Collection n Rule Coverage q q A criterion for testing grammars A test case is said to cover a grammar rule if that rule is used at least once in deriving the test case For example, The following test module test; int x; covers the grammar rule Document : : = module Qualified. Identifier Aspects? ; Using. Declaration* Module. Member* , and not cover the rule Document : : = protocol Qualified. Identifier Aspects? ; Using. Declaration* Protocol. Member* 7

Task 1: Rule Coverage Collection n Rule Coverage Collection q q q Define a set of grammar rules (sub-classes of each rule) based on the OPN definitions Implement an OPNRule. Printer to traverse the POM objects to get the rules covered by the test cases Report the rule coverage result 8

9

Task 1: Rule Coverage Collection n Purpose 1: verifying the OPN definitions q To collect the rule coverage, we need to identify the sub-rules of each grammar rule and visit each component of a rule, which can identify Missing definitions n Wrong definitions Some bugs can be identified by experience, some may due to conflicts between the definitions and the implementations n 10

Task 1: Rule Coverage Collection n Purpose 1: verifying the OPN definitions q Results: find 17 bugs in OPN definitions n Missing definitions Bug 6083: Definition of “Reference. Expression” missed in the specification Bug 15386: [Doc] Range expressions are implemented but not defined … n Wrong definitions Bug 14971: [Doc] the definition of Switch Statements misses Default. Case … 11

Task 1: Rule Coverage Collection n Purpose 2: provide a measurement of the test suite for the OPN compiler q q To understand how well the OPN grammar has been tested Compiler errors may occur in some cases of a grammar rule but not in other cases n Define sub-classes of each rule, so as to check corner cases For example, for the definition Document : : = protocol Qualified. Identifier Aspects? ; Using. Declaration* Protocol. Member* 6 rules are defined: Document_Protocol_Aspects, Document_Protocol_No. Aspects, Document_Protocol_No. Protocol. Member, Document_Protocol. Member, Document_Protocol_No. Using. Declaration, and Document_Protocol_Using. Declaration. 12

Task 1: Rule Coverage Collection n Purpose 2: provide a measurement of the test suite for the OPN compiler q Results: n n n Define 518 rules Measure the rule coverage of the Tests. Opn. Export. Roundtrip tests: 459/518=88. 6% A detailed rule coverage report 13

Task 2: Fuzz Testing n Fuzz Testing q q A software testing technique that provides invalid, unexpected, or random data to the inputs of a program. If the program fails (for example, by crashing, hanging, or failing built-in code assertions), the defects can be noted. 14

Task 2: Fuzz Testing n Randomly Mutate The Inputs Original Input Mutated Input 15

Task 2: Fuzz Testing n Fuzz Testing for the type checker q q Given a test input, randomly mutates some parts of it and generates more test inputs. Two kinds of mutation n Unstructured mutation q q n Mutate lines or words of an OPN input. Supports three operators: change, add, and remove. Structured mutation q q Mutate identifiers of a POM input (generated from an OPN input). Supports only the change operator now. 16

Task 2: Fuzz Testing n Fuzz Testing for the type checker q How to check the test result of mutated inputs? n n n The program under test (mainly the type checker) should not crash or hang. If a mutated input passes the type checker, it is suspicious The type checker should not throw exceptions without any error messages. A mutated input that increases the code coverage may deserve an investigation. Randomly inspect some mutated inputs and the execution results. 17

Task 2: Fuzz Testing n Fuzz Testing for the type checker q How to check the test result of mutated inputs? n Log the mutated inputs and the execution results for investigation q q q The mutated inputs and the changes Whether the mutated inputs pass/fail the type checker Detailed error messages 18

Task 2: Fuzz Testing n Fuzz Testing for the type checker q Results n Find 9 bugs in the type checker q Hangs Bug 14982: [Type. Checker] "endpoint U over U" crashes the type checker q Exceptions with error messages Bug 15015: [Type. Checker] type checker throws exception without any error message for code related to Aspects q Type checker passes Bug 15616: [Type. Checker] type checker should report errors for the comparison between 'int' and 'method‘ … 19

Task 2: Fuzz Testing n Fuzz Testing for the type checker q Experience n Structured mutation is much more effective than unstructured mutation q q Generate error-inducing inputs more quickly The mutated inputs are syntactically correct, so we can focus on testing the type checker More easy to identify suspicious results for manual inspection Change of identifiers can result in conflict of type information more easily, so it is more suspicious for the type checker to pass the mutated inputs 20

Conclusions and Future Work n Rule coverage collection is good for verifying OPN definitions q q n n Need to check the definitions carefully, otherwise don’t know how to implement the rule coverage collecting program Also check the consistence between the definitions and the implementations The test suites have achieved high rule coverage Future work q To define rules for the combinations of components of each rule 21

Conclusions and Future Work n Fuzz testing is good for verifying the type checker q q n Especially helpful for ensuring the robustness Also find semantic bugs Future work q To systematically mutate the POM objects using symbolic execution techniques 22

Thank you! Q&A 23