Quick Check A Lightweight Tool for Random Testing

Quick. Check: A Lightweight Tool for Random Testing of Haskell Programs By Koen Claessen, Juhn Hughes ME: Mike Izbicki

About Haskell Purely functional language Same function, same inputs -> same outputs Functions can have no side effects Variables cannot vary Lazy evaluation -> infinite data structures Popular uses Language research High frequency trading handles billions of dollars, so programs must be correct Mike Izbicki / Quick. Check 2

Testing in Haskell Many tests are not applicable or require significant reinterpretation Ex: What would control flow mean? Possible solutions: Small, obviously correct code Strong static typing Proofs of correctness Functional testing Mike Izbicki / Quick. Check 3

Quick. Check's goals Every function has invariants These are the specification Combine specification directly into source code Formal method that is very expressive Easily learned Programmer MUST pay attention Automatically create tests from the specification Mike Izbicki / Quick. Check 4

Specification example: primes Prime sieves generate prime numbers Sieve of Eratosthenes O ( N ) Sieve of Atkin O ( N / (log N) ) Wheel sieves O ( N / (log N) ) Difficult computations to understand, but there is a trivial verification algorithm That is our invariant It is much slower, but we only test it on some inputs Mike Izbicki / Quick. Check 5

Unconditional invariants Always easy to compute Ex: Code: Mike Izbicki / Quick. Check 6

Conditional invariants Sometimes easy to compute Ex: Sometimes hard to compute Ex: Mike Izbicki / Quick. Check 7

Generating the tests Testing simple data types, like ints, is easy Testing complex data types is hard What does a random list mean? Context free grammars? Size / contents / member data types Has been used before, but not powerful enough Attribute grammars? Can define any distribution over any data type By embedding in Haskell, we get this for free Mike Izbicki / Quick. Check 8

Special cases Quick. Check can create random, infinitely large data types Must be very careful to avoid infinite loops Quick. Check can create randomly generated functions Many functions in Haskell take other functions as arguments Ex: map : : (a -> b) -> [a] -> [b] Mike Izbicki / Quick. Check 9

Test Adequacy By simply using enough random tests, we can make our test suite arbitrarily adequate for any coverage criteria Mike Izbicki / Quick. Check 10

Case studies Unification problem Theorem proving Edison, a data structure library Lava, a tool to simulate and formally verify hardware Mike Izbicki / Quick. Check 11

Results: Edison Every data structure, and every function was tested Maintainer of the Edison library reports: Testing requires 25% of the effort without Quick. Check still found more errors Quick. Check not only finds errors in the code, but also errors in the specification Approximately a 50 / 50 split Mike Izbicki / Quick. Check 12

Results: Lava Previously proven correct Quick. Check did not find any errors Quick. Check was useful for speeding up the program Lava is an embedded language in Haskell Quick. Check also works on Lava Very slow process to verify circuits Can “Quick. Check” a circuit to rapidly to find trivial bugs Mike Izbicki / Quick. Check 13

Quick. Check in summary Random testing works well in functional programming languages Quick. Check forces programmers to write specifications Mike Izbicki / Quick. Check 14

Quick. Check in summary Random testing works well in functional programming languages Quick. Check forces programmers to write specifications Questions? Mike Izbicki / Quick. Check 15

Mike Izbicki / Quick. Check 16