Deriving formal specifications almost automatically Glenn Ammons and

Deriving formal specifications (almost) automatically Glenn Ammons and Ras Bodik and James R. Larus

Three pillars of formal verification • Model checkers and other verifiers – well automated (SLAM, Spin, type checkers, Vault) • Program abstractors – getting there (SLAM, Engler’s metacompiler) • Formal specifications – Written by hand – Our goal: bring automation to writing formal specifications

Deriving specs is feasible • Well-debugged software exists – Good code obeys the rules, but doesn’t state them clearly • Common behavior is good behavior – Because testing exposes common behavior • Programmers exist – But they don’t want to write specs!

Rules describe good behavior A rule is a nondeterministic finite automaton: S T = XNext. Event XSet. Selection. Owner(T) XGet. Selection. Owner F

Rules are derived from traces, with user guidance Xt. App. Next. Event() = event(type = 5, window = 22, time = 3: 15) Xt. Dispatch. Event(type = 5, window = 22, time = 3: 15) Xt. Free(NULL) Xt. Malloc(size = 8) = 0 x 10 Xmu. Intern. Strings(names = 0 x 20, count = 2, atoms_return = 0 x 10) Xt. Own. Selection(widget = 0 x 30, selection = 1, time = 3: 15) And so on: the more traces the better

Overview Traces Programs or traces (buggy? ) Seeds Abstraction prescription Rule learner Program abstractor Rules Matcher Bugs! Abstract programs or traces

Case study: selections in X 11 • The rule: Set. Selection. Owner must be passed a timestamp from an Xevent • 25 programs from the X 11 distribution and the contrib directories (all used selections) • Verification done over traces (not statically) • Found two bugs in 29 static uses • Found three benign violations

To do • • Static checking: typestates Better simplifier Better user interaction What else can we learn? – Protocols like socket/bind/accept/close – Operations on data structures

Power What else can we do with this stuff? Compare with Ernst

Detailed figure?

Running example

Testing vs. verification • Examines the complete • Examines only some aspects of programs • Examines some inputs • Examines all inputs • For better coverage, write more specs write more test cases The practice sees writing test cases as easier than writing formal models and specifications, so testing dominates.