Software Reliability Methods Sorin Lerner Software reliability methods

Software Reliability Methods Sorin Lerner

Software reliability methods: issues • What are the issues?

Software reliability methods: issues • What is software reliability? How to measure it? – Bug counts ? Will we ever have bug-free software? – How many 9’s ? – Service Level Agreements ? • What is a bug? – – Adherence to specifications But what is a specification… User unhappy: is that a bug? Different levels of severity

Software reliability methods: issues • Cost of the methods for achieving reliability – Independently develop 5 versions of the software, run them all in parallel ) less likely that they fail at the same time in the same way. But… cost… is… high – For tools, cost of development of the tools • Burden on the programmer – fully automated vs. semi-automated methods – allow progressive adoption

Software reliability methods: issues • Level of guarantee provided by the method – Hard guarantees, statistical guarantees, no formal guarantee – What if tool is broken: trusted computing base • When is the method used? – compile-time, link-time, load-time, run-time • What does the tool see? – source code, assembly, the whole program or part of the program

One way of dividing the spectrum if (…) { x : = …; } else { y : = …; } …; Compiler 0100101 1010010 1011011

One way of dividing the spectrum Static techniques if (…) { x : = …; } else { y : = …; } …; Testing techniques Compiler 0100101 1010010 1011011 Run-time techniques

Static Techniques if (…) { x : = …; } else { y : = …; } …; Code satisfies spec? Spec « ¬ $ r t l • Spec: says what code should and should not do • Complete spec: specifies all behaviors (hard to formalize) • Incomplete spec: only defines some behaviors – e. g. “no null derefs”, “requests received are eventually processed” • Many formalisms exist for specs (Pre/Post conditions, FSMs, Temporal Logic, Abstract State Machines etc. )

Clean. L TSys DSL Static Techniques if (…) { x : = …; } else { y : = …; } …; Code satisfies spec? Spec « ¬ $ r t l • Language Design – Clean language design – Type Systems – Domain-specific languages – … DFA WP/SP MC ATP • Program Analysis Interaction between the two – – Dataflow analysis WP/SP Model checking Automated Theorem Proving – …

ESC/Java [Leino et al PLDI 2002] Clean. L TSys DSL DFA WP/SP MC ATP • Programmer annotates code with pre- and postconditions, tool verifies that these hold Automated Theorem Prover object Foo { //@ PRE (FORMULA) method bar(. . . ) { ) WP(POST, bar) = weakest condition Q such that Q at entry to bar establishes POST at exist . . . Compute Weakest Precondition } //@ POST (FORMULA) }

Rhodium [Lerner et al POPL 2005] Clean. L TSys DSL DFA WP/SP MC ATP Compiler Parser DSL Opt Checker Code Gen

Rhodium [Lerner et al POPL 2005] Rdm Opt VCGen Optdependent Clean. L TSys DSL DFA WP/SP MC ATP Opt-independent Lemma For any Rhodium opt: If Local VC is true Then opt is OK Local VC Checker f oo ¬ Pr « $ r l t Automatic Theorem Prover

ESP [Das et al PLDI 2002] Clean. L TSys DSL DFA WP/SP MC ATP Interface usage rules in documentation – Order of operations, data access – Resource management – Incomplete, wordy, not checked Violated rules ) crashes – Failed runtime checks – Unreliable software

Clean. L TSys DSL ESP [Das et al PLDI 2002] C Program DFA WP/SP MC ATP Rules ESP Safe Not Safe

ESP [Das et al PLDI 2002] Clean. L TSys DSL DFA WP/SP MC ATP • ESP is a program analysis that keeps track of object state at each program point – e. g. : is file handle open or closed? • Challenge: scale to large programs – One of scalability issues: merge nodes – Always analyze both sides of merge node ) exponential (or non-terminating) program analyses • ESP has a heuristic for handling merges that – avoids exponential blow-up and runs fast in practice – maintains enough precision to verify programs

BLAST [Henzinger et al POPL 2000] Interface usage rules in documentation – Order of operations, data access – Resource management – Incomplete, wordy, not checked Violated rules ) crashes – Failed runtime checks – Unreliable software

BLAST [Henzinger et al POPL 2000] C Program Rules BLAST Safe Error Trace

BLAST [Henzinger et al POPL 2000] start with a set of predicates Rules BLAST Perform “Predicate Abstraction” C Program Refine set of predicates Trace infeasible No errors found error trace found Safe augmented set of predicates Analyze trace Trace feasible Error Trace

BLAST [Henzinger et al POPL 2000] start with a set of predicates Rules augmented set of predicates Refine set of predicates Trace infeasible No errors found error trace found Safe DFA WP/SP MC ATP BLAST Perform “Predicate Abstraction” C Program Clean. L TSys DSL Analyze trace Trace feasible Error Trace

Type Systems Clean. L TSys DSL DFA WP/SP MC ATP • What is a type system? • A discipline for writing code that can be mechanically checked, and can prevent certain kinds of run-time errors • For example, java type system prevents calling methods that don’t exists, or calling methods with parameters of the wrong type

Type Systems Clean. L TSys DSL DFA WP/SP MC ATP • Type systems can track and provide guarantees about many other aspects of computation: – Safe explicit memory management (Crary, Walker and Morrisett, POPL 99) – Execution time bounds (Crary and Weirich, POPL 00) – Information flow (Myers, POPL 00) – Security automata (Walker, POPL 00)

Type Systems Clean. L TSys DSL DFA WP/SP MC ATP • Multi. Java [Clifton et al 2000] adds to Java: – multi-methods: methods that dispatch symetrically on the type of all params, not just the first – open classes: classes • Adding these features makes modular type checking harder, and required innovations on the type system side • Interplay between language design and type systems