An Integrated Data Model Verifier with Property Templates

An Integrated Data Model Verifier with Property Templates Jaideep Nijjar Ivan Bocic Tevfik Bultan {jaideepnijjar, bo, bultan}@cs. ucsb. edu University of California, Santa Barbara Department of Computer Science Verification Lab Formali. SE 2013

Web Software Everywhere • Commerce, entertainment, social interaction • We will rely on web apps more in the future • Web apps + cloud will make desktop apps obsolete

Model-View-Controller Pattern DB Model • MVC pattern has become the standard way to structure web applications • • • Ruby on Rails Zend for PHP Cake. PHP Struts for Java Django for Python … View • Benefits of the MVC pattern: Controller • Separation of concerns • Modularity • Abstraction

A Data Model Verification Approach MVC • Ruby on Rails Application MVC Design Principles Data Model • Active. Records Automatic Extraction Formal Model Add data model properties • Sets and Relations Verification • Bounded and Unbounded

i. Da. Ver: Integrated Data Model Verifier Active Records Properties (expressed using Templates) Choice of technique Model Extraction formal data model + property + technique Verification Bounded (Alloy) Property Verified Unbounded (FOL Thm Prover) Property Failed + Counterexample Unbounded (SMT Solver) Unknown

Outline • Motivation • Overview of Approach • Rails Data Models • Basic Relations • Options to Extend Relations • Formalization of Semantics • • Property Templates Verification Techniques Case Study Conclusions and Future Work

A Rails Data Model Example class User < Active. Record: : Base has_and_belongs_to_many : roles has_one : profile, : dependent => : destroy has_many : photos, : through => : profile end class Role < Active. Record: : Base has_and_belongs_to_many : users end class Profile < Active. Record: : Base belongs_to : user has_many : photos, : dependent => : destroy has_many : videos, : dependent => : destroy, : conditions => "format='mp 4'" end class Tag < Active. Record: : Base belongs_to : taggable, : polymorphic => true end class Video < Active. Record: : Base belongs_to : profile has_many : tags, : as => : taggable end class Photo < Active. Record: : Base. . . Role * * User 1 1 * 0. . 1 Photo 1 Taggable * Tag * 1 Profile 1 format=. ‘mp 4’ 1 * Video

Rails Data Models • Data model verification: Analyzing the relationships between data objects • Specified in Rails using association declarations inside the Active. Record files • The basic relationships • One-to-one • One-to-many • Many-to-many • Extensions to the basic relationships using Options • : through, : conditions, : polymorphic, : dependent

The Three Basic Relationships in Rails • One-to-One class User < Active. Record: : Base has_one : profile end User 1 . class Profile < Active. Record: : Base belongs_to : user end 1 Profile . • One-to-Many class Profile < Active. Record: : Base has_many : videos end Profile 1 . class Video < Active. Record: : Base belongs_to : profile end * Video

The Three Basic Relationships in Rails • Many-to-Many class User < Active. Record: : Base has_and_belongs_to_many : users end class Role < Active. Record: : Base has_and_belongs_to_many : roles end User * * Role

Options to Extend the Basic Relationships • : through Option • To express transitive relations • : conditions Option • To relate a subset of objects to another class • : polymorphic Option • To express polymorphic relationships • : dependent Option • On delete, this option expresses whether to delete the associated objects or not

The : through Option class User < Active. Record: : Base has_one : profile has_many : photos, : through => : profile end class Profile < Active. Record: : Base belongs_to : user has_many : photos end class Photo < Active. Record: : Base belongs_to : profile end Profile 0. . 1 1 * 1 User 1 * Photo

The : dependent Option class User < Active. Record: : Base has_one : profile, : dependent => : destroy end class Profile < Active. Record: : Base belongs_to : user has_many : photos, : dependent => : destroy end User 1 0. . 1 Profile 1 * Photo • : delete directly delete the associated objects without looking at its dependencies • : destroy first checks whether the associated objects themselves have associations with the : dependent option set

Formalizing Rails Semantics Formal data model: M = <S, C, D> • S: The sets and relations of the data model (data model schema) • e. g. { Photo, Profile, Role, Tag, Video, User} and the relations between them • C: Constraints on the relations • Cardinality constraints, transitive relations, conditional relations, polymorphic relations • D: Dependency constraints • Express conditions on two consecutive instances of a relation such that deletion of an object from the first instance leads to the other instance

Outline • • Motivation Overview of Approach Rails Data Models Property Templates Verification Techniques Case Study Conclusions and Future Work

Property Templates • User-friendly and makes it easy to express properties • Manually writing properties is a time-consuming and error-prone process • Requires familiarity with input modeling language of solver • Templates are language-neutral • Do not require familiarity with SMT-LIB, Spass and Alloy languages, and understanding of output specifications • Make it easy to rerun the tool and switch the verification technique, by not requiring the user to rewrite the property • Eight property templates available for the most common data model properties

Property Templates • always. Related[class. A, class. B] • To check that objects from class. A are always related to objects of class. B • some. Multiple. Related[class. A, class. B] • To check that it is possible for objects of class. A to be related to more than one object of class. B • some. Unrelated[class. A, class. B] • To check that it is possible for an object of class. A to not be related to any objects from class. B • transitive[class. A, class. B, class. C] • To check that the relation between class. A and class. C is the composition of the relations between class. A and class. B, and class. B and class. C

Property Templates • no. Dangling[class. A, class. B] • To check that when an object of class. A is deleted, there are no objects from class. B that are left with a dangling reference to the deleted object • delete. Propagates[class. A, class. B] • To check that when an object of class. A is deleted, related objects in class. B are also deleted • no. Delete. Propagation[class. A, class. B] • To check that when an object of class. A is deleted, related objects in class. B are not deleted • no. Orphans[class. A, class. B] • To check that deleting an object from class. A does not cause related objects in class. B to be orphaned • An orphan is an object that is related to no other object

Outline • • • Motivation Overview of Approach Rails Data Models Property Templates Verification Techniques • Bounded Verification with Alloy • Unbounded Verification with SMT Solver • Unbounded Verification with FOL Theorem Prover • Case Study • Conclusions and Future Work

Active Records Choice of technique Properties i. Da. Ver UNBOUNDED VERIFICATION Model Extraction SMT-LIB Encoder formal data model + property formula SMT Solver BOUNDED VERIFICATION Alloy Encoder Results Interpreter Property Verified Results Interpreter formula Alloy Analyzer FOL Encoder instance or unsat Property Failed + Counterexample Unknown Results Interpreter instance or unsat or unknown formula Theorem Prover proof found or completion found or timeout

Active Records Choice of technique Properties Bounded Verification UNBOUNDED VERIFICATION Model Extraction SMT-LIB Encoder formal data model + property formula SMT Solver BOUNDED VERIFICATION Alloy Encoder Results Interpreter formula Alloy Analyzer FOL Encoder instance or unsat Results Property Verified Property Failed + Counterexample Unknown Interpreter instance or unsat or unknown formula Theorem Prover proof found or completion found or timeout

Alloy Language • A declarative specification language for object modeling • Based on first-order logic • Set-based representation of objects • Defines sets of objects using signatures (sigs) • Defines relations using fields inside the signatures • Add additional constraints about the model as facts • Well-suited formally specifying data models • Can add assertions to specify properties about the specification

Alloy Analyzer • Automated verification of Alloy specifications is undecidable if the domains are not bounded • To ensure decidability, Alloy Analyzer restricts the domains to a finite scope • User-specified • A finite bound on the sizes of the domains • SAT-based bounded verification • Alloy Analyzer translates the Alloy verification query to a Boolean logic formula satisfiability query • Then invokes an off-the-shelf SAT-solver

Sample Translation to Alloy class User < Active. Record: : Base has_one : profile end class Profile < Active. Record: : Base belongs_to : user end sig Profile {} sig User {} one sig State { profiles: set Profile, users: set User, relation: Profile lone -> one User }

Bounded Verification of Data Models • Automatically translate the formal data model extracted from the Active Records and the property to Alloy • User may specify a bound or use the default • Use Alloy Analyzer to perform bounded verification • Possible outputs: • Assertion holds within the given bound • A counterexample proving falsified assertions Formal data model + Property + Bound BOUNDED VERIFICATION formula Alloy Analyzer instance or unsat Property Verifies (within bound) Alloy Encoder Results Interpreter Property Fails + Counterexample

Active Records Choice of technique Properties Unbounded with SMT Solver UNBOUNDED VERIFICATION Model Extraction SMT-LIB Encoder formal data model + property formula SMT Solver BOUNDED VERIFICATION Alloy Encoder Results Interpreter formula Alloy Analyzer FOL Encoder instance or unsat Results Property Verified Property Failed + Counterexample Unknown Interpreter instance or unsat or unknown formula Theorem Prover proof found or completion found or timeout

Satisfiability Modulo Theories (SMT) • SMT-solvers automatically check the satisfiability of first-order formulas with respect to a set of theories • Typical theories include: Linear Arithmetic, Arrays, Bit Vectors, Equality with Uninterpreted Functions • Only implicit universal quantification • Theory of Equality with Uninterpreted Functions. • Language: Variables, Constants, Uninterpreted function symbols, Predicate ‘=‘, and Boolean connectives • Uninterpreted functions have no properties except their signature and functional consistency: a = b => F(a) = F(b) • Example formula: F(x) = F(G(y)) ˅ H(x, y) = 1

SMT Solvers • There are many SMT solvers out there that support popular theories • However many of them are not suitable for us • we need support for quantified formulas to handle the property templates • Microsoft’s Z 3 supports quantified expressions in theory of uninterpreted functions • Uses heuristics for eliminating quantifiers in formulas • May return ‘unknown’ during satisfiability check

SMT-LIB • • Defines a standard input language for SMT Solvers Defines theories and logics in which formula can be written Lisp-like format: Specifications are a series of s-expressions Declare types using declare-sort command • (declare-sort User) • Declare uninterpreted functions using declare-fun command • (declare-fun foo (Domain) Range) • Quantifier commands: (forall )and (exists ) • Add constraints and properties using assert command • Check satisfiability by using (check-sat) command

Sample Translation to SMT-LIB class User < Active. Record: : Base has_one : profile end class Profile < Active. Record: : Base belongs_to : user end (declare-sort User) (declare-sort Profile) (declare-fun relation (Profile) User) (assert (forall ((p 1 Profile)(p 2 Profile)) (=> (not (= p 1 p 2)) (not (= (relation p 1) (relation p 2) ))

Unbounded Verification using SMT Solvers • Automatically translate formal data model and property into theory of uninterpreted functions with quantification (SMT-LIB) • Use the SMT solver Z 3 to perform satisfiability check • For failing assertion properties, our tool interprets outputs and forms a counterexample • Unknowns (and timeouts) possible since theory is undecidable Formal data model + Property UNBOUNDED VERIFICATION Property Verified SMT-LIB Encoder formula SMT Solver instance or unsat or unknown Results Interpreter Property Failed + Counterexample Unknown

Active Records Choice of technique Properties Unbounded Verification UNBOUNDED VERIFICATION Model Extraction SMT-LIB Encoder formal data model + property formula SMT Solver BOUNDED VERIFICATION Alloy Encoder Results Interpreter Property Verified Results Interpreter formula Alloy Analyzer FOL Encoder instance or unsat Property Failed + Counterexample Unknown Results Interpreter instance or unsat or unknown formula Theorem Prover proof found or completion found or timeout

FOL Theorem Provers • Rails data models and properties are expressible in first-order logic (FOL) with equality and quantifiers • Note that this is an undecidable theory • There automated theorem provers for first-order logic • They use search strategies to find proofs • However due to undecidability of the FOL they cannot always give a definite answer • We use the FOL theorem prover, Spass

Modeling Active Records using FOL • Model object types by declaring a unary predicate • Returns true if an object is a member of that class • Model relations between data objects using binary predicates • Returns true if the two objects are related • Axioms are used to express constraints on the data model, e. g. • Specifying cardinality, dependency, and transitivity constraints on relations • Specifying that predicates denoting classes not related by inheritance are mutually exclusive • Conjectures are used to model the property to be checked • To verify the property holds on the data model, send the following formula to theorem prover: axioms => conjecture

Sample Translation to Spass list_of_symbols. sorts[Profile, User]. predicates[(relation, 2)]. end_of_list_of_formulae(axioms). formula(forall([Profile(a)], not(User(a)))). formula(forall([User(a)], not(Profile(a)))). formula(forall([a, b], implies( relation(a, b), and(Profile(a), User(b))))). formula(forall([a, b 1, b 2], implies( and(relation(b 1, a), relation(b 2, a)), equal(b 1, b 2)))). formula(forall([a, b 1, b 2], implies( and(relation(a, b 1), relation(a, b 2)), equal(b 1, b 2)))). formula(forall( [Profile(a)], exists([b], relation(a, b)) )). end_of_list.

Unbounded Verification of Data Models using Theorem Provers • Automatically translate formal data model and property into Spass’s input language (FOL) • Use theorem prover Spass to see if formula provable • Interpret results to determine whether property holds • Does not produce counterexamples since theorem provers are not designed to do so • If Spass times out (due to undecidability of theory), i. Da. Ver returns ‘Unknown’ Formal data model + Property UNBOUNDED VERIFICATION formula Theorem Prover proof found or completion found or timeout Property Verified FOL Encoder Results Interpreter Property Failed Unknown

Outline • • Motivation Overview of Approach Rails Data Models Property Templates Verification Techniques Case Study Conclusions and Future Work

Case Study • Lovd. By. Less, a social networking application • LOC: 3787 • Number Active Record files: 13 • i. Daver input: • Path of the directory containing the Active Record files • Name of the file containing the properties to check (Expressed using property templates!) • Verification technique

Case Study • Check: always. Related[Photo, Profile] • Solver: Spass + Unbounded verification − No sample instance − May report unknown or timeout

Case Study • Check: some. Unrelated[Forum. Topic, Forum. Post] • Solver: Z 3 + Unbounded verification + Sample instance − May report unknown or timeout

Case Study • Check: delete. Propagates[Profile, Photo] • Solver: Alloy

Case Study • Check: delete. Propagates[Profile, Photo] • Solver: Alloy + Counterexample data model instance + Always returns a result (for small domains) − Bounded − Slower

Summary of Technique Pros and Cons Alloy SMT Solvers Theorem Provers Unbounded Verification No Yes Produces Counterexample Yes No Generates Unknown No Yes Slow Fast* Speed *In particular, Spass was slightly faster than Z 3 in our experiments and timed out less frequently

Outline • • Motivation Overview of Approach Rails Data Models Property Templates Verification Techniques Case Study Conclusions and Future Work

Conclusions and Future Work • It is possible to extract formal specifications from MVC-based data models and analyze them • We were able to find data model errors in real-world applications using some of these techniques (ISSTA’ 11, ASE’ 12) • Integration of multiple automated verification tools makes overall approach more flexible • Property templates simplify property specification • We have some recent work on • automated property inference (ISSTA’ 13) • analyzing actions that update the data store (submitted) • Main goal: Verifiable data model specification!

Questions?

Related Work • Verification of Web Applications • [Krishnamurti et al, Springer 2006 ] focuses on correct handling of the control flow given the unique characteristics of web apps • Works such as [Hallé et al, ASE 2010] and [Han et al, Mo. DELS 2007] use state machines to formally model navigation behavior • In contrast to these works, we focus on analyzing the data model • Formal Modeling of Web Applications • Web. Alloy [Chang, 2009]: user specifies the data model and access control policies; implementation automatically synthesized • Web. ML [Ceri et al, Computer Networks 2000]: a modeling language developed specifically for modeling web applications; no verification • In contrast, we perform model extraction (reverse engineering)

Related Work • Verification of Ruby on Rails applications • Rubicon [Near et al, FSE 2012] verifies the Controller whereas we verify the Data Model • Requires manual specification of application behavior, whereas we verify manually written properties • Limited to bounded verification • Data Model Verification using Alloy • [Cunha and Pacheco, SEFM 2009] maps relational database schemas to Alloy; not automated • [Wang et al, ASWEC 2006] translates ORA-SS specifications to Alloy, and uses the Analyzer to produces instances of the data model to show consistency • [Borbar et al, Trends 2005] uses Alloy to discover bugs in browser and business logic interactions

Related Work • Unbounded Verification of Alloy Specifications using SMT Solvers • [Ghazi et al, FM 2011], approach not implemented • More challenging domain since Alloy language contains constructs such as transitive closures • Specification and Analysis of Conceptual Data Models • [Smaragdakis et al ASE 2009, Mc. Gill et al ISSTA 2011] use Object Role Modeling to express data model and constraints • Focus is on checking consistency and producing test cases efficiently • Using Patterns to Facilitate Formal Property Specification • First proposed for temporal logic properties [Dwyer et al ICSE 1999] • The templates we present are not temporal and are specific to data model analysis