MOP An Efficient and Generic Runtime Verification Framework

MOP: An Efficient and Generic Runtime Verification Framework Feng Chen and Grigore Rosu University of Illinois at Urbana-Champaign OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu

Monitoring of Formal Specifications in Software Runtime verification (RV 2001 – 2007) – Observe a run of the system • Instrumentation – Check it against desired properties • Explicit or implicit properties – React /report if needed • Error messages, throw exceptions, recover OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 2

Runtime Verification Systems • ≤ 2001 – MAC (UPenn), PAX* (NASA), Time. Rover (commercial) • 2002 -2004 – HAWK/Eagle (NASA), MOP* (UIUC), POTA (UTA) • ≥ 2005: – PQL (Stanford), Tracematches (Oxford), PTQL – PEX (Microsoft) – MOP* (added universal parameters, more logics) OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 3

Overview • What is MOP – Philosophy, Examples, Features • • Decentralized Monitors and Indexing Experiments and Evaluation Current limitations Conclusions and future work OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 4

(MOP)* Monitoring-Oriented Programming http: //fsl. cs. uiuc. edu/mop - proposed in 2003 – RV’ 03, ICFEM’ 04, RV’ 05, CAV’ 05, TACAS’ 05, CAV’ 06, CAV’ 07… *(not to be confused with Meta-Object Protocol) OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 5

What is MOP? • Framework where the benefits of monitoring specifications may encourage us to use them – Monitoring is basic design discipline – Recovery allowed and encouraged – Provides to programmers and hides under the hood a large body of formal methods knowledge/techniques – Generic for different application domains • Tool supported – Java. MOP – “compiles” MOP for Java into Aspect. J OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 6

Overview • What is MOP – Philosophy, Examples, Features • • Decentralized Indexing Experiments and Evaluation Current limitations Conclusions and future work OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 7

Java. MOP Example: “Enforce authentication before use” class Resource { /*@ scope = class logic = PTLTL { Event authenticate: end(exec(* authenticate())); Event use: begin(exec(* access())); Formula : use -> <*> authenticate } violation Handler { @this. authenticate(); } @*/ void authenticate() {. . . } void access() {. . . }. . . OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu } 8

Java. MOP Example: Correcting method matching Method open. Reg. Key should be followed by close. Reg. Key, not by close. Handle /*@ scope = class logic = ERE { Event open. Reg. Key: end(exec(* open. Reg. Key())); Event close. Handle: begin(exec(* close. Handle())); Formula : (open. Reg. Key close. Handle)* } validation handler { @this. close. Reg. Key(); return; } @*/ OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 9

Java. MOP Example: Profiling How many times a file is open, written to, and then closed? /*@ scope = class logic = ERE [ int count = 0; int writes = 0; ] { Event open : end(call(* open(. . ))) {writes = 0; }; Event write : end(call(* write(. . ))) {writes ++; } ; Event close : end(call(* close(. . ))); Formula : (open write+ close)* } violation handler{ @RESET; } validation handler{ count ++; File 2. log(count + ": " + writes); } @*/ OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 10

Multi-Parameter Specifications • Until now, MOP was limited to oneparameter specifications; the examples above showed only such specifications: parameter = current object • Because of that restriction monitor states were always stored as fields into the monitored objects in MOP • Situation is more complex for multipleparameter specifications … OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 11

Fail Fast Iterators in Java • Following code throws exception in Java: Vector v = new Vector(); v. add(new Integer(1)); Iterator i = v. iterator(); v. add(new Integer(2)); • No exception raised if one uses Enumeration instead of Iterator – Java language decision, showing that properties referring to sets of objects are important OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 12

Java. MOP Example: Safe Enumeration • (borrowed from Tracematches) – We thank Tracematches team for their useful collection of regular pattern properties! • Safety property: create next* updatesource+ next – If next. Element() invoked on an enumeration object, then the corresponding collection (vector) is not allowed to change after the creation of the enumeration object OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 13

Multiple-Parameters in Java. MOP Safe Enumeration /*@ scope = global logic = ERE Safe. Enum(Vector v, Enumeration+ e) { Event create<v, e> : end(call(Enumeration+. new(v, . . ))) with e; Event updatesource<v> : end(call(* v. add*(. . ))) / … ; Event next<e> : begin(call(Object e. next. Element())); Formula : create next* updatesource+ next } validation handler { System. out. println("Error. . . "); } @*/ Java. MOP generates >200 Aspect. J LOC from above OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 14

Overview • What is MOP – Philosophy, Examples, Features • • Decentralized Monitors and Indexing Experiments and Evaluation Current limitations Conclusions and future work OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 15

MOP Features: Extensible logic framework • Observation: no silver-bullet logic for specs • MOP logic plugins (the “How”): encapsulate monitor synthesizers; so far we have plugins for – – – ERE (extended regular expressions) Pt. LTL (Past-time LTL) and Ft. LTL (Future-time LTL) ATL (Allen temporal logic), JML (fragment of Java modeling language); NEW: CFG and pt. Ca. Ret plugins (need stack) • Generic universal parameters – Allow monitor instances per groups of objects OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 16

MOP Features: Configurable monitors Working scope (the “Where”) – – – Check point: check spec at defined place Method: pre-post condition Class: check spec everywhere during obj lifetime Interface: check spec at boundaries of methods Global: may refer to more than one object Running mode – Inline: shares resources with application – Outline: communicates with application via sockets – Offline: generated monitor has random access to log OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 17

MOP Features: Programmable Actions • • Monitor variables Event-triggered actions (any code) Violation/validation handlers (any code) Logic-based AOP instance – Aspects are formal specifications – MOP aim: generate optimal AOP code OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 18

Overview • What is MOP – Philosophy, Examples, Features • • Decentralized Monitors and Indexing Experiments and Evaluation Current limitations Conclusions and future work OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 19

Decentralized Monitoring/Indexing (works only for inline MOP) • The problem: how to monitor a universally quantified specification efficiently! create<v, e> udatesource<v> next<e> ( v, e) create next* updatesource+ next OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 20

“Super-Logic” Approach • Eagle, PQL, PTQL, Tracematches, … • Choose a universally quantified logic • Device monitor synthesizers for it: ( p) Mon( p) • Doable in MOP, too, but against its spirit – Logic plugins only know how to generate monitors for – No relationship between monitors for and for ( p) OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 21

MOP’s Decentralized Monitors M p 2 M p 3 Monitor instances (one per parameter instance) … M OOPSLA’ 07 – 25 Oct 2007 M p 1000 MOP – Grigore Rosu 22

Indexing … • The next problem: how can we retrieve all needed monitor instances efficiently? udatesource<v> M M v, e 2 v, e 1 M p 1 … M p 1000 Naïve implementation can be very inefficient (both time- and memory-wise) OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 23

MOP’s Centralized Indexing • One index tree per parameter set • Weak references Safe. Enum events create<v, e> udatesource<v> next<e> OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 24

MOP’s Decentralized Indexing • Monitors scattered all over the code • Monitor states piggybacked to object states Safe. Enum events create<v, e> udatesource<v> next<e> OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 25

Overview • What is MOP – Philosophy, Examples, Features • • Decentralized Monitors and Indexing Experiments and Evaluation Current limitations Conclusions and future work OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 26

The Java. MOP Implementation Short story: 1. Parse the MOP specifications – Generate all monitors using the logic plugins – Find instrumentation points 2. Generate standard Aspect. J code 3. Wave everything (observation, monitoring, handlers) together using Aspect. J compiler OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 27

MOP Evaluation on Da. Capo: Statistics OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 28

MOP Evaluation on Da. Capo: Runtime Overhead Summary: MOP runtime overhead larger than 10% in only 8% of programs OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 29

Evaluation of MOP on TM • MOP is generic (logic, etc. ), but its generality does not come at a performance cost! • Following programs part of Tracematches’ (TM) benchmarks (hand optimized by TM) OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 30

Overview • What is MOP – Philosophy, Examples, Features • • Decentralized Monitors and Indexing Experiments and Evaluation Current limitations Conclusions and future work OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 31

Limitations of the Current Implementation of Java. MOP • No nested or existential parameters • Monitor creating events must contain all the parameters of the specification • No support for real-time • Inherit the limitations of Aspect. J – Would like to have dynamic aspects … – Though shouldn’t miss events that trigger actions OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 32

Conclusions and Future Work • MOP is a generic, yet efficient RV framework • Experiments showed that RV is feasible • Proposed a logic-independent approach to support universally quantified specifications – With both centralized and decentralized indexing • Future work – Use MOP for improved and rigorous testing • Test case generation, sliced causality – Real-time logic plugins – Static analysis (plugins) to reduce runtime overhead OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 33

Related Work Extensively discussed in paper • Programming languages – PQL (Lam et al. , OOPSLA ’ 05) – Tracematches (de Moor et al. , OOPSLA ’ 05, ‘ 07) – PTQL (Goldsmith et al. , OOPSLA’ 05) • Runtime verification – – – MAC (Lee et al. , RV’ 01) PAX* (Havelund, Rosu, RV’ 01) Time. Rover (Drusinski, RV’ 02) HAWK/Eagle (Barringer et al. , VMCAI’ 04) PEX (Schulte, Tilman; Microsoft) OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 34

OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 35

Backup slides OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 36

Application to Verification Here is an MOP approach to validate a sorting program: /*@ Fancy. Sort @*/ OOPSLA’ 07 – 25 Oct 2007 logic = JML is. Sorted? { ensures is. Sorted(array); } Violation No Handler { insertion. Sort(array); } Insertion. Sort MOP – Grigore Rosu Yes 37

Monitoring-based Verification? Monitor & Recover: presumably reduces the complexity of program verification Architecture Correct Complex Component φc Array elements unchanged Monitor Incorrect Correct sorting φs φm Is sorted? Simple Component φ p= φc & (φm / φs ) OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 38

Why Monitoring I • Monitoring is well-adopted in many engineering disciplines – Fuses, watchdogs, fire-alarms, etc. • Monitoring adds redundancy – Increases reliability, robustness and confidence in correct behavior, reduces risk OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 40

Why Monitoring II • Ideally, one would like to prove a program correct with respect to its specification … – Known to be hard • Provably correct systems can fail, too – Unexpected environment, wrong/strong assumptions, hardware or OS errors, etc. OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 41

Monitoring of Formal Specifications in Software Runtime verification (RV 2001 – 2007) – Observe a run of the system • Instrumentation – Check it against desired properties • Explicit or implicit properties – React /report if needed • Error messages, throw exceptions, recover OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 42

MOP Conceptual Instances MOP[logic, working-scope, running-mode] • JPa. X = MOP[LTL, class, outline/offline] – NASA (joint work with Havelund) • Temporal. Rover = MOP[Mi. TL, class, inline] – Commercial (Drusinski) • MAC = MOP[Past. LTL, class, inline/outline] – UPenn (Lee, Sokolski, . . . ) • Hawk = MOP[Eagle, global, inline] – NASA (Havelund and collaborators) • Tracematches = MOP[Reg. Exp, global, inline] – Oxford (de Moor and students) • PQL = MOP[CFG, global, inline] – Stanford (Lam and students) • PTQL = MOP[SQL, global, outline] – Berkeley, Stanford (Aiken and students) OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 43

Why “Monitoring-Oriented” ? • We think one can write reliable programs if one adopts a “monitor and recover” style – Not a substitute for writing good code upfront – Write good code, but, “just in case” also say what to do if requirements are violated/validated • MOP tools provide support for that • … may reduce complexity of verification: prove what is easy, monitor the rest OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 44

MOP Approach to Monitoring: Divide & Conquer • What properties/specifications to check? – Combine formal specification and implementation • Where to check it? – Automatic monitor integration • Currently using Aspect-Oriented Programming • How to express the property? – Declarative, automatic monitor synthesis • What if specification violated or validated? – Execute user provided “recovery” code OOPSLA’ 07 – 25 Oct 2007 MOP – Grigore Rosu 45