Verification Setting the Stage Presented by Doron Drusinsky

Verification – Setting the Stage Presented by Doron Drusinsky Joint work with Bret Michael, Man-Tak Shing, and Tom Otani Naval Postgraduate School Monterey, CA September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 1

Verification – Setting the Stage Validation – getting the right product Verification – getting the product right A. Validation overview 1. Role of assertions 2. Sources of assertions 3. Purpose of validation 4. What can be wrong with my assertion? 5. Validation testing patterns 6. SRM and its role B. Verification – discussion September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 2

Role of Correctness Property Assertions Example: September 11, 2008 Assertion vs. implementation: its not about the language (e. g. LTL vs Java) Implementation: NASA IV&V Facility Workshop on Verification Morgantown, WV 3

Role of Correctness Property Assertions vs. implementation: its not about the language (e. g. LTL vs Java) Assertions: September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 4

Role of Correctness Property Assertions Our focus: assertion for reactive (sub)systems – Assertions for sequencing, time-constraints, and time-series constraints Counter example: the accuracy of (fixed-point function) f should be 0. 01 Transformational system: inputs … f in September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV output(s) out – done! Time 5

Role of Correctness Property Assertions Our focus: assertion for reactive (sub)systems – Assertions for sequencing, time-constraints, and time-series constraints Example: when cruise control is active speed should be 98% stable Reactive system: inputs … R output(s) Never done … September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV Time 6

Verification of Transformational Systems Test-vector generation (e. g. , T-VEC) == for transformational components Assertion (e. g. Java / C Assertion/condi tion) ? ? T-VEC – for functions / transformational sub-systems Transformational component inputs September 11, 2008 OR ? Expected output NASA IV&V Facility Workshop on Verification Morgantown, WV 7

Role of Correctness Property Assertions Assertion vs. implementation: a Venn diagram view September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 8

Role of Correctness Property Assertions September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 9

Role of Correctness Property Assertions An assertion is not an implementation: September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 10

Sources of Requirements for Assertions 1. Specification documents (NL) – if created by contractor they should be treated with a grain of salt. Req. in NL Issues: 1. Does contractor have other interest that play a role defining requirements? 2. Sequencing requirements are often not considered by spec. writers September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 11

Sources of Requirements for Assertions 2. Sequencing requirements are often not considered by spec. writers Sequencing/temporal considerations: • Can a track toggle between Data. Stores? DS 1 DS 2… If so, how often? • If toggling is expected then is there an allowed period of overlap (e. g. , being in both DS 1 and DS 2? ) • How soon should thread be published in DS? September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 12

Sources of Requirements for Assertions 2. From Activity-Diagrams/MSC’s to requirements based on concerns. See So. SE-2008 paper Req. in NL September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 13

The Purpose of Validation To assure the representative assertion is a good representative of this and/or this September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 14

Aren’t Assertions Correct by Construction? No…, we need to validate their correctness w. r. t to cognitive and/or NL intent • Ambiguities in cognitive or NL specification • Insufficient detail in specification • Bugs in assertion/formal-specification • Pilot errors: errors in test-case September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 15

Issues Validation can Discover/Resolve 1. Ambiguities in cognitive or NL specification An overdraft account should, within 2 weeks, gain a balance of $50 or more and stay black for a whole year thereafter $50 Black Red 2 weeks September 11, 2008 Ambiguous definition for the beginning time of the one year constraint NASA IV&V Facility Workshop on Verification Morgantown, WV 16

Issues Validation can Discover/Resolve 1. Ambiguities in cognitive or NL specification (cont. ) An overdraft account should, within 2 weeks, gain a balance of $50 or more and stay black for a whole year thereafter $50 Black Red September 11, 2008 2 weeks If this is the definition of the beginning of one year stability, then is the account allowed to be red more than once within the two week period? NASA IV&V Facility Workshop on Verification Morgantown, WV 17

Issues Validation can Discover/Resolve 1. Ambiguities in cognitive or NL specification Event P must never occur between events Q and R. Is Q. R. P. Q. R ok? (A traffic-light) must show yellow between a green and a red. Must it also show yellow when going red green? September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 18

Issues Validation can Discover/Resolve 2. Insufficient detail in specification Every sensor log-request must be acknowledged within 30 seconds. Ignores that fact that a sensor has multiple log-requests, does each need a unique acknowledgement? September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 19

Issues Validation can Discover/Resolve 3. Bugs in assertion/formal-specification Whenever event p occurs then event q must occur within 30 seconds RIGHT September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 20

Issues Validation can Discover/Resolve 4. Pilot errors: errors in driving validation test-case Whenever event p occurs then event q must occur within 30 seconds Expecting a failure but without waiting full 30 second period p p p 30 Test. R 1 b_4: r 1 b. p(); r 1 b. incr. Time(2); r 1 b. p(); r 1 b. incr. Time(11); assert. False(r 1 b. is. Success()); September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 21

Validation Patterns Whenever event p occurs then event q must occur within 30 seconds Validation patterns: p q 30 q p p p q 30 p p See SSIRI 2008 paper q 30 p 30 September 11, 2008 p p q 30 NASA IV&V Facility Workshop on Verification Morgantown, WV 22

Validation Coverage Can use animation or some batch/command -line technique States that were visited and transitions that were traversed by the validation suite are green i. e. , validation didn’t test for timeout first, then publish September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 23

Role of SRM in Eclipse Executable SRM September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 24

Role of SRM Executable SRM = Domain Model + Assertion Repository + Validation test suite + bridge September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 25

Using Code Coverage to find Missing Assertions September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 26

Role of SRM 1. Computer-based name space sanity checking 2. Some requirements may depend on data model 3. Identifying missing assertions September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 27

Role of SRM 1. Computer-based name space sanity checking class Library { … /** * verb/method/event */ public void req. Publish() { // connect to External assertions bundle fire. External. Assertions("req. Publish"); } } September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 28

Role of SRM 2. Some requirements may depend on data model September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 29

Role of SRM 3. Identifying missing assertions see technique presented in validation workshop September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 30

Verification So now we have assertions … Benefits: • Finding specification errors/issues early on. • For computer-aided verification. Lets use them for verification … September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 31

Verification September 11, 2008 NASA IV&V Facility Workshop on Verification Morgantown, WV 32

Verification – Lets have a discussion Possible testing architecture T-VEC for transformational behavior 3. Output events and data System Under Test (SUT) 3. Clock tick Assertion repository (RV) is. Success (including simple assertions for transformational systems) 1. Use observations from repository? 2. Generated events 2 b. Generated data September 11, 2008 Physical world data (e. g. from Matlab) 2 a. Gen data command NASA IV&V Facility Workshop on Verification Morgantown, WV Manual + Automatic Test Generator (ATG) 33