Extreme Grammaring Development of an industrial strength ISO

Extreme Grammaring Development of an industrial strength ISO VDM-SL Grammar

Introduction Need of building a parser for VDM from a VDMSL grammar (Voo. Doo. M project) Although parsing is a subset of a well studied area like compilers, grammars were always looked upon as the “ugly duckling”. Solution: Extreme Programming + Engineering of Grammars = Extreme Grammaring

Background l VDM: l l l Vienna Development Method (VDM) is one of the most mature formal methods Primarily intended formal specification and development of functional aspects of software systems. The importance of a VDM-SL Grammar: l l Documentation Build a parser (metric generators, language generators, . . . )

Starting point l Previous work l l l VDM-SL grammar in Happy + Alex Some problems State of Art (Hacking v. s. Engineering) l l Grammar was encoded directly Difficult to maintain/change (300 rules) Lack of tool support. . .

Principles of Grammar Engineering Introduced by Lämmel in “Towards an engineering discipline for grammaware” 1. 2. 3. 4. 5. 6. Start from specifications - base-line grammar Implement by customization - technology, implementation Separate concerns - modularization Enable evolution - minimize impact of changes Ensure quality - metrics, correctness Automate - traceability and scalability

Extreme Programming 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. The Planning Game - scope, priorities, technical estimates Small Releases - very short release cycle Metaphor - shared story Simple Design - remove complexity when found Testing - continuous unit testing, test-first design Refactoring - restructure without functionality changes Pair Programming - two programmers one machine Collective Ownership - change each others code (anytime) Continuous Integration - build and test 40 -Hour Week - work more = produce less On-Site Customer - user in team Coding Standards - no irrelevant personal preferences

Extreme Grammaring 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. The Planning Game - scope, priorities, technical estimates Small Releases - very short release cycle Metaphor - shared story Simple Design - remove complexity when found Testing - continuous unit testing, test-first design Refactoring - restructure without functionality changes Pair Programming - two programmers one machine Collective Ownership - change each others code (anytime) Continuous Integration - build and test 40 -Hour Week - work more = produce less On-Site Customer - user in team Coding Standards - no irrelevant personal preferences

The Planning Game Scope l l Follow strictly the ISO VDM-SL grammar spec Priorities l 1. 2. 3. Disambiguate types Disambiguate full grammar Tree construction Technical estimates l l Not defined. . .

Small Releases l Programmed releases (completed): l l 0. 0. 1 - Grammar typed from standard 0. 0. 2 - Disambiguated grammar 0. 0. 3 - AST construction Future Releases l l 1. 0 - Haskell front-end (finished - 29 -12 -2004) 2. 0 - Java front-end

Testing l White box l l l Structural testing Full visibility into how system works Black box l l Functional or behavioral testing Only the interface with exterior is available

Grammar Unit Testing l Unit test l l Different types of unit tests: l l l Test a single method, function, etc. . . Parsing (succeeds, fails) Well-formness of the tree Test suite l Combination of all unit test

Test Coverage l Rule coverage l l Introduced by Purdon (1971) Explores all rules of a grammar Simple measure but doesn’t cover all cases Context-dependent rule coverage l l l Introduced by Lämmel in “Grammar Testing” Generalization of the above in which the context is taken in account No known implementations

Test Coverage Metrics version KP KPr S RSa RSm RC 0. 0. 1 0. 0. 2 0. 0. 3 1005 946 900 815 2753 2278 3396 3518 2 2 2 17 17 17 52% 54% 55% l l l KP - Kernel Productions KPr - Kernel Priorities S - States RSa - Rule Size average RSm - Rule Size maximum RC - Rule Coverage Percentage

Test Coverage Metrics (2) l l version Generics expressions functiontypes All 0. 0. 1 0. 0. 2 0. 0. 3 52% 51% 50% 0% 24% 25% 0% 6% 5% 52% 54% 56% Although the “Generics” test-suite does not change the coverage gets lower (Injections, total nr rules) The “expressions” and “functiontypes” were only added in 0. 0. 2 version.

Refactoring l Semantic preserving transformations l l Study made by Lämmel in “Grammar Adaptation” Operators: l l l preserve - replace a phrase by an equivalent fold - replace for its definition unfold - extract definition introduce - introduction of non-terminals eliminate - elimination of non-terminals rename - renaming non-terminals

Continuous Integration l l The integration test suite is a set of generic real world examples Only 52% coverage l l l Examples are difficult to find Most of the examples use language extensions Examples: l l Found on internet Used a pre-processor for extracting code from literal programs.

Code Standards l l Nothing found about the subject The following can be applied: l l l l Limiting the number of children in a rule Limiting the number of alternatives in a rule Prefer some sort of constructs than other Convention for the non-terminal names Convention for syntax specification Limit module size. . .

Supporting the Methodology SDF - Syntax Definition Formalism l Purely declarative l Very expressive with natural and concise syntax l Modular structure l Supported by Scannerless Generalized LR Parsing l l l Supports compositional grammars Allows parsing with ambiguities (allows earlier testing) Disambiguation is separated from the grammar using priority and associative rules

SDF - Technology l Parsing l l Testing l l tree 2 graph, graph 2 dot Transformation l l test-unit, ambtracker, Sdf. Coverage Tree visualization l l sdf 2 table, sglr trm 2 baf, implode. PT Haskell Generation l Sdf 2 Haskell (AST, Pretty Printer)

Setting up the bases l l Hard copy of the ISO VDM-SL standard (ISO/IEC 13817 -1) Initial test suite l l l Real world examples (loc: 1970) Exercises from Formal Methods course (loc: 507) Software: l l l CVS to keep track of all changes parse-unit (sdf unit testing tool) Sdf 2 software bundle (sdf 2 table, sglr) Sdf. Coverage Starndard unix tools (text editor, make, . . . )

Development cycle Initial grammar Correction 1. 2. Correct grammar rules Correct test suite Disambiguation 3. 1. 2. Add filters Change grammar shape Steps 2 and 3 should make heavy use of testing

Grammar correction Isolate problem 1. l l Source location Grammar rules involved Correct grammar 2. l l l Change syntax (test suite) Run to verify test succeeds Run entire test battery Commit Change 3. l Document change in message

Grammar Disambiguation Isolate problem 1. l l Source location Grammar rules involved Create unit test 2. l l Captures error Run to guarantee this Correct grammar 3. l l l Add disambiguation filter (change syntax) Run to verify unit test succeeds Run entire test battery Commit Change 4. l Document change in message

Grammar Metrics l Simple metrics l l l Total Number of Terminals (AVG per rule) Total Number of Non-terminals (AVG per rule) Complex metrics l Introduced by Malloy in “A metrics suite for grammar-base software” l l l Mc. Cabe Cyclomatic complexity Halstead Effort. . .

Problems found l l ISO Document has ambiguities in its specification Syntax l Expressions: l l l Equals. Definition l l Apply v. s. Record. Constructor Apply v. s. Is. Def. Type. Expr Call. Statement v. s. Expression Lexical l Quotes are allowed in strings and in characters

Future plans l Short-term (VDM parser “clients”): l l l Voo. Doo. M Formal methods projects MCZ Objectifier Camila revival? Long-term l Topic open for discussion. . .

What’s next? l Test set completion (fill the rest 44%) l l l Analyze the rules that were not covered l l l Test generation Add examples manually Try to find pathologies Compute Grammar Metrics Test the methodology developing other grammars.

Conclusion l l Work was completed in only 3 weeks A complete grammar of the ISO VDM-SL is for the first time public available (parser) A strong methodology for grammar developing was defined Grammar testing were put to practice l l Different types of tests Test coverage

Thank you!

Questions / Discussion