Assurance techniques for code generators Ewen Denney USRARIACS

Assurance techniques for code generators Ewen Denney USRA/RIACS, NASA Ames Bernd Fischer ECS, U Southampton

Assurance problem • Safety/mission-critical software requires assurance that it meets a certain level of “quality” • What are the issues in assuring automatically generated code? – Different forms of assurance – Different assurance techniques – Diverse generator paradigms

Forms of assurance What exactly might we need to assure? • Compliance with requirements Correctness • Compliance with spec/model • Certification standards • Coding standards Reliability • Absence of run-time errors • Traceability Legibility • Appropriate documentation Minimize “automation surprises”

Code generators in practice Practitioner survey carried out in March 2006 (Code Generators in Safety-critical Applications, Schumann, E. Denney); 23 responses from • • • J. NASA and industry. How are ACGs used for safety-critical applications at NASA and in industry? Which are the primary application areas and domains? Which tools are used? Challenges, benefits and problems? How could ACGs be extended to be more useful in safety-critical applications?

Tools and languages The Big Three: • Real-Time Workshop • Matrix. X • SCADE

Domains and criticality levels • Principle domains: – control – modeling/simulation • Many highly critical applications • ACG used for – – – production code (74%) prototyping (52%) simulation (48%) testing (30%) glue/interface code (30%)

System components

Weaknesses • Steep Learning Curve – applicable problems, features, correct usage, architecture, implied methodology, semantic ambiguities, … – substantial impact on development process • ACG customization – necessary in 1/3 of cases – often (2/3) done by tool vendor • ACG bugs – in 2/3 of applications, bugs were found in ACG

Qualification • A code generator is qualified – with respect to a given standard – for a given project • • if there is sufficient evidence about the generator itself so that V&V need not be carried out on the generated code to certify it Must be done for every project, version Can obtain verification credit Generators are rarely qualified Examples: ASCET-SE (IEC 61508), SCADE, VAPS (DO-178 B)

Certification and V&V • Auto-generated code must be certified for safety-critical use • Techniques used: – testing (90%) – static analysis (58%) – simulation (52%) – manual review (48%) • No formal verification • No review of generator code

Safety properties

Generator features

Domain-specific analyses Mostly numeric issues: • stability (root locus, Lyapunov) • robustness • convergence • transience Some domain-specific design rules: • “forbidden” constructs • block structure

Documentation • Design information • Code derivation • Configuration management information (to “replay” generation) • Safety information • Tracing information • Interface definitions, requirements • User manuals • Installation information Should be customizable

Traceability • Most important: model code • Secondary: code V&V artifacts

Tool integration Also • workflow and process tools • tools for integrating legacy code

Survey summary • Integrated modeling, analysis, and simulation tools are most common in control domain • In-house extensions common for modeling and verification issues • Natural synergy between code generation and certification activities – perceived but not realized – autocode often treated like manual code • Iterative customization of generator should be seen as integral part of development process

Assurance techniques • Testing the generator (qualification) – for all specs, blocks, configurations, backends, … • Post factum verification / certification – verify / certify generated programs individually • Correctness by construction – generator inherently guarantees certain properties • Documentation • Traceability

Discussion questions • What are the interesting assurance artifacts, properties, etc. in your target domains? • What are suitable notions of documentation, traceability, development process? • What assurance techniques have you tried? • How is the generative knowledge represented (templates, transformation rules, etc. ) and how can it be combined with assurance information? • Can we apply Design for Verification (D 4 V) to generators?