Formal Methods Tool Qualification for DO 178 B

Formal Methods Tool Qualification for DO 178 B & DO 178 C Nick Tudor tel: +44 1684 894489 email: njtudor@qinetiq. com www. Qineti. Q. com © Copyright Qineti. Q limited 2010

Agenda • Tool Overview and Qualification Approach • The Current Guidance - DO 178 B • Qualification Considerations – DO 178 B • Applicant’s Claims – DO 178 B • Qualification Considerations – DO 178 C • Applicant’s Claims – DO 178 C and Supplements • Summary www. Qineti. Q. com © Copyright Qineti. Q limited 2010

Tool Overview and Qualification Approach www. Qineti. Q. com © Copyright Qineti. Q limited 2010

Overview of the tool • The tool is a Formal Methods based software verification tool − Automates the verification of automatically generated code − Constraints include a subset of Simulink and a subset of the Ada programming language • Exploits the Z formal specification language − Automatic generation of formal specification of the Simulink in Z − Gives required independence • Uses Carroll Morgan’s Theory of Refinement, also automatic • Built in automated use of an off-the-shelf Theorem Prover (Proof. Power) • Having used the tool manually for independent code verification, we aimed to: − Automatically prove automatically generated code for productivity reasons − Understand the tool qualification issues www. Qineti. Q. com © Copyright Qineti. Q limited 2010

Overview of the CLaw. Z Process B 4 S Simulink Z Producer Ada Development Verification User Interface Refinement Script Generator Z Supertac Compliance Notation Tool Verification Conditions Proof. Power Discharge proof www. Qineti. Q. com © Copyright Qineti. Q limited 2010

The GUI and some functions (PID example) Creating Defining Interface variables for each unit Creating Z specification, Importing Simulink & Ada doing refinement files & defining units and proof Proof results www. Qineti. Q. com © Copyright Qineti. Q limited 2010

The ‘pre-qualification’ approach • As part of the final development steps, we took advice from CAA • This is new because: − No-one has previously approached them to do any form of “pre-qualification” of a tool − NB This does not constitute certification authority pre-approval of this tool − It’s a formal methods tool − A formal methods tool which is intended to only automate code verification [and will be qualified as such] • We needed to examine the guidance from DO 178 B to ensure that we knew what we were aiming at and to check likely impact of DO 178 C − Particularly in light of both the Formal Methods and Tool Qualification Supplements www. Qineti. Q. com © Copyright Qineti. Q limited 2010

How high is the bar? RTCA DO 178 C/EUROCAE ED 12 C Tool Qualification Supplement RTCA DO 178 B/EUROCAE ED 12 B “Plus” RTCA DO 178 B/EUROCAE ED 12 B Minimum Qineti. Q Internal Processes www. Qineti. Q. com © Copyright Qineti. Q limited 2010

A virtual, parallel approach Tool Development Theoretical Applicant Theoretical System DO 178 B Theoretical DO 178 C Includes (theoretically): Core text, Tool Qualification, FM Supplement MBD Supplement www. Qineti. Q. com © Copyright Qineti. Q limited 2010

The Current Guidance - DO 178 B www. Qineti. Q. com © Copyright Qineti. Q limited 2010

Not Shooting for the Minimum • RTCA DO 178 B states that for a Verification tool: − “The qualification criteria for software verification tools should be achieved by demonstration that the tool complies with its Tool Operational Requirements under normal operational conditions” − Production of Tool Operational Requirements, the Verification Plan, the Verification Results and the Tool Accomplishment Summary • NB FAA-8110 -49 says you could also have a TQP and TAS, but specifically: − “Tool Operational Requirements satisfies the same objectives as the Software Requirements Data of the airborne software”. …ie (from 12. 2. 3. 2): − “Tool Operational Requirements describe the tool's operational functionality. This data should include: − a. A description of the tool's functions and technical features. [For software development tools, it includes the software development process activities performed by the tool]. − b. User information, such as installation guides and user manuals. − c. A description of the tool's operational environment. − d. [For software development tools, the expected responses of the tool under abnormal operating conditions. ] ” www. Qineti. Q. com © Copyright Qineti. Q limited 2010

Adapted from FAA-8110 -49 Fig 9 -2 Data Applicability Available/ Submit/Note RTCA/DO-178 B Ref. Plan for Software Aspects of Certification (PSAC) Verification & Development Submit 12. 2, 12. 2. 3. a, & 12. 2. 4 Tool Qualification Plan Development Submit 12. 2. 3. a(1), 12. 2. 3. 1, & 12. 2. 4 Tool Operational Requirements Verification & Development Available 12. 2. 3. c(2) & 12. 2. 3. 2 Software Accomplishment Summary (SAS) Verification & Development Submit 12. 2. 4 Tool Accomplishment Summary Development Submit 12. 2. 3. c(3) & 12. 2. 4 Tool Verification Records (for example, test cases, procedures, and results) Verification & Development Available 12. 2. 3 Tool Qualification Development data (for example, requirements, design, and code) Development Available 12. 2. 3 www. Qineti. Q. com © Copyright Qineti. Q limited 2010

Qualification considerations – DO 178 B www. Qineti. Q. com © Copyright Qineti. Q limited 2010

An Applicant’s perspective • We needed to consider how an Applicant would use the tool and subsequently be able to claim credit − Despite confused guidance, what artefacts would need to be made available − Given that this is a tool based upon FM, what “language” to use (this is non-trivial!) • We needed to be cognisant of context, ie the overall software development life cycle • We have defined CLaw. Z Simulink Subset (CSS) used for Independent Verification and used a well known subset of code (Ada) − This constrains the possible input space to something that is well defined, but also very flexible − Also constrained the output space (code) • By doing this we set the Tool Operational Requirements and hence the scope of the qualification www. Qineti. Q. com © Copyright Qineti. Q limited 2010

Proposed approach Histor y Available information includes: Development Plan Configuration Management Plan QA Plan/records (not for 178 B) Design Description Configuration Records Development Environment Index User Guide Upgrade/ production of Plans Configuration Control Tool Operational Requirements Verification Plan Configuration Index Tool Qualification Plan Applicants PSAC Tool Accomplishment Summary Applicants SAS Verification Cases and Procedures Verification Results QA and Technical Audit www. Qineti. Q. com © Copyright Qineti. Q limited 2010

Applicant’s Claims – DO 178 B www. Qineti. Q. com © Copyright Qineti. Q limited 2010

Verification Objectives – DO 178 B System Requirements A-3. 2 Accuracy & Consistency A-3. 3 HW Compatibility A-3. 4 Verifiability A-3. 5 Conformance A-3. 7 Algorithm Accuracy A-3. 1 Compliance A-3. 6 Traceability (A-2: 1, 2) High-Level Requirements A-6. 1 Compliance A-6. 2 Robustness NB Verification of the Verification, QA, DER liaison and other documentation also required A-4. 1 Compliance A-4. 6 Traceability A-4. 8 Architecture Compatibility (A-2: 3, 4, 5) A-4. 9 Consistency A-4. 10 HW Compatibility A-4. 11 Verifiability A-4. 12 Conformance A-4. 13 Partition Integrity Software Architecture Low-Level Requirements A-5. 2 Compliance (A-2: 6) A-5. 3 Verifiability A-5. 4 Conformance A-5. 6 Accuracy & Consistency A-4. 2 Accuracy & Consistency A-4. 3 HW Compatibility A-4. 4 Verifiability A-4. 5 Conformance A-4. 7 Algorithm Accuracy A-5. 1 Compliance A-5. 5 Traceability Source Code A-6. 3 Compliance A-6. 4 Robustness (A-2: 7) A-5. 7 Complete & Correct Executable Object Code A-6. 5 Compatible With Target Compliance: with requirement Conformance: with standards www. Qineti. Q. com © Copyright Qineti. Q limited 2010

Claims relating to source code verification Table A-5 Objective Description Applicability by Software Level Ref. A B C 1 Source Code complies with low-level requirements. 6. 3. 4 a l l m 2 Source Code complies with software architecture. 6. 3. 4 b l m m 3 Source Code is verifiable. 6. 3. 4 c m m 4 Source Code conforms to standards. 6. 3. 4 d m m m 5 Source Code is traceable to low-level requirements. 6. 3. 4 e m m m 6 Source Code is accurate and consistent. 6. 3. 4 f l m m 7 Output of software integration process 6. 3. 5 is complete and correct. m m m D Partial We can show that eg there are no uninitialised or unused variables or constants. There are no claims re stack usage, resource contention, WCET, etc www. Qineti. Q. com © Copyright Qineti. Q limited 2010

Verification Objectives System Requirements Major Contribution Simulink [Continuous] (A-2: 1, 2) High-Level Requirements Simulink [Discrete] (A-2: 3, 4, 5) Software Architecture Low-Level Requirements Simulink for CLaw. Z A-5. 2 Compliance (A-2: 6) A-5. 3 Verifiability A-5. 4 Conformance A-5. 6 Accuracy & Consistency A-5. 1 Compliance A-5. 5 Traceability Source Code Autocode, proof (A-2: 7) A-5. 7 Complete & Correct Executable Object Code www. Qineti. Q. com © Copyright Qineti. Q limited 2010

Qualification considerations – DO 178 C www. Qineti. Q. com © Copyright Qineti. Q limited 2010

DO 178 C introduces 3 categories of tools (see section 12. 2. 2) DO 178 B Development tool ? Verification tool DO 178 C Criteria 1 Criteria 2 Criteria 3 DAL Criteria 1 Criteria 2 Criteria 3 A TQL 1 TQL 4 TQL 5 B TQL 2 TQL 4 TQL 5 C TQL 3 TQL 5 D TQL 4 TQL 5 Criteria 2: A 1: tool that whose automates verification process(es) and thus Criteria A tool output is part of the airborne could fail to detect an error, whose output is used to justify the software and thus couldand insert an error. elimination or reduction of: Criteria 3: A tool that, within the scope of its intended use, 1. Verification process(es) other than that automated by the tool, or fail to detect an error. 2. could Development process(es) that could have an impact on the airborne software. www. Qineti. Q. com © Copyright Qineti. Q limited 2010

DO 178 C Tool Qualification Supplement • We wanted to know what we might be under ED 12 C/DO 178 C we we’re either − A verification tool (Criteria 3) and hence TQL 5 would be required, or − A super verification tool (Criteria 2) and hence TQL 5 or even TQL 4 • We believe CLaw. Z would be a TQL 5 tool, but this depends largely on what the Applicant will wish claim as a result of using the tool • And whether this is acceptable for the certifier • More on this at the end… • We will also examine TQL 4 …just in case! www. Qineti. Q. com © Copyright Qineti. Q limited 2010

DO 178 C TQL 4 – Tables T-1 & T-2 • We needed to keep and eye on what TQL 4 required at the same time • Referring to Table T-1 and T 2, there is considerable extra material required over DO 178 B Document 178 B TQL 4 Note The Tool Qualification Plan (TQP) Y Y Tool Development Plan (TDP) N Y Not strictly needed, but certifier will look for it Tool Verification Plan (TVP) Y Y Not entirely clear if this is needed, but… Tool Configuration Management Plan (TCMP) N Y Not entirely clear if this is needed, but the Configuration Index is required Tool Quality Assurance Plan (TQAP) N Y Not needed, but needed for eg Tick. IT? • No requirement for Tool Operational Requirements, but is implicit • Interesting that QA Plan appears to be required, but not QA records • Also, that Tool Plans do not have to comply with the Supplement (T-1 -6) www. Qineti. Q. com © Copyright Qineti. Q limited 2010

TQL 4 – DO 178 C – Tables T-3, T-4 and T-5 • There is large concentration of effort in Table T-3 (requirements) which is not “needed” under DO 178 B − It is implied that the Tool Operational Requirements are the “requirements”. • Suspect that in practice, for all but trivial tools under qualified under DO 178 B, there is a more clearly defined set of requirements contained within the TOR • It was not immediately clear why the only requirement for Table T 4 is T -4 -10 (protection mechanisms). Sought guidance from FAQ #2 Tool Qualification Supplement : − Seeks to show evidence for separation of functions, essentially through architecture – fine, but… − There are no other architecture requirements for TQL 4 (ie compatible with requirements, consistency, standards), so does this objective help safety? • No objectives for Table T-5 verification of the coding process… www. Qineti. Q. com © Copyright Qineti. Q limited 2010

TQL 4 - Tables T-4 and T-5 - A closer examination • From DP #2: Rationale for Tool Qualification Criteria Definition, para 2. 2. 3. 2 Rationale for introducing Criteria 2 “These additional uses of tools raise some concerns about the rigor required for qualification. For example: − From a safety perspective, the impact of these tools on the final software can vary. − The required confidence may not be able to be assessed by considering only the Tool Operational Requirements. − The maturity and soundness of a mathematical theory and its implementation may be necessary to provide the required confidence • The major concern would appear to be the implementation, so no examination of low level requirements or the source code? • Reliance is therefore left to Table T-6… • NB No objectives are required to be met for TQL 5 www. Qineti. Q. com © Copyright Qineti. Q limited 2010

TQL 4 – ED 12 C/DO 178 C – Tables T-6, T-7, T-8 and T-9 • Focus here is on how the tool functions when actually undertaking verification • Examines object code with respect to the Tool Requirements − Checks compliance and robustness • Also looks at verification results − Checks that results were correct/discrepancies explained and − Coverage of Tool Requirements • Ensures that the configuration was identified www. Qineti. Q. com © Copyright Qineti. Q limited 2010

Applicant’s Claims – DO 178 C and Supplements www. Qineti. Q. com © Copyright Qineti. Q limited 2010

Possible claims relating to source code verification – DO 178 C FM Supplement • Specific claims formal verification could be made in accordance with the Formal Methods Supplement (see table FM A-5) − Specific claims can be made FM A 5 -1 to 6 (FM 7 refers to core text as above) − Also need to examine possible to claims to Extra Objectives FM 8 -11 Objective Description Applicability by SW Level Ref. A B C FM 8 Formal analysis cases and procedures are correct. FM. 6. 3. 6 a FM. 6. 3. 6 b l m m FM 9 Formal analysis results are correct and discrepancies explained. FM. 6. 3. 6 c l m m FM 10 Requirements formalization is correct. FM. 6. 3 i l l m FM 11 Formal method is correctly defined and sound. FM. 6. 2 l m m D m www. Qineti. Q. com © Copyright Qineti. Q limited 2010

Perspective of OTS tools for DO 178 C vs in-house tool development for DO 178 B • Number of views from Tool Qualification Supplement − Re-use – see section 11. 2 − OTS - see section 11. 3 − Service History – see section 11. 4 (not applicable) − Alternative methods – see section 11. 5 (not applicable) • Section 11. 3. 2 shows activities required by the tool developer − Which are to produce: TOR, TQP, TCI, TAS − Also outlines which objectives are modified • Section 11. 3. 3 shows activities for the tool user − To assess the TOR, TQP, TCI and TAS and plan extra activities such as division of responsibilities www. Qineti. Q. com © Copyright Qineti. Q limited 2010

Summary • We believe we can support an Applicant in system certification to DO 178 B • We have gone further than was required because this tool was “predeveloped” and is a formal methods tool • We believe that it would be a TQL 5 tool under DO 178 C Tool Qualification Supplement, but… • Could be TQL 4, depending on development processes of the Applicant • Interesting that our approach has mirrored the COTS advice to tool qualification • So does TQL 4 achieve anything extra in terms of system safety? www. Qineti. Q. com © Copyright Qineti. Q limited 2010