Verification and Validation on LSST Brian Selvy LSST

Verification and Validation on LSST Brian Selvy LSST All-Hands Meeting Bremerton August 17, 2015

Agenda • Project Systems Engineering’s Responsibilities • Review V&V Process – Verification Plan Requirements – Compliance Assessment Requirements – Final Verification Matrix Requirements • Verification as Part of AIV • Non-Conformances and the Relationship between V&V and QA • PSE & Subsystem Collaboration on Verification 8/17/2015 LSST Systems Engineering 1

PSE’s Verification Responsibilities • Project Systems Engineering is responsible for the overall, systemlevel integration, verification, and validation of the LSST observatory. PSE has responsibility for the Commissioning phase of the LSST project. • Verification is one critical aspect of the broader manufacturing, assembly, integration, and verification (AIV) set of activities • In order for PSE to properly plan and understand the inputs to the early integration, and commissioning phases, PSE needs to understand the activities being conducted by the subsystems that directly impact system level requirements, interfaces, assemblies, and verification activities. – This leads to the need for collaboration and input from the subsystem teams on developing and reviewing verification plans, observing verification activities, reviewing verification results, and collaborating on the response to deviations/waivers for NCRs affecting project-controlled requirements (via the CCB process)

Verification & Validation on LSST • LSE-160 Verification and Validation Process is the governing document for V&V on LSST – Establishes a consistent, project-wide process for the development of V&V plans, compliance assessments, V&V reporting, and deliverables • LSE-160 is currently undergoing a revision to: – Harmonize with LPM-55 LSST QA Plan (also in revision) • Define interactions and dependencies between V&V and QA – Further define the Assembly, Integration, and Verification (AIV) Framework during Early I&T and Commissioning – Document the Sys. ML Implementation methodologies for bottoms up and top down modeling 8/18/2015 LSST Systems Engineering 3

LSE-160 Applicability • Applies to all Project-Controlled requirements: – Specifications – Requirements Documents – Interface Control Documents (ICDs) • Each “shall” statement in each of these documents must be formally verified Project-Controlled Specifications Project-Controlled ICDs

LSST Verification & Validation Process

Verification Planning Requirements • For each requirement (“shall statement”) a Verification Plan will be created that includes the following: – Verification Owner – the subsystem team that is responsible for verification – Responsible Technical Authority (RTA) – the point-of-contact assigned responsibility for the verification of the requirement from the responsible subsystem. The RTA, along with the responsible QA individual, has responsibility for overseeing all associated verification events. – Verification Method(s) – Test, Analysis, Inspection, Demonstration – Verification Level – Same Level, Higher Level, Lower Level – Verification Requirement – A statement that defines precisely what will be done to verify the requirement. If there is any vagueness in the requirement, the Verification Requirement should clearly address the noted issues and define what precisely will be verified any limitations. The statement should define what will be done, where it will be done, what special test equipment (SPE) is needed, and what project hardware/software is needed. – Success Criteria - A statement that defines the explicit pass/fail criteria. This statement should be clear enough that an independent third party observer should be able to determine if the verification event was successful or not.

Verification Planning Requirements (Cont) • Documented in a Verification Matrix for Planning (VM-P) – One required for each Specification – One required for each subsystem involved in an ICD

Compliance Assessment Requirements • Compliance is defined as the ability of the current (any point in time) “as-designed” system to meet its associated requirements. • The difference between compliance and verification is that verification is conducted on the final designed and built system, whereas compliance can be done at any earlier time and is an early step in the overall verification process. • Compliance Assessments are required at each major subsystem and component design review. • Required documentation: – Compliance Method(s) – Analysis, Test, Demonstration, Inspection – Verification Requirement – Success Criteria – Compliance Status (Y/N) – References to any additional documentation that further justifies the assessment, if available.

Compliance Assessment Rqmts (Cont) • For any non-compliances: – A risk must be documented in the Risk & Opportunity Register – The risk Number should be cross-referenced in the compliance table – The Risk entry should identify mitigation activities to correct the noncompliance

Final Verification Reporting Rqmts • A final Verification Record is compiled after all requirements within a specification / ICD have been verified. • A Verification Matrix for Final Verification (VM-V) serves as the final record and summary of the verification process. • For each requirement, summary information from the Verification Plan is included along with: – Responsible Technical Authority (RTA) – Quality Assurance Officer (QAO) or designee’s name – Verification Successful (Y/N) – Verification Result Summary – a concise summary narrative explaining why the verification activities were successful or not. – Verification Report – A reference to the Verification Report that contains the details of the results of the verification activities. • For each requirement, an RTA and QAO (or designee) will be identified. These individuals are responsible for vouching that the requirement has been verified and generating initial responses to Non-Conformances (more on later slides)

Verification as Part of AIV • Verification is one critical aspect of the broader manufacturing, assembly, integration, and verification set of activities • Project Systems Engineering needs to understand the early integration and verification activities being conducted by the subsystems that impact system level requirements, interfaces, assemblies, and verification activities. • PSE is assembling a more detailed Project-level AIV plan. It is crucial that subsystem inputs are provided to support this activity. • A general pattern has been defined that PSE will use to document these activities in Enterprise Architect using the Sys. ML language (next slide)

Verification Process Steps 4 and 5 • After Verification Plans are generated, PSE uses this information, along with additional input from the subsystem teams, to develop a comprehensive systemlevel Verification model, : • Identify Task Interdependency (Step 4) – Some Verification Activities can be naturally grouped and conducted at the same time – These Verification Activities are then grouped into a single Verification Event. • Can result in cost and schedule savings from eliminating redundant or nearly redundant V&V activities • Schedule Verification Events (Step 5) – Events are scheduled, identifying predecessor/ successor relationships and other schedule constraints

AIV Pattern Input Objects Output Objects Verification (Next Slide) Transformation Processes IF R E V IC O ATI N Integration Processes

Creation of Verification Plans & Test Cases in the Model

Sequencing Test Cases (Verification Events) • Test Cases (Verification Events) are sequenced on Activity Diagrams to show: – Predecessor/ successor relationships – Events that are conducted in parallel/ series – Outside constraints that must be met before a Verification Event can be executed • Results can be used to validate or update the project’s schedule for the Commissioning period.

Refinement of Individual Test Cases (Verification Events) • As plans mature, individual Verification Events can be further detailed via association with its own detailed behavior diagram • Serves as refined and more detailed input to the commissioning planning effort – Can be used directly as inputs to writing detailed test & analysis procedures

Non-Conformances • The are two categories of non-conformances: – Incidents – short-term anomalies or observations that require immediate attention • Examples: – Failure of a Verification Activity due to operator error – Need to redline a verification procedure due to realized constraints of the verification environment that were not anticipated by the author of the procedure. These redlines only affect the details of how the activity is conducted and do not impact the validity of the results – Problems – confirmed nonconformities that would cause the project to fail to meet requirements. • Examples: – Repeated Failure of a verification activity that was executed to the procedure. The results indicate a real problem with the form, fit, and/or function of the device. • The RTA and QAO should directly respond to and disposition Incidents. Responses should be documented, and then the team may proceed immediately. • Problems must be documented in a Non Compliance Report (NCR) and submitted as soon as possible to the Change Control Board (CCB) for communication purposes.

Relationship between V&V and QA • The Verification Team (with RTA lead) is responsible for technical execution of the verification activities, which includes proper planning, execution of the activities, and reporting. • Quality Assurance is responsible for surveillance and auditing of verification processes, including witnessing selected verification activities, to ensure that all steps are executed according to plan and that deviations and non-compliances are handled in a methodical way. QA provides confidence to stakeholders that products and services conform to requirements • An RTA and QA Individual are jointly responsible for the verification of each requirement – RTA is responsible for the technical execution of the verification events – QA is responsible that the verification events are carried out according to plan

Acceptance and Verification Process

PSE Working with the Subsystems • In order for the PSE team to meet its responsibility of overall verification of the LSST Observatory system, the PSE will work collaboratively with the subsystems during subsystem component verification. – Enhances the PSE team’s understanding of the performance and functional capabilities of the as-built hardware and software – Ensures the PSE understands more comprehensively how lower level verification activities support subsystem level verification activities, which ultimately feed into project-level commissioning activities. • The PSE will work with each subsystem to iterate and refine a list of subsystem milestones that the PSE requests visibility into – PSE will work with the subsystems to refine these lists and generate PMCS milestones, which will act as the notification mechanisms to PSE. – Work to be completed by Oct. 30, 2015

Subsystem Level Milestones Hardware Centric Software Centric Review, Verification, and Acceptance Milestones to be identified for each Component: Requirements Review Release Objectives Review Final Design Review Verification Plan Review Procurement Review Unit Test Manufacturing Readiness Review Low Level Integration Test Verification Plan Review End to End Test Start of Verification Activities (i. e. Tests) Acceptance Test Subsystem Pre-shipment Review (if applicable) Acceptance Test Review Subsystem Acceptance Review …. . Lists to be iterated and Milestones finalized by Oct. 30, 2015

Conclusions • LSST Systems Engineering Process has been in place for several years – It is now time to start using it! – The process has been expanded to deal with non-conformances and integrate Verification with Commissioning planning in more detail • Initial interactions between V&V and QA have been proposed • The modeling methodology has been defined • Next steps: – PSE and the subsystems to work collaboratively to generate the Verification Planning Matrices for the subsystem specifications and project-level ICDs. – PSE to generate AIV Activity Diagrams for early integration, system level integration, and commissioning

Backup Slides

Modeling Methodology for V&V using Sys. ML

Sys. ML Implementation – Defining Requirements • All project-controlled requirements are captured as elements in the EA Sys. ML model – Each specification from the LSST Specification Tree is modeled as a version-controlled package – Requirements are modeled as Requirement elements under the applicable package.

Sys. ML Implementation – Definition of Rqmts Hierarchy • Requirements Diagrams used to show: – Model hierarchy (using Containment relationship) – Requirements traceability via decomposition and allocation (using Derived relationship)

Extending the Sys. ML Stereotype for Verification Planning • Sys. ML does not have a predefined element capable of capturing LSST’s Verification Planning information • Sys. ML is extensible, allowing for the definition of additional stereotypes • LSST created a Verification. Planning stereotype as an extension of the Sys. ML 1. 3: : requiremen t metaclass