Successful Quality Assurance Activities Our Pillars for Success

So What Happened! Quality is everyone’s job and essential for survival 1 9/10/2020 Copyright

You Have a JOB § Quality is everyone’s job and essential for survival 2

My Objectives – Today Is: § Share with you some concepts about Quality and

Quality Assurance Exposed § § § What does Quality Assurance mean to you? What

Is It MY Responsibility? Quality is everyone's responsibility Project Managers Ultimately responsible for the

Definition § Quality Assurance Definitions § A program for the systematic monitoring and evaluation

Program Phases and Quality System Deployment Management Vision, Commitment, and Review Design Review Configuration

Basic Quality System Infrastructure § § § Management Design review Configuration management Document control

Pillars of Successful Quality Systems § Management Support § Attitudes § Cooperation § Adequate

Pillars of Successful Quality Systems (cont’d) § Meticulous data and configuration management § Revision

It Could Happen to You! “If the procedure is right, implementation should pose no

Developer Walkthroughs Milestone Reviews Example Lifecycle of a SW Build Start of of Increment

Said Another Way. . Requirements / Top Level Design Information Architecture Requirement s Design

SW Development/Production Cycle Example Initial Review (IR) Definition Phase Software Implementation Phase Test Walkthrough

QA Role: SW Development/Production Cycle Definition Phase • Objectives, capabilities, & requirements of the

QA Role: SW Development/Production Cycle Software Implementation Phase • Transforms the detailed software design

QA Role: SW Development/Production Cycle Requirements Test Phase Test Walkthrough (TWT) • Assesses readiness

Successful Process Pacing Bench Mark Schedule Example Mar Apr 3/17 3/24 3/31 4/7 May

Project-x Defect Density Example Defects = STRs since TWT (after Dev) DD = STR/KSLOC

Project-x First-Time Quality Example FTQ = STRs since RR FTQ = STR/KSLOC >1 Defect/KSLOC

Product Ver X STR History Example 23 9/10/2020 Copyright 2005 Northrop Grumman Corporation

Product Ver X STRs by Severity Example Self-assessed as GREEN SEVERITY Definitions: 1 -

Product Ver X STRs by Phase Example Self-assessed as GREEN 25 9/10/2020 Copyright 2005

Project-x SW Defect Containment Example Code & Unit Test - Most Complex Code only

How Can We Experience Failure? 27 9/10/2020 Copyright 2005 Northrop Grumman Corporation

How Can We Experience Failure? § Knowns (We know what we know) § Ops

How Can We Experience Failure? § Unknown Knowns (We don't know what we do

Failure-Maybe: Have Tools to Fill the Holes NEW PROGRAM DEFECTS DEVELOP PROCEDURES, AUDIT COMPLIANCE

Summary § § § § Definition of Quality Assurance QA elements of a typical

BACK UP 32 9/10/2020 Copyright 2005 Northrop Grumman Corporation

For YOU who Like to See a Flow Diagram RWT Development Standalone Testing IR

QA Activities: Design and Development Fabrication and Assembly Subcontract and Supplier Management Integration and

HW QA Activities: Fab and Assembly Design and Development Fabrication and Assembly Subcontract and

QA Act: Subcontract and Supplier Mgmt Design and Development Fabrication and Assembly Subcontract and

QA Activities: Integration and Test Design and Development Fabrication and Assembly Subcontract and Supplier

QA Activities: Acceptance and Delivery Design and Development Fabrication and Assembly Subcontract and Supplier

Slides: 44

Download presentation

Successful Quality Assurance Activities Our Pillars for Success March 3 -4, 2008 Darrell Smith Mission Assurance Manager Northrop Grumman Corporation 0 9/10/2020 Copyright 2005 Northrop Grumman Corporation

My Objectives – Today Is: § Share with you some concepts about Quality and Processes § Share with you the QA role in our processes § Provide some examples of success 3 9/10/2020 Copyright 2005 Northrop Grumman Corporation

Quality Assurance Exposed § § § What does Quality Assurance mean to you? What are the popular perceptions of Quality Assurance? Why have Quality Assurance at all? When does Quality Assurance become important? How do we make Quality Assurance successful? Could this be your motto? Quality is important in everything we do. Quality is everyone’s responsibility and is achieved through continuous improvement. We routinely seek ways to do things better. 4 9/10/2020 Copyright 2005 Northrop Grumman Corporation

Is It MY Responsibility? Quality is everyone's responsibility Project Managers Ultimately responsible for the quality of the products and services that they provide customers in accordance with contractual agreements and consistent with corporate policies 5 9/10/2020 Every Employee Responsible for the quality of the products they produce and the services they provide Copyright 2005 Northrop Grumman Corporation

Definition § Quality Assurance Definitions § A program for the systematic monitoring and evaluation of the various aspects of a project, service, or facility to provide adequate confidence that the product or service will satisfy given requirements for quality § Assurance that we deliver what we promise to our customers § Described by a System of Controls § Infrastructure that describes how to run the business § Detailed in policies, handbooks, plans, procedures, etc. § Usually structured around and compliant to a standard § AS 9100 § ISO § CMMI QA Processes Supports The Bottom Line 6 9/10/2020 Copyright 2005 Northrop Grumman Corporation

Program Phases and Quality System Deployment Management Vision, Commitment, and Review Design Review Configuration Management Document Control Supplier Quality Design Management Supplier Quality Management Calibration, Inspection, and Documentation Non-conforming Material Control Stamps Servicing 7 9/10/2020 Design Manufacturing Testing Production R&D Test Configuration Management Documentation Calibration Production Configuration Management, Documentation, Calibration, Performance Measures, Training, Audits Configuration Stability Performance Measures and Product Improvement Copyright 2005 Northrop Grumman Corporation

Basic Quality System Infrastructure § § § Management Design review Configuration management Document control Supplier quality Calibration Training Stamps Non-conforming material handling Manufacturing Inspection Test § § Audits Production Measurement Servicing If the process is right, the results will be right 8 9/10/2020 Copyright 2005 Northrop Grumman Corporation

Pillars of Successful Quality Systems § Management Support § Attitudes § Cooperation § Adequate resources § Coverage and emphasis § Early involvement § Teaming § Detailed supplier involvement § Gather information § Choose partners wisely § Develop relationships and trust § Aggressively manage progress § Free communication § Information is shared freely, respected, and appropriately considered § Lessons learned are captured and shared § The QA counterpart relationship Communication & Teamwork are Keys that Open Many Doors 9 9/10/2020 Copyright 2005 Northrop Grumman Corporation

Pillars of Successful Quality Systems (cont’d) § Meticulous data and configuration management § Revision control § Accurate description of deliverables § Careful regard for risk § Proactively characterized § Incidents are thoroughly investigated and understood § Effective corrective action § Identification § Tracking § Root cause analyses § Closure § Measurement § Desire to improve § Performance evaluation Those using the best processes will win 10 9/10/2020 Copyright 2005 Northrop Grumman Corporation

Developer Walkthroughs Milestone Reviews Example Lifecycle of a SW Build Start of of Increment GFC/C Project Increment Product Baseline Established Increment Requirements Baseline Established CINC/Site Configurations Tested IR GFC/C Increment Requirements Integration and Test Planning -x Tests Conducted • GFC/C subsystems RWTs GFC/C Products Increment Test Baseline Established TPWTs DWTs SW SWCapability. Refine Integration. Prep Integration • Object Class Definitions CWTs • Integrated Software Subsystems TWT SW SW Integration Increment Development Baseline Established Object Classes RR CWTs • Coded Object Classes Object. Class. Implementation Pacing Benchmarks 12 9/10/2020 Copyright 2005 Northrop Grumman Corporation

Said Another Way. . Requirements / Top Level Design Information Architecture Requirement s Design Walkthrough Development Software Architecture Internal Defect Reports Development Team Code Walkthrough Pacing Benchmarks Integration Team Test Walkthrough Requirements Testing 13 9/10/2020 Resolution Trouble Reports Copyright 2005 Northrop Grumman Corporation

SW Development/Production Cycle Example Initial Review (IR) Definition Phase Software Implementation Phase Test Walkthrough (TWT) Design Phase Code Walkthroughs (CWTs) Software Integration & Test Phase Requirements Test Phase Design Walkthroughs (DWTs) Pacing Benchmarks (PBMs) Release Review (RR) Product Delivery Peer Code Inspections A Proven Cycle… 14 9/10/2020 Copyright 2005 Northrop Grumman Corporation

QA Role: SW Development/Production Cycle Definition Phase • Objectives, capabilities, & requirements of the build/version are developed/reviewed • Requirements are allocated from the Requirements Document • QA Audits • QA participates in Requirement Walkthroughs • QA participates in Build Team Meetings Initial Review (IRs) • Establishes content/plans w/regard to requirements and software capability for the version • Build Lead gives approval to proceed with development • Requirements are baselined through the Change Board • QA audits • QA impounds data from IR • QA participates in Change Board activities • QA participates in Build Team Meetings Design Walkthroughs (DWTs) Design Phase • Software Architectural Design - transforms software requirements into software design at the software architectural design level • Software Detailed Design transforms software requirements into a documented software design at the detailed design level that can be directly implemented into code • QA audits • QA participates in Change Board activities • QA reviews/approves all documentation • QA participates in Build Team Meetings • Purpose is to identify defects via internal defect reports (IDRs) in the design • QA audits • QA attends and participates in peer reviews/DWTs • QA impounds data from peer reviews • QA reviews IDRs QA objectively evaluates processes and work products… 15 9/10/2020 Copyright 2005 Northrop Grumman Corporation

QA Role: SW Development/Production Cycle Software Implementation Phase • Transforms the detailed software design into software products (e. g. , code) Peer Code Inspections • Identifies defects in the software via IDRs • Ensure adherence to project standards and processes Code Walkthroughs (CWTs) • Identifies defects in the software via IDRs • QA attends and participates in CWTs • QA impounds data from CWTs • QA reviews IDRs • QA audits Software Integration & Test Phase Pacing Benchmarks (PBMs) • Integration Test integrate the software units/components into a reliable software product that operates correctly • Provides demonstrationbased progress • QA reviews/approves all documentation • QA attends PBMs as an observer • QA audits • QA impounds data from PBMs • QA audits • QA participates in change Board activities • QA reviews/approves all documentation • QA participates in Test Procedure Walkthroughs • QA attends and participates in peer reviews • QA impounds data from peer reviews • QA reviews IDRs • QA audits 16 QA identifies and documents noncompliance issues… 9/10/2020 Copyright 2005 Northrop Grumman Corporation

QA Role: SW Development/Production Cycle Requirements Test Phase Test Walkthrough (TWT) • Assesses readiness of the Product/system for Requirements Testing • CM captures configuration • CM locks configuration • Change Board establishes Test BL • Release Notices (RNs) • Code • Documentation • QA reviews all test procedures prior to test • QA participates in Change Board • QA gathers all data available to prepare for configuration verification • QA audits • Evaluates Product against the baselined requirements to determine whether requirements have been successfully met • Conducted by an independent test team • • QA witness of test QA verifies configuration QA verifies test results/data QA reviews all change paper (STRs) • QA participates in Change Board • QA audits Product Delivery Release Review (RR) • Verifies that the Product satisfies requirements, performs as specified, and is ready for release • CM captures configuration • Change Board establishes Product BL • RNs • Code • Documentation • QA impounds data “objective evidence” • QA reviews all change paper (STRs) • QA reviews all documentation (SVDD and test reports) • QA audits • QA participates in Change Board to release product to customer • QA supports verification review with customer QA provides feedback to project staff and managers… 17 9/10/2020 Copyright 2005 Northrop Grumman Corporation

Successful Process Pacing Bench Mark Schedule Example Mar Apr 3/17 3/24 3/31 4/7 May 4/14 4/21 4/28 5/5 Jun 5/12 5/19 5/26 6/2 6/9 Jul 6/16 6/23 6/30 7/7 Aug 7/14 7/21 7/28 8/4 Sep 8/11 8/18 8/25 9/1 9/8 9/15 9/22 9/30 2 a 2 b DWTs PBM 1 Design Demonstration Walkthroughs Based c/o Completed Part 1 18 PBM 3 CWTs • New Functionality • Demonstration Based c/o Part 3 Code Walkthroughs Completed PBM 2 PBM 4 • New Functionality • Demonstration Based c/o Part 2 • New Functionality • Demonstration Based c/o Part 4 PBM 5 TWT • New Functionality • Demonstration Based c/o Part 5 RR Test Release Walkthrough Review Conducted • Release Tests Conducted • Dryruns • Integration Tests • Requirements Test QA involvement throughout 9/10/2020 Copyright 2005 Northrop Grumman Corporation

Project-x First-Time Quality Example FTQ = STRs since RR FTQ = STR/KSLOC >1 Defect/KSLOC for New & Modified code Outstanding product quality in delivered SW 21 9/10/2020 Copyright 2005 Northrop Grumman Corporation

Project-x First-Time Quality Example FTQ = STRs since RR FTQ = STR/KSLOC >1 Defect/KSLOC for New & Modified code Outstanding product quality in delivered SW 22 9/10/2020 Copyright 2005 Northrop Grumman Corporation

Product Ver X STRs by Severity Example Self-assessed as GREEN SEVERITY Definitions: 1 - Crash 2 - Major function perturbation 3 - Minor function perturbation 4 - Superfluous 24 9/10/2020 Copyright 2005 Northrop Grumman Corporation

Project-x SW Defect Containment Example Code & Unit Test - Most Complex Code only Internal Integration – Primary Error Finding Process Post-RR – Inc. Field Development Test = Req/CR/TR/Regr ession IDRs TRs Evaluation of Defect Containment is done with DD and FTQ and comparison Over builds. There is no stand-alone objective scoring for green-yellow-red status. 26 9/10/2020 Copyright 2005 Northrop Grumman Corporation

How Can We Experience Failure? § Knowns (We know what we know) § Ops Data/Demonstrated Performance/Test or Analysis Data § Address by: QA involvement, sound program, engineering, and operational management, Test as you operate (understanding waiver implications) § Known Unknowns (We know what we don't know) § § Generic but unseen failure modes and hazards Unexplained and irreproducible anomalies Risk analysis uncertainties Acknowledged test and analysis limitations; unverified modeling and simulation based predictions § Address by: conservative operations, technical standards and safety factors, stress testing, attention to anomalies, trending, follow QA processes 28 9/10/2020 Copyright 2005 Northrop Grumman Corporation

How Can We Experience Failure? § Unknown Knowns (We don't know what we do know) § § Mis-communicated test or analysis results Hesitancy in coming forward with issues Poor documentation combined with loss of corporate memory Address by: clear org structure, open communications, peer review, skill retention, knowledge capture during “recertification” processes, training, QA involvement § Unknowns (We don't know what we don't know) § § § 29 9/10/2020 Bad assumptions, unclear requirements Untested new or changed environments Inadvertent operation outside of certification limits Unknowable complex interactions Address by: closer customer involvement, research and testing, challenging past assumptions, independent assessment, follow processes, QA audit processes Copyright 2005 Northrop Grumman Corporation

Failure-Maybe: Have Tools to Fill the Holes NEW PROGRAM DEFECTS DEVELOP PROCEDURES, AUDIT COMPLIANCE QA REVIEWS REQUIREMENTS INVOLVED IN SUPPLIER SELECTION S/W REQUIREMENTS VERIFICATION, S/W TEST VALIDATION ASSURANCE OF PROPER AND COMPLETE SUPPLIER REQUIREMENTS FLOWDOWN QA SITE REPRESENTATION, SOURCE INSPECTION AT SUPPLIERS RECEIVING AND LINE INSPECTIONS CALIBRATED TEST AND INSPECTION EQUIPMENT WITNESSES TESTS TO VALID TEST PLANS NONCONFORMING MATERIAL IDENTIFICATION, SEGREGATION, CONTROL, AND DISPOSITION PROPERLY DEVELOPED PRODUCT THAT WORKS, IS DEPLOYABLE AND SERVICEABLE MISSION SUCCESS 30 9/10/2020 Copyright 2005 Northrop Grumman Corporation

Summary § § § § Definition of Quality Assurance QA elements of a typical quality system infrastructure Program phases and quality system deployment Pillars of successful quality system Reviewed SW Life Cycle Reviewed QAs Role in SW Life Cycle Example Metrics Be Proactive and Communicate 31 9/10/2020 Copyright 2005 Northrop Grumman Corporation

For YOU who Like to See a Flow Diagram RWT Development Standalone Testing IR Design DWT CWT Assessment Yes Required ? Coding CASE Tool Quality Analysis Peer Code Inspection Engineering Change Board (ECB) CWT No Integration and Development Thread Testing TPWT TRs RR 33 9/10/2020 Formal Release Testing TWT Adequate Test Not Assessment Adequate ? * ? Additional Testing Copyright 2005 Northrop Grumman Corporation

QA Activities: Design and Development Fabrication and Assembly Subcontract and Supplier Management Integration and Test Acceptance and Delivery § QA verifies that project design and development activities comply with contractual requirements, policy requirements & project plans § Includes activities for project management, hardware fabrication & assembly, software design & development, CM/DM, procurement, etc. § QA verifies that work products of each design and development activity comply with established standards § 34 9/10/2020 Proces s Audits Produc t Audits Includes hardware assemblies, code, SDFs, TDFs, plans, specs, drawings, procedures, records, tools, baselines, libraries, etc. Copyright 2005 Northrop Grumman Corporation

QA Activities: Design and Development Fabrication and Assembly Subcontract and Supplier Management Integration and Test Acceptance and Delivery § QA participates in or monitors critical meetings & reviews and conducts process audits § Peer reviews/walkthroughs § Technical reviews (Technical Reviews, Test Reviews, Code turnovers) § Management reviews (Staff meetings, Sr. Mgmt, PMRs, etc. ) § Product acceptance/sell-off meetings (e. g, FCA/PCA, etc. ) § Review Boards (CCB, MRB, etc. ) § QA documents non-compliances from audits and inspections § Assures that non-compliances are communicated to the appropriate personnel, documented, resolved, and tracked to closure § Assures that nonconforming products are not used or released except as defined in project plans 35 9/10/2020 Copyright 2005 Northrop Grumman Corporation

HW QA Activities: Fab and Assembly Design and Development Fabrication and Assembly Subcontract and Supplier Management Integration and Test Acceptance and Delivery § QA performs activities on prior chart as well as the following: § Monitors Manufacturing Planning § Reviews Manufacturing Shop Orders (MSOs) to verify that § Appropriate inspection points are included § Appropriate instructions or workmanship standards are referenced § Parts lists identify (per drawing) correct part numbers, revisions and trace code § Quality inspection planning stamp is applied to documents that have been approved by inspection planning § Participates in Material Review Board (MRB) 36 9/10/2020 Copyright 2005 Northrop Grumman Corporation

HW QA Activities: Fab and Assembly Design and Development Fabrication and Assembly Subcontract and Supplier Management Integration and Test Acceptance and Delivery § Performs hardware fabrication and assembly inspections § Verify Hardware component/parts/assembly conform to current drawings § Verify Visual and Identification § Verify ESD Requirements § Generate Inspection Records § Monitors the use of Measurement Test and Support Equipment (MT/SE), when applicable, to verify that § Test procedures using MT/SE are approved and controlled § MT/SE is calibrated in accordance with the manual § When MT/SE is inoperative, inaccurate, or malfunctions, or when its calibration period has expired, the equipment is segregated and designated as “Invalid for Use” § Suspect MT/SE used during testing are reviewed to determine the extent of possible impact upon product conformance 37 9/10/2020 Copyright 2005 Northrop Grumman Corporation

QA Act: Subcontract and Supplier Mgmt Design and Development Fabrication and Assembly Subcontract and Supplier Management Integration and Test Acceptance and Delivery § On projects with subcontractors, project quality requirements are flowed to the subcontractor in two ways: § QA requirements may be stated explicitly in the subcontract SOW § Quality requirements may be defined in a PAR, SMART, or equivalent document that must be referenced in the subcontract SOW § Where specific quality requirements are to be imposed on suppliers (e. g. compliance with ISO 9001/CMMI), quality clauses are included in the Purchase Order and/or SOW 38 9/10/2020 Copyright 2005 Northrop Grumman Corporation

QA Act: Subcontract and Supplier Mgmt Design and Development Fabrication and Assembly Subcontract and Supplier Management Integration and Test Acceptance and Delivery § QA performs procurement document reviews (PDRs) § Assures that quality requirements are provided to suppliers, as required § Assures that drawings or specifications provided to suppliers are accurate, controlled and of the correct version § QA may be required to perform source surveys of subcontractors/suppliers to determine process capability prior to procurement § QA performs in-process surveillance inspections as defined in subcontract or procurement document § Assures that contractually-defined processes are followed § Assures that contractually-specified product quality is met 39 9/10/2020 Copyright 2005 Northrop Grumman Corporation

QA Act: Subcontract and Supplier Mgmt Design and Development Fabrication and Assembly Subcontract and Supplier Management Integration and Test Acceptance and Delivery § QA performs receiving inspections on procured items, GFE and COTS to § Verify that was procured or requested is what was received § Verify that procured item meets its procurement specifications § Verify that GFE meets contract requirements § Verify that the condition of the items is acceptable (e. g. , not damaged) § QA provides oversight of subcontractor QA activities as defined in the subcontract 40 9/10/2020 Copyright 2005 Northrop Grumman Corporation

QA Activities: Integration and Test Design and Development Fabrication and Assembly Subcontract and Supplier Management Integration and Test Acceptance and Delivery § QA monitors integration and test activities to assure that: 41 § Test documentation (plans, procedures, results) are reviewed § Test support software is tested and qualified prior to start of qualification/acceptance testing and MT/SE is properly calibrated § Only the latest released versions of the test procedures and test support software used § Test readiness reviews are held to identify and/or resolve open issues § The customer’s quality personnel, when required, are apprised of test scheduling and performance issues § The integration and test activities are monitored as defined in the test planning documentation 9/10/2020 Copyright 2005 Northrop Grumman Corporation

QA Activities: Integration and Test Design and Development Fabrication and Assembly Subcontract and Supplier Management Integration and Test Acceptance and Delivery § For Qualification/Acceptance Tests, QA assures that: § Test anomalies or problems are documented and resolved per project plans § Test Data Files (TDFs) and other records of tests are initiated and maintained as described in the project plans § Changes made to the test procedures during testing are documented and controlled § The hardware/software configuration for the test is defined in the test plan/procedure and is verified to be accurate prior to test initiation § Tested hardware/software meets all of the performance parameters specified in the test plan/procedure § The final test report accurately reflects what occurred during the test § All actions have been closed or are identified as being open in the test report § There appropriate approvals of the test report § Test failure reviews (lessons learned) are performed 42 9/10/2020 Copyright 2005 Northrop Grumman Corporation

QA Activities: Acceptance and Delivery Design and Development Fabrication and Assembly Subcontract and Supplier Management Integration and Test Acceptance and Delivery § For product deliveries, QA assures that: § Items to be delivered/shipped: § Have received a planned level of acceptance § Waivers/deviations have been documented § The hardware/software configuration is correct and properly identified and marked § Are, in fact, present § Packaging and container marking is proper § Delivery/shipping documentation is completed and coordinated in accordance with contract requirements § Customer Certification, if applicable, is included 43 9/10/2020 Copyright 2005 Northrop Grumman Corporation