Systems Engineering and Analysis Chapter 6 System Test

Systems Engineering and Analysis Chapter 6 System Test, Evaluation, and Validation 12/22/2021 rd 1

OBJECTIVES LEARNING AFTER STUDYING THIS CHAPTER, YOU SHOULD BE ABLE TO: Determine requirements for system test, evaluation and validation; Describe categories of system test and evaluation; Plan for system test and evaluation Prepare for system test and evaluation Conduct system test, collect test data, prepare test report; Incorporate system modifications as needed. 12/22/2021 rd 2

Donald Rumsfeld The Unknown "As we know, There are knowns. There are things we know. We also know There are known unknowns. That is to say We know there are some things We do not know. But there also unknowns, The ones we don't know We don't know. " 12/22/2021 rd 3

Sayings The uniforms of this conflict will be bankers' pinstripes and programmers' grunge just as assuredly as desert camouflage. - Donald Rumsfeld If this is coffee, please bring me some tea. If this is tea, please bring me some coffee. - Abraham Lincoln One dies only once, and it's for such a long time - Moliere Mathematicians write symphonies. Scientists create instruments. Engineers build music boxes. 12/22/2021 rd 4

Test and Evaluation Master Plan Ensures that all aspects of the system are adequately tested and can be implemented; documents the scope, content, methodology, sequence, management of, and responsibilities for test activities. Acceptance tests will be performed in a test environment that duplicates the production environment as much as possible. Unit/Module Testing Subsystem Integration Testing Independent Security Testing Functional Qualification Testing User Acceptance Testing Beta Testing 12/22/2021 rd 5

Requirements for System T&E Started in conceptual phase with operational requirements Identify and prioritize TPMs How will we be able to determine if each requirement will be met, and what test and evaluation approach should be used to verify so? Test and Evaluation Master Plan (TEMP) Evaluation is recursive and iterative CAD for interfaces of components, mock-ups later. Testing philosophy in the operating environment. SDI example (jury-rigged) 12/22/2021 rd 6

Test and Evaluation Master Plan Provides a road map for integrated simulation, test, and evaluation plans, schedules, and resource requirements necessary to accomplish T&E program objectives. Integrative testing Design of test equipment equivalent to design of component. 12/22/2021 rd 7

Stages of System Evaluation Conceptual Preliminary Detail Production Operations Analytical breadboard production model full continuous Commercial off-the-shelf (COTS) not much testing needed. Software programming tests for example. Analytical (CAD, CAE, CAM, CAS) Type 1 Testing -- early phase testing Type 2 Testing -- initial qualification testing of TPMs Type 3 Testing -- initial system test prior to production Type 4 Testing -- deployment testing 12/22/2021 rd 8

Type 1 Testing Evaluating system components in the laboratory Breadboards, Bench and service tests, Engineering models, rapid prototyping Mockups to verify performance of physical design characteristics. Entire System or Component level (jury-rigged) Software – data structures 12/22/2021 rd 9

Type 2 Testing Prototype formal demonstrations (towards end of detail design phase) Initial qualifications of the system for operational use. Performance tests – (form, fit, interchangeability), safety Environmental – temperature, shock, vibration, humidity, salt spray, dust and sand, fungus, acoustic noise, and vice-versa effects on environment. Structural tests – Stress, strain, fatigue, bending, torsion. Reliability – MTBF, MTBM, component life, determine modes of failure. Maintainability – mean active (corrective, preventive, maintenance) time, MLH/OH Support equipment compatibility Personnel test and evaluation between human and equipment Technical data verification Software verification 12/22/2021 rd 10

Type 3 Testing Conducted by user personnel at some operational site. Continuous testing over a a period of time for entire system on an integrated basis. Check for compatibility of prime and support equipment. Measure TPMs. Almost a complete operational situation. 12/22/2021 rd 11

Type 4 Testing Specific area testing Try to determine if operational availability (Ao) can be improved. True capability assessed. Look for possible improvements. Software example. 12/22/2021 rd 12

Four Basic Test Categories Development Test – proof of concept test on new items Qualification Test – verify the design on the first item made Acceptance Test – conducted before giving to customer. Operational Test – conducted to validate specification requirements in the actual environment. 12/22/2021 rd 13

Verification Activities Analysis ~ modeling and simulation of requirements to determine compliance with design Inspection ~ visual determination physical design Demonstration ~ usually jury-rigged by lack of detail data Test ~ data gathering Associated Risks with each activity. 12/22/2021 rd 14

Developing a System T&E Plan 1. Identify all test items with schedule, inputs, expected outputs, sequencing with an evolving, integrative plan. 2. Identify responsible organization, personnel, facilities. 3. Describe test location and environment. 4. Describe test preparation phase for each type test. 5. Describe the formal test phase (what one is looking for) 6. Retesting contingency depending on results. 7. Describe final test report (format, distribution, feedback and recommendations for corrective actions. 12/22/2021 rd 15

The "-ilities" Affordability (life cycle cost) Supportability Reliability (RAMS) => Dependability Maintainability Availability Disposability Producibility (manufacturability) Reconfigurability (flexibility / standardization) Security Survivability Serviceability Vulnerability Packaging, Handling, Storage and Transportation (PHS&T) 12/22/2021 rd 16

Planning for System Test and Evaluation Begins in conceptual design phase When a need established, a test to measure is conceived. TEMP as part of SEMP. Testing is costly and experimental design is needed to integrate testing. Integrate what to test, how to test, who is testing, where to test, if to retest, data collection and analysis, Final test report 12/22/2021 rd 17

Preparation for System T & E Selection of test components for utmost realism Procedures - both operational and maintenance tasks leading to technical manuals. Test-site selection – single versus multiple, tropics or poles Test Personnel and Training -- actual operators and support personnel (OJT). Test Facilities and Resources – May have to be created. Test and Support Equipment – Maintenance Concept Test Supply Support – spares, repairables, consumables Conducting Test – true operational test of what to measure. Data Collection 12/22/2021 rd 18

Measuring Effectiveness System-effectiveness – performance, availability, dependability, supportability, related technical factors, but not necessarily cost Operational availability – system up and down time or MTBM and MDT (maintenance down time) Life-cycle cost – all future cost for R&D, design, T&E, production, utilization, maintenance, support, disposal, etc. Reliability – MTBF, operating time, failures, causes of failures, and repairs 12/22/2021 rd 19

Test Performance and Validation How well it performed and did it accomplish the mission? What is the true effectiveness of the system? What is the true effectives of the system support capability? Does system meet requirements through specified TPMs? Does the system meet all consumer requirements? 12/22/2021 rd 20

Summary System Test, Evaluation, and Validation Needs design --> configuration tests Distinguish data from information and seek specific information from a well-design data collection plan that provides the right information in the right format, to the right location, at the right time, and the right cost. Seek integrative testing to reduce cost. Provide feedback by communicating test results. 12/22/2021 rd 21

Terms and Concepts Test Plan. The test plan - set of ideas that guide or represent the intended test process. Often those ideas are only partially documented, spread across multiple documents, and subject to change as the project evolves. Test Plan Document. A test plan document is any document intended to convey test plan information. However, test plan documents are not the only source of information about the test plan. Test plan information is also contained in the oral tradition of the project and the culture of the company. Test Strategy. The test strategy is the way tests will be designed and executed to support an effective quality assessment. Test strategy is the plan for what parts of the product will be covered by tests and what test techniques will be used. Test strategy is distinct from the logistics of implementing the strategy. Test strategy is essentially the “brains” of the test process. Test Project. The test project is the means by which the test strategy is implemented and results delivered. The test project is the “brawn” of the test process. 12/22/2021 rd 22