9 SYSTEM INTEGRATION and Verification Validation Software Engineering

9. SYSTEM INTEGRATION and Verification & Validation

Software Engineering Roadmap: Chapter 9 Focus Construct system in stages Identify corporate practices - Plan integration of parts to yield whole - Test subassemblies - Assemble in “builds” - Test whole system in a variety of ways Plan project Maintain Analyze requirements Integrate & test system Design Implement Test units Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission.

Chapter Learning Goals • Understand link between testing and integration • Be able to plan the integration of modules • Understand types of testing required • Be able to plan and execute testing – beyond the unit level Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission.

Verification vs validation • Validation: "Are we building the right product" – The software should do what the user really requires (the requirements should express this) • Verification: "Are we building the product right" – The software should conform to the architecture and design as driven by the requirements

Two methods of V & V • Software inspections Concerned with analysis of the static system representation to discover problems (static verification) – May be supplement by tool-based document and code analysis • Software testing Concerned with exercising and observing product behaviour (dynamic verification) – The system is executed with test data and its operational behaviour is observed

V& V Goals • Verification and validation should establish confidence that the software is ready for use • This does NOT mean completely free of defects • Rather, it must be good enough for its intended use and the type of use will determine the degree of confidence that is needed

V & V Planning • Careful planning is required to get the most out of testing and inspection processes • Planning should start early in the development process – during requiremenst analysis • The plan should identify the balance between static verification and dynamic testing

1. Introduction to system integration Before we can talk more about V & V we have to discuss an approach to integration

Unified Process for Integration & Test Jacobson et al: USDP Inception Elaboration Construction Transition Requirements Analysis Design Implementation Integration Unit Tests Test Prelim. Iter. iterations #1 Iter. #n #n+1 Integration tests. . . System tests …. . Iter. …. . #m #m+1 Iter. #k

Customer loss of information … Why? Development Overview Requirements loss Architecture loss Detailed design Interface specs loss Function code loss Module (e. g. , package) code After Myers System code

Requirements Docs. Architecture Testing Overview: Artifact Flow System Tests Detailed design Docs. order of testing (11) Acceptance tests (10) Installation tests (9) Usability tests (8) System tests (7) Regression tests Integration Interface specs Tests (6) Integration tests Function code Docs. (5) Interface tests Code + Docs. Code + Module (e. g. , package) code Code + Iteration or System code Code + Unit Tests (2), (3) Module tests (1), (4) Function tests Complete code

(11) Acceptance tests* Requirements Tested against requirements (“validation”) (10) Installation tests*‡ (9) Usability tests*‡ (8) System tests*‡ (7) Regression tests* Testing for Validation and Verification (After Myers) Includes… : *use-cases ‡performance testing (1), (4) Function tests

Testing for Validation and Verification (After Myers) Requirements “validation 1” (11) Acceptance tests* (10) Installation tests*‡ (9) Usability tests*‡ Architecture “verification 2” (8) System tests*‡ (7) Regression tests* Interface specs “verification 2” (6) Integration tests* (5) Interface tests Detailed design “verification 2” (1), (4) Function tests (2), (3) Module tests Note 1: Tested against requirements Note 2: Tested against documents indicated Includes… : *use-cases ‡performance testing

2. The integration process

The Build Process for a Bridge Builds for single level iteration Double level iteration . . Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission.

Design Integration in Spiral Development First iteration Requirements analysis Second iteration 2. Design for additional requirements 1. Get additional requirements 3. Code additional 5. Test 4. Integrate new code Implementation Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission. Test

Relating Builds and Iterations in the Unified Process Jacobson et al: USDP Inception Elaboration Construction Transition Requirements Analysis Design Implementation First build for iteration i Last build for iteration i Test Prelim. Iter. iterations #1 Iter. …. . #n #n+1 #i #m #m+1 Iter. #k

Build Sequences: Ideal vs. Typical Build a Bridge Build a Video Game

1. Understand the architecture decomposition. – try to make architecture simple to integrate 2. Identify the parts of the architecture that each iteration will implement. – – – build framework classes first, or in parallel if possible, integrate “continually” build enough UI to anchor testing document requirements for each iteration try to build bottom-up • so the parts are available when required Plan Integration & Builds – try to plan iterations so as to retire risks • biggest risks first – specify iterations and builds so that each use case is handled completely by one 3. Decompose each iteration into builds if necessary. 4. Plan the testing, review and inspection process. 5. Refine the schedule to reflect the results. Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission.

1. Decide extent of all tests. 2. For each iteration … 2. 1 For each build … 2. 1. 1 Perform regression testing from prior build 2. 1. 2 Retest functions if required 2. 1. 3 Retest modules if required 2. 1. 4 Test interfaces if required 2. 1. 5 Perform build integration tests -- section 3. 1 Development of iteration complete 2. 2 Perform iteration system and usability tests -- sections 3. 4, 3. 5 System implemented 3. Perform installation tests -- section 3. 8 System installed 4. Perform acceptance tests -- section 3. 7 Job complete Roadmap for Integration and System Test

Framework layer Game. Artifacts RPG Video Game Architecture Packages - built around the domain classes Game. Environment «framework package» Role. Playing. Game Characters «framework package» Application layer 2 Encounter. Characters «uses» Encounter. Game 3 «application package» Encounter. Character «uses» Encounter. Game Refers to both Engagement. Display «uses» Player. Character Foreign. Character Player. Quality. Window Encounter. Environment 1 «application package» Area Encounter. Area. Connection. Hyperlink Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission.

Factors Determining the Sequence of Integration • Usage of modules by other modules technical (factors) – build and integrate modules used before modules that use them • Defining and using framework classes • Exercising integration early risk reduction • Exercising key risky parts of the application as early as possible • Showing parts or prototypes to requirements customers Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission.

Integration Month 1 Month 2 Month 3 Month 4 Month 5 Schedule 1 2 3 4 1 2 3 4 1 Milestones Iterations Proto. requirements Inception iteration Iteration 1 “view characters in areas” Builds Modules build 1 build 2 Complete prototype Elaboration iterations Iteration 2 “elementary interaction” build 3 . . . . Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission.

Integration Month 1 Month 2 Month 3 Month 4 Month 5 Schedule 1 2 3 4 1 2 3 4 1 Milestones Iterations Proto. requirements Inception iteration Iteration 1 “view characters in areas” Builds build 1 Game. Environment package Modules build 2 Integrate & test Elaboration iterations Iteration 2 “elementary interaction” build 3 Role. Playing. Game package Characters package Encounter. Environment package Complete prototype Encounter Characters package Encounter. Game package Integrate & test Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission. Integrate & test

3. The testing process

Encounter Continual Integration week 3 Role. Playing. Game Encounter. Game Characters Game. Character Layout Encounter. Characters Encounter. Layout Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission.

Encounter Continual Integration week 7 Role. Playing. Game Encounter. Game Characters Game. Character Layout Encounter. Characters Map Encounter. Layout Encounter. Environment «facade» Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission.

Encounter Continual Integration week 11 Role. Playing. Game RPGame Encounter. Game Characters Game. Character Layout Encounter. Characters Map Encounter. Layout Encounter. Cast Encounter. Environment «facade» Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission.

Encounter Continual Integration week 15 Role. Playing. Game RPGame Encounter. Game Characters Game. Character Encounter. Game «facade» Layout Encounter. Characters Encounter. Character Map Location Encounter. Layout Encounter. Cast Encounter. Environment «facade» Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission. Area

Plan and Execute Integration Tests 1. Decide how and where to store, reuse and code the integration tests. – show this in the project schedule 2. Execute as many unit tests (again) as time allows – this time in the context of the build • no drivers or stubs required this time – prioritize by those most likely to uncover defects 3. Exercise regression tests – to ensure existing capability has not been compromised 4. Ensure build requirements are properly specified 5. Exercise use cases that the build should implement – test against the SRS 6. Execute the system tests supported by this build Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission.

Relationship between Use Cases, Iterations and Builds … 4 Iteration 5 5. 2 Use case 7 build 5. 3 … 6 5. 4 14 builds on details of 7 * Use case 14 * Use case 9 * Use case 23 * «extends» or «includes» Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission.

Final Code Build and Integration Schedule: a Banking Example week 23 weekly code builds Twice weekly Over night week 31 biweekly daily baseline Frequency of regression testing increases toward the end of project. release

Final Code Build and Integration Schedule: a Banking Example week 23 week 31 weekly code builds biweekly daily baseline bank query module tasks bank deposit module Integration of module into baseline bank withdrawal module Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission. release Code is frozen when being tested. Why?

Typical Day-by-Day Code Integration Process Run regression tests development 6 pm 7 am development time Freeze development weekly builds Confirm baseline or revert to previous baseline biweekly daily week week week 23 24 25 26 27 28 29 30 31 = overnight regression test Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission. release

Artifacts and Roles for Integration Testing* Test engineer Component engineer Integration System tester . . . . responsible for. . . . Use-case model Test evaluation Test plan Test case *Jacobson et al: USDP Test procedure Test component Defect management

Success in Interface Testing • Understand interface requirements • Perform early “smoke” tests to weed out unanticipated interface issues • Test thoroughly as per “black box” testing (see Unit Testing) so as to fully exercise the interface in terms of: – Variety of values for each parameter – Parameter combinations – Volume and timing

Interface Testing for Encounter Interface. Test «test class» References all packages Encounter. Characters Encounter. Cast Encounter. Game «facade» get. The. Encounter. Game() get. State() set. State() handle. Event() Encounter. Environment «facade» get. The. Encounter. Cast() get. The. Player. Character() get. The. Foreign. Character() Encounter. Environment «facade» set. Location() Class relationships not fully shown. Area Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission.

Types of System Tests* 1 • Volume Subject product to large amounts of input. • Usability Measure user reaction (e. g. , score 1 -10). • Performance Measure speed under various circumstances. • Configuration Configure to various hardware / software e. g. , measure set-up time. • Compatibility -- with other designated applications e. g. , measure adaptation time. • Reliability / Availability Measure up-time over extended period. *see Kit [Ki]

• Security Types of System Tests* 2 Subject to compromise attempts. • e. g. , measure average time to break in. • Resource usage Measure usage of RAM and disk space etc. • Install-abililty Install under various circumstances. • measure time to install. • Recoverability Force activities that take the application down. • measure time to recover • Serviceabililty Service application under various situations. • measure time to service • Load / Stress Subject to extreme data & event traffic * see Kit [Ki]

4. Documenting integration and tests

System Test Documentation in Context SPMP Software Project Management Plan SCMP Software Configuration Management Plan -- statement of procedures Appendix -- references all versions of documents SDD Source Code Software Design Document STD Software Test Documentation SRS Software Requirements Specification Key: Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission. “Refers to”

Organization of Integration and System Test Documentation** STD‡ SCMP Software Test Documentation consists of. . . Integration T. D. ‡ System T. D. ‡ Acceptance T. D. ‡ references. . . Build 1 T. D. ‡ Build 2 T. D. ‡ Build 3 T. D. ‡ Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission. Installation T. D. ‡ ‡ Test Documentation, each divided into: • Introduction • Test plans • Test designs • Test cases • Test procedures • Test log ** With R. Bostwick

ANSI/IEEE 829 -1983 Software Test Documentation (reaff. 1991) 1. Introduction 2. Test plan items under test, scope, approach, resources, schedule, personnel 3. Test design items to be tested, the approach, the plan in detail 4. Test cases sets of inputs and events 5. Test procedures steps for setting up and executing the test cases 6. Test item transmittal report items under test, physical location of results, person responsible for transmitting 7. Test log chronological record, physical location of test, tester name NOT REQUIRED IN PROJECTS 8. Test incident report documentation of any event occurring during testing which requires further investigation 9. Test summary report summarizes the above

5. The Transition iterations

Goals of the Transition Iterations Transition Requirements h Find defects through customer use h Test user documentation and help Analysis h Determine realistically whether application meets customer requirements Design Implementation h Retire deployment risks Test Iter. … Iter. #m+1 #k h Satisfy miscellaneous marketing goals

Alpha- and Beta- Releases Alpha Beta In-house and highly trusted users – Several repetitions by real users – Previews customer reaction – Benefits third-party developers – Forestalls competition Selected customers – Many repetitions by real users – Gets customer reaction Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission.

Roadmap for the Transition Iterations • Define population 1. Plan alpha and beta testing. • Plan defect collection • Identify stopping criteria (num. allowed defects/test or /hour) 2. Conduct alpha testing. • Prepare • Distribute & install • Carry out (users / customers) • Gather defect reports 3. Conduct beta testing. • Observe stopping criteria • Correct defects Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission.

Number per 1000 hrs Stopping Criteria: Graphical Representation Error detection rate Target: <= 7 per 1000 hrs for 4 weeks % Percentage of deposit transaction Target: 91% types tested Target: 98% Percentage of withdrawal transactions tested week 10 week 20 Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission. End tests

7. Tools for integration and system testing

Capabilities of Automated System Test Tools 1. Record mouse and keyboard actions to enable repeated playback 2. Run test scripts repeatedly 3. Enable recording of test results 4. Record execution timing 5. Record runtime errors 6. Create and manages regression tests 7. Generate test reports 8. Generate test data 9. Record memory usage 10. Manage test cases 11. Analyze coverage

Types of Capture/Playback Tests* • Native / software intrusive test software intermingled with software under test – could compromise software under test – least expensive • Native / hardware intrusive test hardware intermingled with software under test – could compromise software under test • Non-intrusive uses separate test hardware – does not compromise software under test – most expensive * adapted from Kit [Ki]

Memory Usage Test Tools • Memory leaks – detect growing amounts of unusable memory • inadvertently caused by the implementation • Memory usage behavior – confirm expectations – identify bottlenecks • Data bounds behavior – e. g. , confirm integrity of arrays – e. g. , detect attainment of limiting values • Variable initialization – indicate un-initialized variables • Overwriting of active memory

Test Case Management* • Provide user interface – to manage tests • Organize tests – for ease of use – for maintenance • Manage test execution sessions – to run tests selected by user • Integrate with other test tools – to capture and play back – to analyse coverage • Provide reporting • Provide documentation * adapted from Kit [Ki]

Case study - SCMP: Appendix A Plan for Integration Baselines

Integration Plan Build 3 Encounter. Game Build 1 Encounter. Game Encounter. Characters Encounter. Cast Build 2 Encounter. Environment Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission.

Build 1 Game. Characters «framework package» Game. Character Encounter Characters Encounter. Character Encounter. Cast «façade» «singleton» Player. Character Foreign. Character Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission.

Build 2 Game. Environment Game. Area. Connection Game. Area Game. Character Game. Layout Area Encounter. Environment «facade» Encounter. Environment Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission. Encounter. Cast «facade»

Status After Build 2 Characters «framework package» Game. Environment «framework package» Encounter. Characters Encounter. Layout Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission.

Role. Playing. Game Build 3 -- Includes Builds 1 and 2 (not shown) RPGMouse. Event. Listener notify. Of. Event() RPGame. State handle. Event() Encounter. Game <<singleton>> Engagement. Display Engaging Waiting handle. Event() Reporting Preparing handle. Event() Encounter. Game Adapted from Software Engineering: An Object-Oriented Perspective by Eric J. Braude (Wiley 2001), with permission. Key: Domain class