CS 5103 Software Engineering Lecture 14 System Testing

CS 5103 Software Engineering Lecture 14 System Testing Coverage

Higher level testing Ø Integration Testing Ø Ø System Testing Ø Ø 2 Testing the interaction among a number of interactive components Testing the system as a whole, considering various environments, external exceptions Acceptance Testing Ø Final user testing Ø Usually on GUI

Integration Testing Ø 3 Strategies Ø Big Bang Ø Top down Ø Bottom Up

Big Bang! 4 Ø Prepare all relevant components Ø Data, Global variables… Ø Put them together Ø Pray!

Usage Scenario of Big Bang Ø Quite common in small projects Ø Requires no extra effort for integration Ø 5 May work well if all interfaces are well-defined … Not likely to happen…

Top down strategy Ø Feature decomposition Ø Ø 6 A hierarchical structure of all software features May not require extra efforts: You should use it in requirement and design

An example feature decomposition 7

Top down strategy Ø Focus on the validation of the whole system first Ø Test an empty system with no components at first Ø Ø Ø Gradually add components Ø Until all components are added Advantages Ø Ø 8 All components are replaced with test doubles, e. g. , stubs Easier to understand the process of the integration May have a working system earlier (or always have a working system): important for modern development, such as XP

Top-down strategy: Illustration 9

Top down strategy Ø Disadvantages Ø Requires to write test doubles Ø Ø Centralized Ø 10 Since DOC matters in integration, so more complex test doubles is usually required So the integration cannot be done parallelly

Bottom-up strategy Ø Focus on the integration of lowest units at first Ø Ø Ø 11 Start from an unit depending nothing else When all sub-unit of a component are integrated, integrate the component Until the whole system is built

Bottom-up strategy: Illustration Bottom Level Subtree Middle Level Subtree Top Level Subtree 12

Bottom-up strategy Ø Advantages Ø Ø No test stub is required Requires test driver but may re-use unit test cases (view the whole component as a unit) Support parallel integration Issues Ø No working system is available until the end Ø Ø 13 Higher risk Require more interactions among sub-teams working on each component

System testing 14 Ø Test the system as a whole Ø Usually test against specifications Ø For each item in the specification Ø Work out a test case and a test oracle Ø Test boundary values Ø Test with invalid inputs Ø Test with environment errors

Environment issues Ø Ø Building Ø Compile options and configurations Ø Environment variables Ø Third party dependencies Underlying platforms Ø Ø 15 OS, database, application server, browser Compatibility is a huge problem Ø Different platforms (if cross-platform) Ø Different versions of dependencies and platforms Ø Different configurations

Acceptance Testing Ø Command Line Ø Writing test scripts Ø With different options and inputs Ø Ø 16 Try invalid cases : missing option parameter, invalid paths, etc. GUI Testing

GUI testing Ø Ø 17 Testing Graphics User Interface is easier Ø But harder to automate Ø And harder to compare results with oracles Manual testing is still widely performed for GUI testing Ø Manually explore the user interface Ø Record the steps in the test for future testings Ø Observe the GUI for errors

GUI testing Ø Record and Replay Ø Ø Screen record and replay Ø Record click on the screen and keyboard inputs Ø Replay all the clicks and inputs Ø Not robust, affected by screen size, resolution, resize, … Event record and replay Ø Ø Ø 18 Record all the UI events in the UI framework (swing, MFC, android etc. ), and outputs (texts) in the UI Re-trigger all the events and compare the UI texts More robust, but requires more preparation and overhead

Today’s class 19 Ø Higher Level Testing Ø Software Testing Coverage Ø Code Coverage Ø Input Combination Coverage Ø Mutation Coverage

Test Coverage Ø Ø The most straightforward: input coverage Ø # of inputs tested / # of possible inputs Ø Ø 20 After we have done some testing, how do we know the testing is enough? Unfortunately, # of possible inputs is typically infinite Not feasible, so we need approximations…

Test Coverage 21 Ø Code Coverage Ø Specification Coverage Ø Model coverage Ø Error coverage

Code Coverage Ø Basic idea: Ø Ø Ø So the test suite is definitely not sufficient Definition: Ø Ø 22 Bugs in the code that has never been executed will not be exposed Divide the code to elements Calculate the proportion of elements that are executed by the test suite

Code Coverage Ø Criteria Ø Ø 23 Statement (basic block) coverage, are they the same? Branch coverage (cover all edges in a control flow graph), same with basic block coverage? Ø Data flow coverage Ø Class/Method coverage

Control Flow Graph How many test cases to achieve full statement coverage? 24

Statement Coverage in Practice Ø Ø Ø 25 Microsoft reports 80 -90% statement coverage Safely-critical software must achieve 100% statement coverage Usually about 85% coverage, 100% for large systems is usually very hard

Statement Coverage: Example 26

Branch Coverage Ø Ø Ø 27 Cover the branches in a program A branch is consider executed when both (All) outcomes are executed Also called multiple-condition coveage

Control Flow Graph How many test cases to achieve full branch coverage? 28

Branch Coverage: Example 29

Branch Coverage: Example An untested flow of data from an assignment to a use of the assigned value, could hide an erroneous computation Even though we have 100% statement and branch coverage 30

Data Flow Coverage Ø Cover all def-use pairs in a software Ø Def: write to a variable Ø Use: read of a variable Ø Use u and Def d are paired when d is the direct precursor of u in certain execution 31

Data Flow Coverage Ø Formula Ø Not easy to locate all use-def pairs Ø Easy for inner-procedure (inside a method) Ø Very difficult for inter-procedure Ø 32 Consider the write to a field var in one method, and the read to it in another method

Path coverage Ø 33 The strongest code coverage criterion Ø Try to cover all possible execution paths in a program Ø Covers all previous coverage criteria? Ø Usually not feasible Ø Exponential paths in acyclic programs Ø Infinite paths in some programs with loops

Path coverage Ø N conditions Þ 2 N paths Ø Many are not feasible Ø e. g. , L 1 L 2 L 3 L 4 L 6 X = 0 => L 1 L 2 L 3 L 4 L 5 L 6 X = -1 => L 1 L 3 L 4 L 6 X = -2 => L 1 L 3 L 4 L 5 L 6 34

Control Flow Graph How many paths? How many test cases to cover? 35

Path coverage, not enough 1. main() { 2. int x, y, z, w; 3. read(x); 4. read(y); 5. if (x != 0) 6. z = x + 10; 7. else 8. z = 1; 9. if (y>0) 10. w = y / z; 10. else 11. w = 0; 12. } 36 Test Requirements: – 4 paths • Test Cases – (x = 1, y = 22) – (x = 0, y = 10) – (x = 1, y = -22) – (x = 1, y = -10) • We are still not exposing the fault ! • Faulty if x = -10 – Structural coverage cannot reveal this error

Method coverage Ø So far, all examples are inner-method Ø Ø Ø 37 Quite useful in unit testing It is very hard to achieve 100% statement coverage in system testing Ø Need higher level code element Ø Method coverage Similar to statements Ø Node coverage : method coverage Ø Edge coverage : method invocation coverage Ø Path coverage : stack trace coverage

Method coverage 38

Code coverage: summary Ø Coverage of code elements and their connections Ø Node coverage: Ø Ø Edge coverage: Ø Ø Branch/Dataflow/Method. Invok Path coverage: Ø 39 Class/method/statement/predicate coverage Path/Use. Def. Chain/Stack. Trace

Code coverage: limitations Ø Not enough Ø Ø 40 Some bugs can not be revealed even with full path coverage Cannot reveal bugs due to missing code

Code coverage: practice Ø 41 Though not perfect, code coverage is the most widely used technique for test evaluation Ø Also used for measure progress made in testing Ø The criteria used in practice are mainly: Ø Method coverage Ø Statement coverage Ø Branch coverage Ø Loop coverage with heuristic (0, 1, many)

Code coverage: practice Ø Far from perfect Ø Ø Ø A lot of corner (they are not so corner if just not found by statement coverage) cases can never be found 100% code coverage is rarely achieved Ø Ø 42 The commonly used criteria are the weakest, recall our examples Some commercial software products released with around 60% code coverage Many open source software even lower than 50%

Thanks! 43