Outline Types of errors Component Testing Unit testing

Outline • Types of errors • Component Testing • Unit testing • Integration testing • Testing Strategy • Design Patterns & Testing • System testing • Function testing • Structure Testing • Performance testing Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 1

How do we deal with Errors & Faults? Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 2

Verification? Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 3

Modular Redundancy? Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 4

Declaring the Bug a Feature? Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 5

Patching? Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 6

Testing? Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 7

Examples of Faults • Interface specification Faults • Expectation mismatch between the client & the server Cannot happen with Jini • Mismatch between requirements & implementation • Algorithmic Faults • Mechanical Faults • Documentation does not match actual conditions • Timing errors • Throughput / performance errors Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 8

Some Observations • It is impossible to completely test any nontrivial module • Theoretical limitations: Halting problem • Practical limitations: Prohibitive in time & cost • Testing only shows the presence of bugs, not their absence Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 9

Testing Requires Creativity • Is testing dirty work? • An effective test requires: • Detailed understanding of the system • Knowledge of test techniques • Skill to apply these techniques in an effective & efficient manner • Book says: Testing is done best by independent testers • XP says: developers devise/do unit tests Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 10

Testing Activities Code Unit Tested Subsystem Requirements Analysis Document System Design Document Integration Test Integrated Subsystems Functional Test User Manual Functioning System Tested Subsystem Code Unit Test Bernd Bruegge & Allen Dutoit All tests by developer Object-Oriented Software Engineering: Conquering Complex and Changing Systems 11

Testing Activities … Client’s Understanding of Requirements Global Requirements Validated Functioning System Performance. System Tests by developer Bernd Bruegge & Allen Dutoit Accepted System Acceptance Test User Environment Installation Tests by client Object-Oriented Software Engineering: Conquering Complex and Changing Systems 12

Fault Handling Techniques Fault Handling Fault Avoidance Design Methodology Verification Fault Detection Fault Tolerance Atomic Transactions Reviews Modular Redundancy Configuration Management Testing Component Testing Integration Testing Correctness Debugging Bernd Bruegge & Allen Dutoit System Testing Performance Debugging Object-Oriented Software Engineering: Conquering Complex and Changing Systems 13

Component Testing • Unit Testing • Developers confirm that class functions correctly. • Integration Testing • Developers confirm that subsystem interface functions correctly Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 14

System Testing • Developers confirm that the system meets its requirements • Acceptance Testing • Clients confirm that the system meets its requirements Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 15

Unit Testing • Informal • Incremental coding • Static Analysis • Hand execution: Read the source code • Walk-Through: Informal presentation to others • Code Inspection: Formal presentation to others • Testing • Black-box testing: Test the input/output behavior • White-box testing: Test the internal logic of the class Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 16

Black-box Testing • I/O behavior • If for any input, we can predict the output, the module passes the test. • Generally intractable to test all possible inputs • Reduce number of tests by equivalence partitioning • Partition the inputs into equivalence classes • Devise a test case for each equivalence class. • Example: If an object is supposed to accept a negative number, include a test of 1 negative number. Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 17

Black-box Testing (Continued) • Selection of equivalence classes • No rules, only guidelines • Input is valid across range of values. Select 4 test cases from 3 equivalence classes: • Just below the lower boundary • Just above the lower boundary • Just below the upper boundary • Just above the upper boundary • Input is valid if it is from a discrete set. Select test cases from 2 equivalence classes: • Valid discrete value • Invalid discrete value Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 18

White-box Testing • Thoroughness (Coverage) • Every statement in the component is executed at least once. • 4 types of white-box testing • Statement Testing • Loop Testing • Path Testing • Branch Testing Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 19

White-box Testing … • Statement Testing • Test single statements • Loop Testing • Execution of the loop is skipped. • Exception: Do-While loops • Loop is executed exactly once • Loop is executed more than once • Path testing (implies conditional testing) • Make sure all paths in the program are executed Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 20

White-box Testing … public int sum. Positives( int[] a ) t 1 = sum. Positives( null ); { t 2 = sum. Positives( { 0 } ); int total = 0; for ( int i = 0; i < a. length; i++ ) if ( a[i] > 0 ) t 5 = sum. Positives( { 1, 2, 3 } ); total += a[i]; return total; t 3 = sum. Positives( { 1 } ); t 6 = sum. Positives( { 2, 0, 0 } ); // ? } Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 21

Comparison of White & Black-box Testing • Black-box tests are devised when: • Functional requirement is defined • Class is defined • White-box tests are devised when class is implemented. • Create a test harness: A program that executes the tests. • First examination of test results is by eye. • Test Oracle: Save output of tests in a file. • Automate regression testing. Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 22

Bottom-up Testing Strategy • Subsystem classes in the lowest layer of the call hierarchy are tested individually • The next subsystems are tested that call the previously tested subsystems • Do until all subsystems are included in the testing • Test driver A routine that passes a test case to a subsystem. Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 23

Bottom-up Integration A Test E C B Test B, E, F E Layer I D G F Layer III Test F Test C Test A, B, C, D, E, F, G Test D, G Test G Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 24

Bottom-Up Integration Testing • Useful for integrating: • Object-oriented systems • Systems with strict performance requirements • Real-time systems Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 25

Top-down Testing Strategy • Test the top layer subsystem first • Then test that layer + all subsystems called by that layer. • Do until all subsystems are incorporated into the test • A test stub is needed A program or a method that simulates a missing method or subsystem by returning fake data. Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 26

Top-down Integration Testing A C B E Test A Layer I Bernd Bruegge & Allen Dutoit Layer I F D G Layer III Test A, B, C, D, E, F, G Layer I + II All Layers Object-Oriented Software Engineering: Conquering Complex and Changing Systems 27

Sandwich Testing Strategy • Combines top-down strategy with bottom-up strategy • The system is viewed as having 3 layers • A target layer in the middle • A layer above the target • A layer below the target • The target layer is tested. Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 28

1. Unit test all the classes in the selected component. 2. Do functional testing: Define test cases that exercise all uses cases with the selected component 3. Execute performance tests Repeat steps 1 to 3 until the full system is tested. Keep records of the test cases. Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 29

System Testing • • • Functional Testing Structure Testing Performance Testing Acceptance Testing Installation Testing Impact of requirements on system testing: • The more explicit the requirements, the easier they are to test. • Quality of use cases determines the ease of functional testing • Quality of subsystem decomposition determines the ease of structure testing • Quality of nonfunctional requirements & constraints determines the ease of performance tests Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 30

Structure Testing • Essentially the same as white box testing. • Cover all paths in the system design • Exercise all input / output parameters of each component. • Exercise all components & methods • Use conditional & iteration testing as in unit testing. Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 31

Functional Testing. Essentially the same as black box testing • Test functionality of system • Test cases are designed from the RAD, center around requirements & use cases. • The system is treated as black box. Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 32

Performance Testing • Timing testing • Stress Testing • Stress limits of system (maximum # of users, data, peak demands, time to perform a function • Environmental test extended operation) • Test tolerances for heat, • Configuration testing • Test the various software and hardware configurations humidity, motion, portability • Recovery testing • Tests system’s response to • Security testing presence of errors or loss of • Try to violate security requirements Bernd Bruegge & Allen Dutoit • Evaluate response times and data. Object-Oriented Software Engineering: Conquering Complex and Changing Systems 33

Testing has its own Life Cycle (? ) Establish the test objectives Design the test cases Write the test cases Test the test cases Execute the tests Evaluate the test results Change the system Do regression testing Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 34

Professional Tester Test Team Programmer too familiar with code Analyst User Test Team System Designer Configuration Management Specialist Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 35

Summary • Testing is still a black art • Testing consists of: • component-testing (unit testing, integration testing) • system testing • Testing has its own lifecycle (? ) Bernd Bruegge & Allen Dutoit Object-Oriented Software Engineering: Conquering Complex and Changing Systems 36