Software Testing Topics n The testing process unit

Software Testing

Topics n The testing process ¨ unit testing ¨ integration and system testing ¨ acceptance testing n n Test case planning and tracking Testing types ¨ black box vs. white box testing ¨ coverage testing ¨ regression testing ¨ stress testing n Test case prioritization CMSC 345, Version 4/12 2

Testing Goal is to discover defects n The only validation technique for nonfunctional requirements n Successful test: causes a program to behave in an anomalous way n Testing shows the presence, not the absence, of defects. n CMSC 345, Version 4/12 3

Testing Process Overview Integration and System Testing Unit Testing Acceptance Testing Subsystem 1 unit unit 2 Subsystem 2 unit 3 unitn unit Final System unit Subsystemm unit CMSC 345, Version 4/12 unit 4

The Testing Process Elaborated n Unit (component) testing Testing of individual program components (e. g. , functions, methods, or classes) ¨ Usually the responsibility of the component’s developer unit ¨ Must be structured ¨ White box testing ¨ n Integration and system testing ¨ ¨ ¨ n Testing of groups of components integrated to create a subsystem or the entire system Sometimes the responsibility of an independent testing team Subsystem unit Tests are based on system specification (black box testing) unit Is iterative (keeps building) May include alpha and beta testing Acceptance testing ¨ ¨ Run in the presence of customer or by customer Used to validate all system requirements CMSC 345, Version 4/12 Final System $$$$! 5

Unit (Component) Testing You may do this on your own, but always … n have a plan n keep records n automate! ¨ JUnit (for Java) or other tools ¨ even #ifdef - #endif pairs (C/C++) ¨ main( ) method (Java) CMSC 345, Version 4/12 6

Integration Testing Approaches – Procedural Programming n Top-down testing Start with high-level components and integrate from top down ¨ Uses stubs ¨ n Bottom-up testing Start with low-level components and integrate from bottom up ¨ Uses drivers ¨ n Sandwich testing ¨ n Combination of top-down and bottom-up Thread testing ¨ Tests one piece of functionality at a time CMSC 345, Version 4/12 7

Top-down Integration & Testing Level Being Tested 0 1 Stub: 1) says “I’ve been called!”, 2) confirms receipt of inputs, if any, and 3) returns hard-coded outputs, if necessary Stub n CMSC 345, Version 4/12 8

Top-down Integration & Testing Level Tested 0 1 Being Tested Stub: 1) says “I’ve been called!”, 2) confirms receipt of inputs, if any, and 3) returns hard-coded outputs, if necessary Being Tested Stub n CMSC 345, Version 4/12 9

Top-down Integration & Testing Level Tested 0 1 Tested n Being Tested CMSC 345, Version 4/12 Tested Being Tested Stub: 1) says “I’ve been called!”, 2) confirms receipt of inputs, if any, and 3) returns hard-coded outputs, if necessary Tested Being Tested 10

Bottom-up Integration & Testing Level Driver: 1) Calls functions, 2) passes in inputs, if any, and 3) confirms receipt of outputs, if any 0 1 Driver n Being Tested CMSC 345, Version 4/12 Driver Being Tested 11

Bottom-up Integration & Testing Level Driver: 1) Calls functions, 2) passes in inputs, if any, and 3) confirms receipt of outputs, if any 0 1 Being Tested n Tested CMSC 345, Version 4/12 Being Tested Being Tested 12

Bottom-up Integration & Testing Level Driver: 1) Calls functions, 2) passes in inputs, if any, and 3) confirms receipt of outputs, if any 0 1 Driver Tested n Tested CMSC 345, Version 4/12 Driver Tested Tested 13

Sandwich Integration & Testing Uses both stubs and drivers Level 0 1 n CMSC 345, Version 4/12 14

Thread Integration & Testing Level May be done top-down, bottom-up, or sandwich 0 1 n CMSC 345, Version 4/12 15

Exercise (on your own!) What are some of the benefits associated with each of the following integration and testing methods? n Top-down n Bottom-up n Sandwich n Thread CMSC 345, Version 4/12 16

Integration Testing Approaches – Object-oriented Programming n Use based Start with classes that have no dependencies ¨ Move “out” to next “level” of classes, and so on ¨ Similar to bottom-up for procedural programming ¨ n Cluster Test clusters of associated classes ¨ Integrate the clusters ¨ May be combined with use based testing (above) ¨ n Scenario Based on functional specifications (possibly expressed as use cases) ¨ Similar to thread testing for procedural programming ¨ CMSC 345, Version 4/12 17

System Testing n Alpha ¨ Internal to development organization, but not the developers themselves n Beta ¨ External n to development organization Acceptance ¨ Performed in the presence of or possibly by the customer ¨ Determines final “acceptance” by the customer CMSC 345, Version 4/12 18

Test Case Planning and Tracking ID Natural language description CMSC 345, Version 4/12 Specific values (if possible) How to Pass, Fail, other determine notes Pass or Fail 19

A Sample Testing Form CMSC 345, Version 4/12 Before Testing 20

A Sample Testing Form CMSC 345, Version 4/12 After Testing 21

A Sample Testing Form CMSC 345, Version 4/12 Before Testing 22

A Sample Testing Form CMSC 345, Version 4/12 After Testing 23

White Box Testing input code code code code code output Sometimes called structural testing n Can “see inside” the code unit n Objectives: n ¨ Exercise unique algorithms ¨ Exercise all program statements at least once n Typically tests small program units such as functions or class methods CMSC 345, Version 4/12 24

Coverage Testing n n n n A technique for white box testing Ensures that each statement is executed at least once Use a program flow graph to derive paths Find minimum paths needed to “cover” all statements (there are tools to do this) Drive the code through all of these paths All branches are executed, but not all combinations of branches. Some paths may be impossible to test. CMSC 345, Version 4/12 25

#include <iostream> using namespace std; void triangle (int, int); triangle. cpp // parameters are sides of the triangle int main() { int s 1, s 2, s 3; char repeat; cout << "This program will accept the sides of a trianglen" << " and tell you the type of triangle. n" << endl; do { cout << "Enter the 3 sides: " ; cin >> s 1 >> s 2 >> s 3; triangle(s 1, s 2, s 3); cout << "Want to do another? (Y or N) "; cin >> repeat; cout << endl; } while (repeat =='Y' || repeat == 'y'); cout <<"Program terminated" << endl; cout << "Enter any character to continue" << endl; cin >> repeat; return 0; } void triangle (int side 1, int side 2, int side 3) { // Error checking /******************/ // Check sides for validity if (side 1 <= 0 || side 2 <= 0) cout << "Illegal value for side" << endl; // Check Triangle inequality if (side 1+side 2 <= side 3 || side 2+side 3 <= side 1) cout << "Triangle inequality violation" << endl; /******************/ Pick Version triangle type CMSC //345, 4/12 if (side 1 == side 2 && side 2 == side 3) // 1 // 2 // 3 // 4 // 5 26

triangle. exe Control Graph 1 a 1 b 2 2 3 a 3 Baseline path 3 b 4 1 4 5 5 a 5 b 6 7 b 9 a 11 7 9 7 c 9 b 7 a 11 Vertices= 18 Edges = 28 Regions = 12 6 8 8 10 10 12 12 V(g) = 28 - 18 + 2 = 11 + 1 = 12 CMSC Northrup 345, Version 4/122009 D. Cheslock, Grumman, (e-n+2) (p+1) 27

A Set Of Basis Paths For triangle. cpp 1) 1 a - 1 b - 3 a - 3 b - 5 a - 7 b -7 c -9 a - 9 b - 11 - 12 sample data: 3, 5, 7 (baseline path) 2) 1 a - 2 - 3 a - 3 b - 5 a - 7 b -7 c -9 a - 9 b - 11 - 12 sample data: -3, 7, 3 3) 1 a - 1 b - 2 - 3 a - 3 b - 5 a - 7 b -7 c - 9 a - 9 b - 11 - 12 sample data: can’t perform this test 4) 1 a - 1 b - 3 a - 4 - 5 a - 7 b -7 c - 9 a - 9 b - 11 - 12 sample data: 2, 3, 7 5) 1 a - 1 b - 3 a - 3 b - 4 - 5 a - 7 b -7 c - 9 a - 9 b - 11 - 12 sample data: 7, 3, 2 6) 1 a - 1 b - 3 a - 3 b - 5 a - 5 b - 6 - 12 sample data: 5, 5, 5 7) 1 a - 1 b - 3 a - 3 b - 5 a - 5 b - 7 a - 7 b -7 c - 9 a - 9 b - 11 - 12 sample data: can’t perform this test 8) 1 a - 1 b - 3 a - 3 b - 5 a - 5 b - 7 a - 8 - 12 sample data: 4, 4, 2 9) 1 a - 1 b - 3 a - 3 b - 5 a - 7 b - 8 - 12 sample data: 2, 4, 4 10) 1 a - 1 b - 3 a - 3 b - 5 a - 7 b -7 c - 8 - 12 sample data: 4, 2, 4 11) 1 a - 1 b - 3 a - 3 b - 5 a - 7 b -7 c - 9 a - 10 - 12 sample data: 3, 4, 5 12) 1 a - 1 b - 3 a - 3 b - 5 a - 7 b -7 c - 9 a - 9 b - 10 - 12 CMSC sample 345, Version 4/12 data: 3, 5, 4 282009 D. Cheslock, Northrup Grumman,

Metrics – Cyclomatic Complexity The minimum number of tests needed to execute all statements CC = number_edges – number_nodes + 2 OR CC = number_decisions + 1 OR the number of regions CMSC 345, Version 4/12 29

Black Box Testing input output Cannot “see inside” code unit n Test cases are: n ¨ based on the system specification and/or interfaces (preconditions, postconditions, invariants, and parameters) n Use equivalence partitions when conducting black box testing CMSC 345, Version 4/12 30

Equivalence Partitions and Boundary Testing Equivalence Partition n A group of input values that will produce equivalent behavior n Example ¨ Square root function the set of all numbers >= zero n the set of all negative numbers n n Test cases should be chosen from each partition, especially at the boundaries. CMSC 345, Version 4/12 31

Equivalence Partitions and Boundary Testing Exercise n A system accepts 4 to 10 inputs that are 5 digit integers >= 10, 000 n Partition the system inputs into groups (partitions) that should cause equivalent behavior. Include both valid and invalid inputs. Ian Somerville, Software Engineering, 6 th edition CMSC 345, Version 4/12 32

The six partitions with corresponding boundary values Ian Somerville, Software Engineering, 6 th edition CMSC 345, Version 4/12 33

Regression Testing Code A Code C Code B Code D Code A Test 1 Code C Test 1 … and so on Test 2 Test 1 Test 3 Test 2 Test 4 Test 3 Test 5 Test 4 Test 6 Test 5 Test 7 Test 8 CMSC 345, Version 4/12 34

Stress Testing n n Example: A child’s car seat Exercises the system beyond its maximum design load. Examples: ¨ Exceed string lengths ¨ Store/manipulate more data than in specification ¨ Load system with more users than in specification n n Often causes defects to come to light Systems should not fail catastrophically. ¨ Stress testing checks for unacceptable loss of service or data. CMSC 345, Version 4/12 35

Test Case Prioritization Why? n To locate defects quickly to: ¨ ¨ n n begin debugging ASAP debug and deliver in increments ASAP To locate the majority of errors quickly Other How? n Can prioritize according to: ¨ ¨ ¨ module or functional dependencies module size module complexity module suspected to be the most error-prone most “important” module or functionality other CMSC 345, Version 4/12 36

References Somerville, I. , Software Engineering, 6 th ed n Source. Forge. net, JUnit, http: //junit. sourceforge. net, accessed 11/6/07 n JUnit. org, http: //www. junit. org, accessed 11/6/07 n CMSC 345, Version 4/12 37