TESTING PROGRAMMING STUDIO SPRING 2019 ROBERT LIGHTFOOT Reference

TESTING PROGRAMMING STUDIO, SPRING 2019 ROBERT LIGHTFOOT Reference: Code Complete 2 nd Edition Ch. 22, Ch. 8, . .

TESTING – WHAT IT IS AND WHAT IT IS NOT • Testing helps find that errors exist, does not fix them • • Systematic attempt to break a program that is working • • • Requires a different mindset by the developers Can you think of all possible cases? Testing alone does not improve quality • • Unlike all other parts of software development, whose goal is to avoid errors Can never prove absence of errors • • Debugging finds their source and fixes them To improve quality, the development has to be made better instead of just running more test cases Testing is less effective than “collaborative construction” (will get to that later) • Different statistics show that Testing alone can only find between 50 -60% errors present

TESTING FOCUS • Broadly, there are 2 types of testing: • • Black-box testing White-box testing • Black-box testing is done without knowing the code • • Done by non-developers, customers etc. Example: Beta testing of Windows outside Microsoft • White-box is performed by the developers • Knowledge of the code is used to form adversarial and tactical test cases • This is a. k. a developer testing, which is the main focus here

TYPES OF DEVELOPER TESTING • Unit testing • • • Testing of a single class, routine, program Code comes from a single programmer Testing in isolation from the system • • Testing of a class, package, program Usually involves code from multiple programmers or teams Still in isolation from the whole system • Component testing • • Integration testing • • Combined test of two or more classes, packages, components, or subsystems Absolutely necessary before finish/release, but can start very early

TYPES OF DEVELOPER TESTING (CONTINUED) • Regression testing • • Repetition of previously tested cases to find new errors introduced The set of test cases keeps growing • System testing • • Executing software in final configuration, including integration with all other systems and hardware Security, performance, resource loss, timing issues

OTHER TYPES OF TESTING • Usually by specialized test personnel • • • User tests Performance tests Configuration tests Usability tests Etc. • We’re interested now in developer tests (and later in some usability tests)

WRITING TEST CASES FIRST • Helps identify errors more quickly • Doesn’t take any more effort than writing tests later • Just requires resequencing work • Requires thinking about requirements and design before writing code • Shows problems with requirements sooner (can’t write code without good requirements) • Foundation of Test Driven Development (we will get to this…)

TESTING AS YOU WRITE CODE • Motivation: Waiting until later means you have to relearn code • • Fixes will be less thorough and more fragile Fixes further from the source are more costly • • • Boundary Testing Pre- and Post-conditions Assertions Defensive Programming Error Returns • Types of Testing in this category: Table taken from “Code Complete”

BOUNDARY TESTING • Most bugs occur at boundaries • If it works at and near boundaries, it likely works elsewhere • Check loop and conditional bounds when written • Check that extreme cases are handled • e. g. Full array, Empty array, One element array • Usually should check value and +/- 1 • Mental test better than none at all • Very effective in many cases if (a < MAX_VAL){ …. } Test cases: a = {MAX_VAL-1, MAX_VAL+1} for(i=0; i<num; i++){ …. } Test cases: num = {-1, 0, 1} vector<int> a, b(1); sort (a); sort (b);

PRECONDITIONS AND POSTCONDITIONS • Verify that routine starts with correct • preconditions and produces correct postconditions Check to make sure preconditions met • • Handle failures cleanly without surprise Example: Check the denominator before dividing, check if the file was actually opened before writing etc. • Verify that post-conditions are met • • No inconsistencies created The array does not get deleted after sorting • “Provable” software relies on this approach • Need to define pre-/post-conditions clearly

ASSERTIONS • Available in most languages • • assert. h (C/C++), Assertion. Error (Java) • Way of checking for pre-/post-conditions • Helps identify where problem occurs • • Before the assertion e. g. usually in calling routine, not callee • Problem: causes abort • So, useful for cleansing errors in the beginning … int a = 1; assert (a >= 1); assert (a < 1); // will run w/o any problem // will abort the program after. . //. . reporting assertion violation

DEFENSIVE PROGRAMMING • Add code to handle the “can’t happen” cases • Program “protects” itself from bad data • Idea is similar to “defensive driving” training • Skeptical about the rest of the world (i. e. , gracefully handle bad input w/o crashing) • Example: Negative ISBN for the books in the IA

ERROR RETURNS • Good API and routine design includes error codes • Need to be checked • Similar in functionality is Exception handling (via try-catch) supported in many languages now • Need to be clear about what exceptions are thrown, though • Should be maintained when code evolves

ERROR RETURN EXAMPLE #define MAX_DATA_LEN 1024 #define ERR_TOO_MUCH_DATA 1 #define ERR_CANNOT_OPEN 2 int writetofile (string filename, string data) { if (data. size () > MAX_DATA_LEN){ return ERR_TOO_MUCH_DATA; } ofstream myfile (filename); if (!myfile. is_open()) { return ERR_CANNOT_OPEN; } myfile << data; myfile. close(); return 0; } int main () { int writestatus = writetofile (somedata); if (writestatus == 0){ cout << "Write operation successful" << endl; return 0; } else if (writestatus == ERR_TOO_MUCH_DATA){ cout << "Cannot write: Data too long" << endl; return 1; } else if (writestatus == ERR_CANNOT_OPEN){ cout << "Cannot write: File cannot be opened" << endl; return 1; } else{ cout << "Unrecognized response from writetofile() function" << endl; return 1; } }

TRY-CATCH EXAMPLE #define MAX_DATA_LEN 1024 #define ERR_TOO_MUCH_DATA 1 #define ERR_CANNOT_OPEN 2 void writetofile (string filename, string data) { if (data. size () > MAX_DATA_LEN){ throw ERR_TOO_MUCH_DATA; } ofstream myfile (filename); if (!myfile. is_open()) { throw ERR_CANNOT_OPEN; } myfile << data; myfile. close(); } int main () { try{ writetofile (somedata); } catch (int writestatus){ if (writestatus == ERR_TOO_MUCH_DATA){ cout << "Cannot write: Data too long" << endl; return 1; } else if (writestatus == ERR_CANNOT_OPEN){ cout << "Cannot write: Cannot open file" << endl; return 1; } else{ cout << "Unrecognized exception caught from writetofile() function" << endl; return 1; } } cout << "Write operation successful" << endl; return 0; }

SYSTEMATIC TESTING SUGGESTIONS • Test incrementally • Test simple parts first • Know what output to expect • Verify conservation properties • Compare independent implementations • Measure test coverage

TEST INCREMENTALLY • Don’t wait until everything is finished before test • Test components, not just system • Test components individually before connecting them

TEST SIMPLE PARTS FIRST • Test most basic, simplest features • Learn to doubt your simplest code – it is not easy • Finds the “easy” bugs (and usually most important) first class Record{ int num; int price; }; cout<<sizeof(Record); class Record{ int num; int price; void f(); }; class Record{ int num; int price; virtual void f(); }; cout<<sizeof(Record); cout<< sizeof(Record);

KNOW WHAT OUTPUT TO EXPECT • Design test cases that you will know the answer to! • Seeing some answer does not mean your program is right • Make hand-checks convenient • • Pick test cases that you can hand-calculate Example: sort an array {1, 3, 2} • Not always easy to do • e. g. compilers, numerical programs, graphics

VERIFY CONSERVATION PROPERTIES • Specific results may not be easily verifiable • Have to write the program to compute the answer to compare to • But, often we have known output properties related to input • e. g. #Start + #Insert - #Delete = #Final • Can verify these properties even without verifying larger result

COMPARE INDEPENDENT IMPLEMENTATIONS • Multiple implementations to compute same data should agree • Useful for testing tricky code, e. g. to increase performance • Write a slow, brute-force routine • Compare the results to the new, “elegant” routine • If two routines communicate (or are inverses), different people writing them helps find errors • Only errors will be from consistent misinterpretation of description

MEASURE TEST COVERAGE • What portion of code base is actually tested? • Important, because actual coverage can be less than what developers usually think • Code-structure-dependent and analytically computable • Specific rules to calculate them • Tend to work well on only small/moderate • code pieces For large software, tools help judge coverage

STRUCTURED BASIS TESTING • Testing every line in a program • Ensure that every statement gets tested • Need to test each part of a logical statement • Easily catch basic and typo-like errors • Fewer cases than other tests • But, also not as thorough • Goal is to minimize total number of test cases • One test case can test several statements

STRUCTURED BASIS TESTING (CONTINUED) • Start with base case where all Boolean conditions are true • • Design test case for that situation Each branch, loop, case statement increases minimum number of test cases by 1 • One more test case per variation, to test the code for that variation if (a) statement_1 else statement_2 statement_3 if (b) statement_4 else statement_5 How many test cases? Case #1: a=T, b=T Case #2: a=F, b=F

STRUCTURED BASIS TESTING (CONTINUED) • Start with base case where all Boolean conditions are true • Design test case for that situation • Each branch, loop, case statement increases minimum number of test cases by 1 • One more test case per variation, to test the code for that variation if (a) statement_1 else if (b) statement_2 else statement_6 statement_3 if (c) statement_4 else statement_5 How many test cases now? Case #1: a=T, b=X, c=T Case #2: a=F, b=T, c=F Case #3: a=F, b=F, c=X “X” means does not matter

LOGIC COVERAGE • A. k. a. Code Coverage • Testing every path through the code • Can grow (nearly) exponentially with number of choices/branches • Only suitable for small to medium size codes • Why? - Dependency between statements if (a) statement_1 else statement_2 statement_3 if (b) statement_4 else statement_5 How many possible paths in this code? Path #1: Statements 1, 3, 4 (a=T, b=T) Path #2: Statements 1, 3, 5 (a=T, b=F) Path #3: Statements 2, 3, 4 (a=F, b=T) Path #4: Statements 2, 3, 5 (a=F, b=F) # of test cases = # of paths (here, 4) How about code with more if conditions? ?

COVERAGE PROBLEM • • Structured basis testing is: • More practical since test cases grow linearly with number of branching • However, does not cover dependency between statements Logic coverage on the other hand: • • Covers all possible dependency issues But very quickly becomes overwhelming • • Due of exponential growth with branching statements We need something in-between: • Answer is Data Flow Testing covered next

DATA FLOW TESTING • • • Examines data rather than control Data in one of three states: • • • Defined – Initialized but not used (int a =1) Used – In computation or as argument (b = a+1) Killed – Undefined in some way, e. g. , • • memory deleted (delete x) File closed (fclose (fp)) Variables related to routines • • Entered – Routine starts just before variable is acted upon Exited – Routine ends immediately after variable is acted upon

DATA FLOW TESTING (CONTINUED) • First, check for any anomalous data sequences • • Defined-defined Defined-exited Defined-killed Entered-used Killed-killed Killed-used Used-defined int a = 1; a = 5; … • Often can indicate a serious problem in code design • After that check, write test cases

DATA FLOW TESTING (CONTINUED) • First, check for any anomalous data sequences • • Defined-defined Defined-exited Defined-killed Entered-used Killed-killed Killed-used Used-defined void func () {. . . int a = 1; return; } • Often can indicate a serious problem in code design • After that check, write test cases

DATA FLOW TESTING (CONTINUED) • First, check for any anomalous data sequences • • Defined-defined Defined-exited Defined-killed Entered-used Killed-killed Killed-used Used-defined int* x = new int [10]; delete x; • Often can indicate a serious problem in code design • After that check, write test cases

DATA FLOW TESTING (CONTINUED) • First, check for any anomalous data sequences • • Defined-defined Defined-exited Defined-killed Entered-used Killed-killed Killed-used Used-defined void func () { delete x; fclose (fp); . . . } • Often can indicate a serious problem in code design • After that check, write test cases

DATA FLOW TESTING (CONTINUED) • First, check for any anomalous data sequences • • Defined-defined Defined-exited Defined-killed Entered-used Killed-killed Killed-used Used-defined void func () { int a = x; . . . } • Often can indicate a serious problem in code design • After that check, write test cases

DATA FLOW TESTING (CONTINUED) • First, check for any anomalous data sequences • • Defined-defined Defined-exited Defined-killed Entered-used Killed-killed Killed-used Used-defined delete x; . . . delete x; • Often can indicate a serious problem in code design • After that check, write test cases

DATA FLOW TESTING (CONTINUED) • First, check for any anomalous data sequences • • Defined-defined Defined-exited Defined-killed Entered-used Killed-killed Killed-used Used-defined delete x; . . . int a = *x + 1 • Often can indicate a serious problem in code design • After that check, write test cases

DATA FLOW TESTING (CONTINUED) • First, check for any anomalous data sequences • • Defined-defined Defined-exited Defined-killed Entered-used Killed-killed Killed-used Used-defined int x; . . . int a = x + 1; . . . x = 100; • Often can indicate a serious problem in code design • After that check, write test cases

DATA FLOW TESTING (CONTINUED) • Write test cases to examine all defined-used paths • Usually requires: • • More cases than structured basis testing Fewer cases than logic coverage

EXAMPLE – LOGIC/CODE COVERAGE if (cond 1) { x = a; } else { x = b; } if (cond 2) { y = x+1; } else { y = x+2; } if (cond 3) { z = c; } else { z = d; } 1. 2. 3. 4. 5. 6. 7. 8. Conditions: T T T Conditions: T T F Conditions: T F T Conditions: T F F Conditions: F T T Conditions: F T F Conditions: F F T Conditions: F F F Tests all possible paths

EXAMPLE - STRUCTURED BASIS TESTING if (cond 1) { x = a; } else { x = b; } if (cond 2) { y = x+1; } else { y = x+2; } if (cond 3) { z = c; } else { z = d; } 1. 2. Conditions: T T T Conditions: F F F Tests all lines of code

EXAMPLE - DATA FLOW TESTING if (cond 1) { x = a; } else { x = b; } if (cond 2) { y = x+1; } else { y = x+2; } if (cond 3) { z = c; } else { z = d; } 1. 2. 3. 4. Conditions: T T T Conditions: T F F Conditions: F T X Conditions: F F X Tests all defined-used paths l Note: cond 3 is independent of first two

TEST CASE DESIGN (IF YOU DON’T KNOW THE CODE) • Boundary analysis still applies • Equivalence partitioning • Don’t create multiple tests to do the same thing • • • Too much/little Wrong kind/size Uninitialized • • • Minimum/maximum normal configuration “Middle of the Road” data Compatibility with old data • Bad data • Good data

TEST AUTOMATION • • Should do lots of tests, and by-hand is not usually appropriate Scripts can automatically run test cases, report on errors in output • • But, we need to be able to analyze output automatically… Can’t always simulate good input (e. g. interactive programs) People cannot be expected to remain sharp over many tests Automation reduces workload on programmer, remains available in the future

REGRESSION TESTING • Goal: Find anything that got broken by “fixing” something else • Save test cases, and correct results • With any modifications, run new code against all old test cases • Add new test cases as appropriate • Part of Test-Driven Development (TDD) (coming up)

TEST SUPPORT TOOLS • Test Scaffold • Framework to provide just enough support and interface to test • Stub Routines and Test Harness • Test Data Generators • System Perturber

STUB ROUTINES • This is the thing (routine/class) under test • Doesn’t provide full functionality, but pretends to do something when called • • • Return control immediately Burn cycles to simulate time spent Print diagnostic messages Return standard answer Get input interactively rather than computed Could be “working” but slow or less accurate

TEST HARNESS • Calls the routine being tested • • • Fixed set of inputs Interactive inputs to test Command line arguments File-based input Predefined input set • Can run multiple iterations

TEST DATA GENERATORS • Can generate far more data than by hand • Can test far wider range of inputs • Can detect major errors/crashes easily • Need to know answer to test correctness • Useful for “inverse” processes – e. g. encrypt/decrypt • Should weight toward realistic cases • The feasible scope is always limited

SYSTEM PERTURBERS • Modify system so as to avoid problems that are difficult to test otherwise • Reinitialize memory to something other than 0 • Find problems not caught because memory is “usually” null • Rearrange memory locations • Find problems where out-of-range queries go to a consistent place in other tests • Memory bounds checking • Memory/system failure simulation

OTHER TESTING TOOLS • Diff tools • Compare output files for differences • Determine which parts of code tested • Log events to files, save state information • Keep track of what’s been found, and rates of errors • Coverage monitors • Data recorder/loggers • Error databases • Symbolic debuggers • Will discuss debugging later, but useful for tests

TEST-DRIVEN DEVELOPMENT • Generally falls under Agile heading • We will come back to what is meant by this • A style of software development, not just a matter of testing your code • Enforces testing as part of the development process

TEST DRIVEN DEVELOPMENT OVERVIEW • Repeat this process: 1. Write a new test 2. Run existing code against all tests; it should generally fail on the new test 3. 4. Change code as needed 5. Refactor the code Run new code against tests; it should pass all tests

TEST WRITING FIRST • Idea is to write tests, where each test adds some degree of functionality • Passing the tests should indicate working code (to a point) • The tests will ensure that future changes don’t cause problems

RUNNING TESTS • Use a test harness/testing framework of some sort to run the tests • A variety of ways to do this, including many existing frameworks that support unit tests • JUnit is the most well-known, but there is similar functionality across a wide range of languages

TEST FRAMEWORK • Specify a test fixture • • • Test suite run against each fixture • • • Basically builds a state that can be tested Set up before tests, removed afterward Set of tests (order should not matter) to verify various aspects of functionality Described as series of assertions Runs all tests automatically • Either passes all, or reports failures • Better frameworks give values that caused failure

REFACTORING • As code is built, added on to, it becomes messier • Need to go back and rewrite/reorganize sections of the code to make it cleaner • Do this on a regular basis, or when things seem like they could use it • Only refactor after all tests are passing • Test suite guarantees refactoring doesn’t hurt.

REFACTORING COMMON OPERATIONS • • Extract Class Extract Interface Extract Method Replace types with subclasses Replace conditional with polymorphic objects Form template Introduce “explaining” variable Replace magic number with symbolic constant

RESOURCES • Test-Driven Development By Example • Kent Beck; Addison Wesley, 2003 • Test-Driven Development • A Practical Guide David Astels; Prentice Hall, 2003 • Software Testing A Craftsman’s Approach (3 rd edition) • • Paul Jorgensen; Auerback, 2008 Many other books on testing, TDD, also