Testing Your Program Input Space Partitioning CS 1110

Testing Your Program: Input Space Partitioning CS 1110 Introduction to Programming Spring 2017

Software is Everywhere CS 1110 2

Software Failures • (1997) Ariane 5 explosion: Exception-handling bug forced self- destruct on maiden flight (64 -bit to 16 -bit conversion), causing $370 millions • (1999) NASA’s Mars lander: Crashed due to a unit integration fault • (2003) Northeast blackout: The alarm system in the energy management system failed due to a software error and operators were not informed of the power overload in the system – affected 40 million people in 8 US states, 10 million people in Ontario, Canada • (2012) Knight Capital’s trading software: Faults in a new trading algorithm causes $440 millions • (2014) Dropbox’s outage was due to a fault in a maintenance script • (2016) Information lost after clicking the back button while using Turbo. Tax web software CS 1110 3

Terminologies • Software failures: external, incorrect behaviors with respect to the requirements or other descriptions of the expected behavior • Example: A patient gives a doctor a list of symptoms (failures) • Three kinds of failures we’ve seen • Syntax failure • Runtime failure • Logical failure • Software faults: static defects in the software • Example: The doctor tries to diagnose the root cause (fault) • Faults in software ≈ design mistakes in hardware. CS 1110 4

An Example in Python # Return index of the first occurrence of a letter in string, # Otherwise, return -1 def get_index_of(string, letter): Fault: Should start at 0, not 1 index = -1 for i in range(1, len(string)): if string[i] == letter: index = i return index Test 1 Error: i is 1, not 0, on the first iteration Failure: none # Test 1: inputs “python”, “z” print(get_index_of("python", "z")) # expected: -1, actual: -1 # Test 2: inputs “python”, “z” print(get_index_of("python", "p")) # expected: 0, actual: -1 Test 2 Error: i is 1, not 0, on the first iteration -- Error propagates to the variable index Failure: index is -1 at the return statement For simplicity, this example assumes a function accept a letter of size 1 CS 1110 5

Overview: Testing • What is software testing? • Why do we test software? • When should we test software? • Who should test software? • How do we test software? • Input space partitioning • When do we stop testing? Good enough? CS 1110 6

Here! Test This! My first “testing” assignment Micro. Steff big Big software–program software system for the mac V. 1. 5. 1 Jan/2007 Jan/2011 Verdatim Data. Life MF 2 -HD 1. 44 MB A stack of computer printouts – and no documentation CS 1110 7

What is Software Testing? • Testing = process of finding input values to check against a software (today’s discussion) Test case consists of test values and expected results • Debugging = process of finding a fault given a failure (out of today’s scope) Test values (inputs) Program Actual results vs Expected results 1. Testing is fundamentally about choosing finite sets of values from the input domain of the software being tested 2. Give the test inputs, compare the actual results with the expected results CS 1110 8

Why do We Test Software? • Goal of testing • Not to prove correctness, but to increase our confidence in • • correctness Improve quality Reduce overall software development cost Preserve customer satisfaction Get good grades in this course (and any other courses) • What fact does each test try to verify? • Benefits • You are not biased that your code works • You will better understand what you need to build • You will get insights on how to built it CS 1110 9

When should We Test Software? • Testing is the most time consuming and expensive part of software development. Not testing is even more expensive. • If we have too little testing effort early, the cost of testing increases • Planning for testing after development is prohibitively expensive $2 60 50 Assume $1000 unit cost, per fault, 100 faults K 50 • Cost of late testing $3 60 40 K 30 20 0 ts re ui m en D 00 Fault detection (%) K Unit cost (X) t ni t U / s m te n ig es $1 0 K $1 3 K $6 K 10 $2 Fault origin (%) a r og r P t In n io st t ra Te g e t em t ys s Te st t o P en m oy q pl S e Re D Software Engineering Institute; Carnegie Mellon University; Handbook CMU/SEI-96 -HB-002 CS 1110 10

Who should Test Software? • CS 1110 students • Software designers • Software developers/programmers • Software testers • Project managers • End users CS 1110 11

How do We Test Software ? • Acceptance testing (equivalence classes) • Boundary testing (corner cases) • Exception testing CS 1110 12

How do We Test Software ? Input Space Partitioning (ISP) – Choose Inputs 1. Testing is fundamentally about choosing finite sets of values from the input domain of the software being tested • Input space partitioning describes the input domain of the software • Input domain contains all the possible values that the input parameters can have • Input parameters can be • Parameters to a function • Data read from a file • Global variables • User level inputs CS 1110 13

ISP: Choosing Test Inputs 1. Identify inputs 2. Find their characteristics • Each characteristic allows the testers to define a partition. 3. Partition each characteristic and identify values • The partition must cover the entire domain; not overlap • Two approaches • Interface-based (simpler) • Develop characteristics from individual input parameters • Can be partially automated in some situations • Functionality-based (harder) • Develop characteristics from a behavior view • May result in better tests, or fewer tests that are as effective 4. Choose tests by combining values of inputs CS 1110 14

How do We Test Software ? Input Space Partitioning (ISP) – Compare Results 2. Give the test inputs, compare the actual results with the expected results • If the actual results == expected result, the program passes the test • Otherwise, the program fails the test • The test inputs reflect the characteristics of the input parameters • The characteristics of the inputs signifies the kinds of faults/bugs in program • The kinds of faults/bugs tell what to fix CS 1110 15

ISP: Example 1 (Interface-based) Choose Test Inputs # Return index of the first occurrence of a letter in string, # Otherwise, return -1 def get_index_of(string, letter): 1. Identify inputs • Two parameters: string , letter 2. Find their characteristics • C 1 = string is empty • C 2 = letter is empty CS 1110 16

ISP: Example 1 (Interface-based) Choose Test Inputs 3. Partition each characteristic and identify values Characteristic P 1 P 2 C 1 = string is empty True possible values: "" False possible values: "python" C 2 = letter is empty True possible values: "" 3. Choose tests by combining values False possible values: "p" or "z" of test inputs • (assume choosing letter = "p") Test CS 1110 string letter T 1 (C 1=True, C 2=True) "" "" T 2 (C 1=True, C 2=False) "" "p" T 3 (C 1=False, C 2=True) "python" "" T 4 (C 1=False, C 2=False) "python" "p" 17

ISP: Example 1 (Interface-based) Compare Results Specify the expected result of the given test inputs Test string letter Expected result T 1 (C 1=True, C 2=True) "" "" -1 T 2 (C 1=True, C 2=False) "" "p" -1 T 3 (C 1=False, C 2=True) "python" "" -1 "p" 0 T 4 (C 1=False, "python" Put a test case T 4 in Python C 2=False) print(get_index_of("python", "p")) # expected: 0 If actual result == expected result, the program passes the test Otherwise, the program fails the test We don’t even have to look at the code to come up with the test cases CS 1110 18

ISP: Example 1 (Functionality-based) Choose Test Inputs # Return index of the first occurrence of letter in string, # Otherwise, return -1 def get_index_of(string, letter): 1. Identify inputs • Two parameters: string , letter 2. Find their characteristics • C 1 = number of occurrence of letter in string • C 2 = letter occurs first in string • C 3 = letter occurs last in string No need to look at the code to come up with the test cases. The specification is sufficient. CS 1110 19

ISP: Example 1 (Functionality-based) Choose Test Inputs 3. Partition each characteristic and identify values Characteristic P 1 P 2 P 3 C 1 = number of occurrence of letter in string 0 1 >1 C 2 = letter occurs first in string True False C 3 = letter occurs last in string True False Possible values C P 1 P 2 P 3 C 1 "intro to python", "z" or "intro to python", "" or "", "z" "intro to python", "r" "intro to python", "o" C 2 "intro to python", "i" "intro to python", "o" C 3 "intro to python", "n" "intro to python", "o" CS 1110 20

ISP: Example 1 (Functionality-based) Choose Test Inputs 4. Choose tests by combining values of test inputs • Choose a characteristic, then choose test values for the characteristic C P 1 P 2 P 3 C 1 T 1("intro to python", "z") T 2("intro to python", "") T 3("", "z") T 4("intro to python", "r") T 5("intro to python", "o") C 2 T 6("intro to python", "i") T 5("intro to python", "o") C 3 T 7("intro to python", "n") T 5("intro to python", "o") CS 1110 21

ISP: Example 1 (Interface-based) Compare Results Specify the expected result of the given test inputs Test string letter Expected result T 1 "intro to python" "z" -1 T 2 "intro to python" "" -1 T 3 "" "z" -1 T 4 "intro to python" "r" 3 T 5 "intro to python" "o" 4 T 6 "intro to python" "i" 0 T 7 "intro to python" "n" 14 print(get_index_of("intro to python", "z")) # expected: -1 … print(get_index_of("intro to python", "i")) # expected: 0 print(get_index_of("intro to python", "n")) # expected: 14 CS 1110 22

When do We Stop Testing? • Time constraint • (Assignments) due date • Coverage • Test cases cover all characteristics and partitions • Characteristics and partitions determine the quality of test cases • The quality of test cases determine how thorough the program is tested CS 1110 23

Note and Tip I didn’t fail the test. I just found 100 ways to do it wrong. – Benjamin Franklin Did you know? There is an actual tactic called “rubber duck debugging” where programmers verbally explain a broken code to a rubber duck in hopes of finding the solution by describing the problem CS 1110 24