Objectives and Limits of Testing Chapter 2 ITIS

Objectives and Limits of Testing Chapter 2 ITIS 3320

Overview l l Can’t completely test Why test – – – Good vs. bad testing How much testing is enough? When are we done testing?

Complete Testing l Cannot test a program completely – All inputs l – All outputs l – All possible outputs All internals l – All possible values and combinations All paths in all possible combinations Other l l All possible timing Effects of other processes on computer

All Inputs - Manual l Simple example: – Adding 2 8 bit numbers (0 -255) l – 65, 536 combinations Assuming 10 seconds per manual test l l 655, 360 seconds 1092 minutes 18 hours Over two 8 hour days!

All Inputs - Automated l Multiplying two 32 bit numbers – – 4, 294, 967, 96 numbers to combine 18, 446, 744, 073, 709, 551, 616 combinations! l – – Assuming can do 1, 000 tests/sec 18, 446, 744, 073, 709 seconds l l l – l l 307, 445, 734, 561 minutes 5, 124, 095, 576 hours 213, 503, 982 days 584, 942 years! Even if could increase test rate to 1 billion tests/sec – l 18 quintillion! Over 584 years to complete the exhaustive test Don’t even want to think about two 64 bit numbers! Number names – http: //www. uni-bonn. de/~manfear/numbers_names. php

Testing Inputs l Valid Inputs – l Invalid Inputs – l All possible invalid inputs Edited Inputs – l All possible good inputs Every possible edited input Variation on Input timing – Does going too fast or too slow change output

Inputs l l Can’t test all possible inputs What to do? – Test intelligently selected samples of the possible error cases l l Valid data Invalid data Edited data Timing varied

Paths l The sequence of statements executed when the programs runs

Paths l 1 Book example (pg. 20) 2 3 4 5 6 Fails Iterate 30 times before a 5 1 path is taken

Paths l Myers’ program – – – l l 100 line program 1018 unique paths Universe is ~ 4 x 1017 seconds old It is a “cooked” example Still shows how complex a program of 20, 000 -lines, 400, 000 -lines or even bigger could be – – – Windows XP: 40 million lines of code! Linux kernel: 6 million lines of code Debian 3. 1 Linux distro: 213 million lines of code!

Design Errors l l Program may do what it was designed to do But, what if the design is wrong? – Need to test for design errors

Proof l Can’t prove a program is correct using logic – The old “time” problem again l – Assume you can prove the program is in “spec” l l Not enough time to check all possible paths and variations Is the spec right? Is the proof procedure correct? Was the proof procedure done correctly? This is just a tip of the iceberg!

Program Verification? l l Is testing a verification process? NO! – – Can’t verify a program works correctly Program DO have errors Testing is not a failure if the program does not work correctly Testers should not want to prove the program works correctly

Program Verification l Can’t verify a program works correctly – – Earlier showed can’t test fully Ergo can’t verify it works correctly

Program Verification l l Programs DO have errors How many? – – l When released to testing Estimate 1 -3 bugs per 100 lines of code Public vs. private bugs – Public bugs: ones remaining after declared “error-free” l – Estimate 1 error per 100 statements Private bugs: one the programmer makes and corrects while designing and coding the program l l l Estimate 1. 5 errors per executable statement Most programmers catch most of their mistakes before releasing code Note: that is most of their mistakes, not all!

Program Verification l Testing is not a failure if the program under test does not work correctly – – Finding errors in a program is a GOOD thing Beware of managers who think finding errors creates problems

Program Verification l l Testers should not want to prove the program works correctly Question of attitude: I want this program to be correct vs. – I know there are errors, how can I find them? –

Very Apt Quote: l If you want and expect a program to work, you will be more likely to see a working program – you will miss failures. If you expect it to fail, you’ll be more likely to see the problems. If you are punished for reporting failures, you will miss failures. You won’t only fail to report them – you will notice them.

So, Why Test?

Testing Purpose l Find errors – l Not to prove the program works – l Uncover as many errors as possible Cannot verify all functions Watch the Attitude! – – If looking for bugs testers will look harder If trying to prove correct testers will not seek bugs

Find Problems l l A successful test uncovers an error “A test that reveals a problem is a success. A test that did not reveal a problem was a waste of time” – Note: that means reveal an existing problem

Summary l l l Cannot test a program completely Do not want to try to do program verification Attitude! – – There are errors We can find them