Software Testing Using Model Program DESIGN BY HONG

Software Testing Using Model Program DESIGN BY HONG NGUYEN & SHAH RAZA Dec 05, 2005

Overview Software Testing Using Model Program n Briefly introduction of M-mp testing n n The problem of software testing n n What is M-mp testing? How it work? Why we need program base testing Oracle test N-Version programming n n n What is N-Version Programming? Disadvantages Advantages

Overview (continues) n M-mp testing n n Framework for M-mp testing Asymptotic behavior of M-mp testing Input domain partitioning An M-mp Experiment Background and motivation n Selecting the application n Phases of the experiment n n Conclusion

Introduction n What is M-mp testing? n Testing using M Model program Abbreviated to ‘M-mp testing’ n It is alternative to software testing base on manual outcome prediction M-mp testing is base on N-Version programming technique n n n Which is apply specially in the well-known software fault tolerant technique.

Introduction (Continues) n How it work n n Model program implements selected part of functional specification of the software to be tested. The M-mp testing strategy requires that M (M>= 1) n n Model programs as well as the program under test, should be independently developed P and M model program are then subjected to the same test data. Difference analysis is conducted on the outputs and appropriate corrective action taken P and the M model program jointly constituted an approximate test oracle (out come correct or not)

Introduction (Continues) n Why we need program base testing For small number of test data, it may be feasible to work out the expected output manually n For thorough testing which required large test datasets manual testing could be time consuming and error prone n n That is why M-mp base testing may be more feasible for those kinds of data set

The problem of software testing n n n The difficult task in software testing is to assess the correctness to the outcome of a program that is subjected to particular test input In testing theory, the mechanism to adjudicate on whether or not an output is correct, somewhat neglected this problem is referred to as “the oracle problem. Testing theory concern with the choice of test and testing methods, usually ignore the oracle problems.

The problem of software testing (continues) n Oracle Test n It means of determining whether a program passed or failed a test.

The problem of software testing (continues) n Oracle test (continues) n n n P is the program to be tested Test strategy determines the set of test data to be used To get the test result we compare P’s output with the oracle ‘s output n n n For small out put oracle manually base oracle technique is feasible But for large industry base application manually base oracle testing is not feasible One of alternative of manually base oracle is N- Version programming

N-Version Programming n n N-Version is comprises N independently written version of the software All has the same functional specification At runtime voting base on majority agreement is used to decided the probable output Disadvantage: n n Concern in N-Version systems is the fact that correlated failures may limit the practical gain in reliability N-Version systems should not be casually assumed to be completed reliable.

N-Version (continues) n Disadvantage (continues) n n Is the increased development cost because at least three version being required for a voting system to work Advantage n n N-Version design appear to offer more reliability that we can gain from any other way Even though the cost of failure is high: n It would be more cost effective to build N-Version systems rather than focusing on building ‘one good version’

M-mp Testing n Framework for M-mp testing

M-mp Testing (continues) n Framework (continues) n n n In M-mp testing P is primary program (the program under test) mp 1 –mp. M are M so called ‘model program of P M-mp testing treat P and mp 1 -mp. M to be same test input Any disagreement on the output indicates the presence of specification defects Or software faults in at least one of program Once defects are detected and removed. The program are re-run

M-mp Testing (continues) n Framework (continues) n n The cycle is repeat until all disagreements for a particular dataset are resolved. Asymptotic behavior of M-mp testing n Difference between the M-mp and manual approaches to testing is the cost of outcome verification

M-mp Testing (continues) n n Asymptotic behavior of M-mp testing (continues) n On the graph: the cost of M-mp outcome verification is high initially but it increase only slightly. n The cost growth is modeled as linear n In manual approach, the growth of the output verification cost may be expected to be much steeper. Input domain partitioning n Divided into a standard domain and an exception domain n n Exception domain may be split into an incomplete data domain and an invalid data domain. The standard domain consists of inputs that a program can process ‘as is’

M-mp Testing (continues) n Input domain partitioning (continues) n n The invalid domain contains the input that a program can not process it should be rejected with appropriate error message The incomplete data domain is ‘between’ the standard and invalid data domains n n It is made up of inputs that the program has to completed with default values before processing them An M-mp Experiment n Background and motivation

M-mp Testing (continues) n Background and motivation (continues) The primary program is a scheduling application, a noninteractive data processing subsystem of a large logistics management system n The program can be characterized as algorithmically complex n In practice, the testing process in the company was well defined and supported by standards and procedures but had several drawback n Designing test cases based on manual outcome prediction was not only difficult, error prone and time consuming n n But consequently also unattractive and demoralizing

M-mp Testing (continues) n Selecting the application n n n The scheduling application was chosen as the primary program for following reasons Developer was not familiar with the scheduling domain and application The cost of developing from scratch could be assesses more authentically The risk of correlated failures caused by common design error was reduces scheduling application is representative of the family of operation management program The time the experiment started, the scheduling subsystem had been in use for more than year n This seem a good opportunity to test the fault detection ability of M-mp testing, as the likelihood of finding defect was expected to be low

M-mp Testing (continues) n Phase of the experiment

M-mp Testing (continues) n Phase of the experiment n n The experiment entailed a test design, a test execution and test evaluation phase as depicted Test design phase comprised test data selection n Which includes activities related to test data generation, Model program design and model program implantation The test execution phase includes activities relating to setting up the test environment and executing the program The test evaluation phase includes the disagreement analysis procedures whereby disagreements between the program need to be arbitrated.

Conclusion n M-mp approach could test a scheduling program more adequately than manually designed test and at lower cost M-mp testing should be judged on a case by case basic M-mp approach will tend to be better option than the manual one when the number of tests required to achieve a particular adequacy level become large

Question n n Finished Please be easy on us. Thank you