Software Testing and Reliability Testing Graphical User Interfaces

Software Testing and Reliability Testing Graphical User Interfaces Aditya P. Mathur Purdue University May 19 -23, 2003 @ Guidant Corporation Minneapolis/St Paul, MN Graduate Assistants: Ramkumar Natarajan Baskar Sridharan Last update: April 17, 2003 Software Reliability Copyright Aditya P. Mathur

References 1. Regression Testing of GUI Event-Interactions, Lee White, Proc. of the International Conference on Software Maintenance, Washington DC, pp. 350 -358, November 1996. 2. Generating test cases for GUI responsibilities using complete interaction sequences, Lee White and Husain Almezen, International Symposium on Software Reliability Engineering, San Jose, CA, pp. 110 -121, October 2000. 3. User-based testing of GUI sequences and their interactions, Lee White, Husain Almezen, and Nasser Alzeidi Material from 1. and 2. is used extensively in this presentation. Software Reliability © Aditya P. Mathur 2003 2

References [contd. ] 4. Hierarchical GUI test case generation using automated planning. Atif Memon, Martha E. Pollack, Mary Lou Soffa, IEEE Transactions on Software Engineering, V 27, N 2, February 2001, pp 144 -155. 5. Coverage criteria for GUI testing, Atif Memon, Mary Lou Soffa, and Martha E. Pollack, , 8 th European Conference and 9 th ACM SIGSOFT Foundation of Software Engineering (FSE-9), Austria, September 10 -14, 2001. 6. Orthogonal Latin Squares, Alexander Bogomolny, http: //www. cut-the-knot. com/arithmetic/latin 3. shtml Software Reliability © Aditya P. Mathur 2003 3

References [contd. ] 7. Model Checking Graphical User Interfaces Using Abstractions, Matthew B. Dwyer, Vicki Carr, Laura Hines, Proceedings of the Sixth European Software Engineering Conference (ESEC/FSE 97), September 1997. 8. The Black Art of GUI Testing, Lawrence Kepple, Dr. Dobb’s Journal, pp. 40 -46, Feb. 1994. Software Reliability © Aditya P. Mathur 2003 4

GUI Testing n Learning objectives 1. GUI test tools: strengths and weaknesses 2. Why is GUI testing difficult? 2. Modeling and test generation for GUI testing 3. Automated test oracles for GUI testing 4. Coverage Criteria for GUI testing Software Reliability © Aditya P. Mathur 2003 5

GUI Testing: Questions n What types of GUI test tools are available? n What makes GUI testing difficult? n n Are there formal techniques for testing GUIs? If so, then what are they and how well do they function? What are the advantages and drawbacks of capture-replay techniques and ho can one do better? Software Reliability © Aditya P. Mathur 2003 6

Features of GUI Test Tools n Record and playback of physical events n Screen image capture and comparison n Shell scripts to control and execute test runs of the GUI We focus on how to test interactions amongst GUI objects. Software Reliability © Aditya P. Mathur 2003 7

GUI Testing: Techniques n n Pair-wise testing of GUI interactions. [Due to Lee White. ] Testing GUIs using Complete Interaction Sequences corresponding to responsibilities. [Due to Lee White, and Husain Almezen. ] Software Reliability © Aditya P. Mathur 2003 8

GUI Interaction Testing for interactions amongst all GUI objects. Problem: How to automatically and efficiently devise tests for testing pair wise interactions amongst GUI objects? n GUI interactions: Static, Dynamic, and a mix of the two. Software Reliability © Aditya P. Mathur 2003 9

Sample GUI Options 6 options 5 options X Y 7 options Z 5 Modes Total number of possible combination of selections: 5 x 6 x 5 x 7=1050 Software Reliability © Aditya P. Mathur 2003 10

Testing Static Interactions n n n Testing is considered static in this example because only a single screen is involved. In the previous example, there is a total of 1050 pair wise interactions. There are too many higher order interactions. At present we focus only on testing second order, or pair wise, interactions. Can we reduce the number of pair wise tests and if so then how ? Software Reliability © Aditya P. Mathur 2003 11

Brute Force versus Efficient Test Generation n Suppose that a GUI has three factors denoted by X, Y, and Z. Let X, Y, and Z, have, respectively, 2, 3, and 2 possible selections. Using a brute force scheme, a total of 12 tests are needed to cover all pair wise selections. n Sample tests: <X 1, Y 1, Z 1>, <X 2, Y 1, Z 2>, <X 1, Y 3, Z 2> n Reduced test set consisting of only 7 tests: : <X 1, Y 1, Z 1>, <X 1, Y 2, Z 2>, <X 2, Y 3, Z 2> <X 2, Y 1, Z 1>, <X 1, Y 3, Z 1>, <X 1, Y 1, Z 2> <X 1, Y 2, Z 1> Software Reliability © Aditya P. Mathur 2003 12

Controlling the number of possible interactions n n n In both static and dynamic cases, the number of possible interactions could be very large. In the dynamic case a selection might bring up a new screen where additional selections are made. One method to tackle the combinatorial growth was proposed by Dalal [1994]. Software Reliability © Aditya P. Mathur 2003 13

Problem abstraction n Consider each GUI object from which selections are made as a factor and denote it by F’i , for the i th factor. In our GUI example, we have four factors F’ 1, F’ 2, F’ 3, and F’ 4, corresponding to Mode, X, Y, and Z. Order the factors by their cardinality such that |F 1|>=|F 2|…. >=|Fk| n In the GUI shown in an earlier example we get: |F 1|=7, |F 2|=6, |F 3|=5, |F 4|=5. Software Reliability © Aditya P. Mathur 2003 14

Problem Statement The problem now is to select a set of tests each set consisting of one selection from each factor. Question: What is the lower bound on the number of tests? Software Reliability © Aditya P. Mathur 2003 15

Testing Dynamic Interactions n n n This involves making a selection from one or more menus, buttons, or any other GUI object on one screen. Such a selection brings up a new screen where additional selections are made. Assumption: Regardless of the selection made, the new screen is always dependent on the current screen and not on the entire past history. Software Reliability © Aditya P. Mathur 2003 16

Hierarchy of Screens Prune S 1 Software Reliability © Aditya P. Mathur 2003 Prune S 1 S 2 S 3 S 5 Prune 17

Pruning Paths n n Starting with the initial screen S 1, generate a hierarchy where each path of screen transitions ends at a pruned screen. Select all paths from S 1 to pruned screens but do not include the pruned screens. With each selected path, associate only those GUI events which lead to the paths being selected. Map these factors and selections onto the GUI Interaction test problem. Software Reliability © Aditya P. Mathur 2003 18

Review: Selection of Test cases n The problem is to select a set of test cases, each test case consisting of one selection from each factor. n The lower bound on the number of tests is 7 x 6=42. n One can often get close to the lower bound. Software Reliability © Aditya P. Mathur 2003 19

Three Methods for GUI Test Generation n GUI Test: Enumerate each factor and duplicate the elements when necessary. Cover all possible interaction pairs. Random Test: Generate the elements of each factor randomly, duplicating an element when necessary. Cover all possible interaction pairs. Mutually Orthogonal Latin Squares (MOLS Test): Generate the elements of each factor by using MOLS. Software Reliability © Aditya P. Mathur 2003 20

Latin Squares A Latin square of order n is an n x n matrix of n symbols in which every symbol occurs exactly once in each row and column. n n Latin Square of order 2 Latin Square of order 3 a b b a x y z z x y y z x Introduced by Leonhard Euler in 1783. Software Reliability © Aditya P. Mathur 2003 21

Mutually Orthogonal Latin Squares n A pair of latin squares A=(aij) and B=(bij) are orthogonal iff the ordered pairs (aij, bij) are distinct for all i and j. 1 2 3 1 3 1 2 A Software Reliability © Aditya P. Mathur 2003 1 2 3 3 1 2 2 3 1 B 11 22 33 23 31 12 32 13 21 A and B superimposed 22

Procedure to Generate Tests Using MOLS 1. Identify factors. Determine their cardinalities. |X|=3, |Y|=4, |Z|=2 2. Arrange factors in descending order of cardinality. |Y|>=|X|>=|Z|, rewritten as: |F 1|>=|F 2>=|F 3|, where F 1, F 2, and F 3 denote, respectively, Y, X, and Z. 3. Let k=number of factors and n=|F 2| k=3, n=3 Software Reliability © Aditya P. Mathur 2003 23

Procedure to Generate Tests Using MOLS [Contd. ] 4. Prepare a table containing k columns and k x n rows divided into |F 1| blocks. Label columns as F 1, F 2, . . Fk. F 1 F 2 F 3 Block 1 Block 2 Block 3 Block 4 Software Reliability © Aditya P. Mathur 2003 24

Procedure to Generate Tests Using MOLS [Contd. ] 5. Fill column 1 with 1, 2, …|F 1| such that all rows in block 1 contain a 1, all rows in block 2 contain a 2, and so on. F 1 Block 2 Block 3 Block 4 Software Reliability © Aditya P. Mathur 2003 1 1 1 2 2 2 3 3 3 4 4 4 F 2 F 3 25

Procedure to Generate Tests Using MOLS [Contd. ] 6. Fill each block in column 2 with 1, 2, …|F 2|. Block 1 Block 2 Block 3 Block 4 Software Reliability © Aditya P. Mathur 2003 F 1 F 2 1 1 1 2 2 2 3 3 3 4 4 4 1 2 3 F 3 26

Procedure to Generate Tests Using MOLS [Contd. ] 7. Determine, if possible, (k-2) mutually orthogonal latin squares of order n. We will denote these by M 1, M 2, …. M(k -2) M 1 = 8. 1 2 3 1 3 1 2 Fill entries in block 1 of column F 3 with entries from column 1 of M 1. Software Reliability © Aditya P. Mathur 2003 27

Procedure to Generate Tests Using MOLS [Contd. ] M 1 = 1 2 3 1 3 1 2 Block 1 Block 2 Block 3 Block 4 Software Reliability © Aditya P. Mathur 2003 F 1 F 2 F 3 1 1 1 2 2 2 3 3 3 4 4 4 1 2 3 1 2 3 28

Procedure to Generate Tests Using MOLS [Contd. ] 9. Fill entries in blocks 2 and 3 of column F 3 with entries from columns 2 and 3 of M 1. Software Reliability © Aditya P. Mathur 2003 29

Procedure to Generate Tests Using MOLS [Contd. ] M 1 = 1 2 3 1 Block 2 Block 3 Block 4 Software Reliability © Aditya P. Mathur 2003 3 1 2 F 1 F 2 F 3 1 1 1 2 2 2 3 3 3 4 4 4 1 2 3 1 2 3 1 3 1 2 ? 30

Procedure to Generate Tests Using MOLS [Contd. ] 10. We have now exhausted all columns of M 1. How do we fill block 4 of column F 3? Reuse columns of M 1 starting with column 1. Software Reliability © Aditya P. Mathur 2003 31

Procedure to Generate Tests Using MOLS [Contd. ] M 1 = 1 2 3 1 Block 2 Block 3 Block 4 Software Reliability © Aditya P. Mathur 2003 3 1 2 F 1 F 2 F 3 1 1 1 2 2 2 3 3 3 4 4 4 1 2 3 1 2 3 1 3 1 2 3 32

Procedure to Generate Tests Using MOLS [Contd. ] 11. Note that F 3 corresponds to factor Z which can assume only one of two values. Therefore we remove all instances of 3 from column F 3 of our table. Software Reliability © Aditya P. Mathur 2003 33

Procedure to Generate Tests Using MOLS [Contd. ] M 1 = 1 2 3 1 Block 2 Block 3 Block 4 Software Reliability © Aditya P. Mathur 2003 3 1 2 Y X Z F 1 F 2 F 3 1 1 1 2 2 2 3 3 3 4 4 4 1 2 3 1 2 2 1 1 2 34

Procedure to Generate Tests Using MOLS [Contd. ] Are we done generating the tests? Not yet! We need to (a) fill in the blank entries and (b) check if all interaction pairs are covered. Let us begin with (b). 12. We have a total of 3 x 4 x 2=24 interaction pairs amongst factors X, Y, and Z. It is easy to see from the table that all (X, Y) pairs are covered. Also, despite the blank entries under column F 3 (this corresponds to Z), we note that all (Y, Z) and (X, Z) pairs are covered. Voila! We are done and have generated 12, the minimum, number of tests required to cover all pairs in this example. . Software Reliability © Aditya P. Mathur 2003 35

Procedure to Generate Tests Using MOLS [Contd. ] 13. Fill in the blank entries in column F 3. Block 1 Block 2 Block 3 Block 4 Software Reliability © Aditya P. Mathur 2003 Y X Z F 1 F 2 F 3 1 1 1 2 2 2 3 3 3 4 4 4 1 2 3 1 2 2 1 1 1 2 1 36

Procedure to Generate Tests Using MOLS [Contd. ] 14. The table we just completed provides test requirements. It is easy to derive test specifications from this table. Sample tests for X={New, Open, Close), Y= {Select All, Cut, Paste, Undo}, Z={Symbol, Equation} From Row 1: X=New Y =Select All Z=Symbol From Row 12: X=Close Y =Undo Z=Symbol Software Reliability © Aditya P. Mathur 2003 37

Special Cases: k-2>n n n k-2>n: This implies that while filling in the columns for factors, you will run out of matrices. For example, suppose that k=10, and n=7, i. e. , that there are 10 factors and we will be using MOLS of order 7. Now, it turns out that there are 6 MOLS of order 7. We can use matrices M 1, M 2, …M 6 for factors F 3, F 4, …F 8. What do we do for F 9 and F 10? The solution to the problem is to randomize the generation of entries in columns F 9 and F 10. Of course, this might mean that some pairs may remain uncovered and hence additional tests may need to be generated. Software Reliability © Aditya P. Mathur 2003 38

Special Cases: |Fi|<n n n |Fi|<n, i>2: This implies that factor Fi has a cardinality less than the order of the MOLS we are using. F 3 in our earlier example is one such factor. As illustrated earlier, this will lead to blank entries. These are known as “don’t care” entries and can be filled appropriately. Software Reliability © Aditya P. Mathur 2003 39

Special Cases: MOLS of order n do not exist n n This is true for n=2 and 6. In this case we are out of luck! However, one could always use MOLS of the next higher degree. This will lead to many don’t care entries that can be handled as described earlier. Think of the advantages and disadvantages of this approach. Software Reliability © Aditya P. Mathur 2003 40

Special Cases: n<|F 1| n n When the order of MOLS is less than the cardinality of |F 1|, we run out of matrix columns to use. In this case we reuse the matrix columns. Software Reliability © Aditya P. Mathur 2003 41

Experiments with three algorithms n Recall the three algorithms for generating GUI tests: GUITest, Random, and MOLS test. Which of these is the best? n Lee White conducted experiments to answer the above question. n Results follow. . Software Reliability © Aditya P. Mathur 2003 42

(Sample) Number of tests generated by three algorithms k Factors Min. MOLS Random GUITest 4 3/2/2/2 6 6 8 10 5 5/4/3/3/2 20 20 20 30 7 6/6/5/4/3/3/2 36 38 38 63 10 10/10/10/9/9/8/7/6/5 100 /4 127 161 315 13 11/11/10/10/9 121 /9/9/8/8/8/7 163 217 485 Software Reliability © Aditya P. Mathur 2003 43

Conclusions from the Experiments n n n MOLS achieves lower bound in most cases. Not in n=6 and 10 and when the randomization is to be done. GUITest and Random perform comparably for small k but become worse as k increases. In all cases, MOLS performed better than Random and GUITest, and Random performed better than GUITest. Software Reliability © Aditya P. Mathur 2003 44

Implications for GUI Maintenance: Addition of a Screen n n Development of GUIs is often incremental; screens are often developed one or a few at a time and full functionality of all GUI objects may not be provided. . The incremental changes to a GUI must be tested. How does the addition of a screen, or any change in the GUI, affect the set of tests already developed? n If the screen transitions depend only on the current screen, and not on the entire history, then the addition of a screen will mean the addition of GUI events to corresponding test sequences and not to all tests sequences. Software Reliability © Aditya P. Mathur 2003 45

GUI Object Modification or Screen Deletion n n Similar effect on test sequences could occur when a GUI object is modified or a screen is deleted. In general, any screen transitions could be affected by a change and must be checked. What is the effect on a given test sequence of adding a new GUI object? n Note that a new object corresponds to a new factor. Software Reliability © Aditya P. Mathur 2003 46

Addition of a GUI Object n n n If there exists an unused MOLS matrix then the test table can be easily modified by adding a column in the proper location corresponding to the new factor. The remaining columns would not be affected unless the cardinality of the new factor is greater than that of F 1 or F 2. In this latter case the number of tests would increase and the entire table regenerated. What is the effect on a given test sequence of deleting a GUI object, ie. , deleting a factor? Software Reliability © Aditya P. Mathur 2003 47

Deletion of a GUI Object n There will be no change in the number of tests if the factor deleted is not F 1 or F 2. What is the effect on a given test sequence of adding a GUI screen that has new GUI objects? What is the effect on a given test sequence of deleting a GUI screen and thereby deleting some GUI objects? ? Software Reliability © Aditya P. Mathur 2003 48

Stability of Tests What is the impact on the test sequence of small changes in the GUI? n n n If the GUI is modified slightly then MOLS is the most stable algorithm as long as there are more MOLS matrices available. The Random algorithm is less stable but allows the reuse of previous tests. The GUITest algorithm is the least stable and requires many more tests to be added. Which of the three approaches would you recommend when the GUI is unpredictable both during design and maintenance? Software Reliability © Aditya P. Mathur 2003 49

Using Don’t Care Entries: Prioritized List How best to use the don’t care entries generated when using the MOLS approach? n n n Use these entries to test for additional interactions between modified GUI objects. Use these entries to test for modified GUI elements and closely related GUI objects or screens; for example other unmodified GUI objects within the same screen, or within screens that immediately precede or follow a modified screen. Use these entries to test for interactions between modified GUI objects and other arbitrary GUI objects in the test sequence. Software Reliability © Aditya P. Mathur 2003 50

Using Don’t Care Entries: More Ideas! n n n Identify screens and GUI objects where failures contribute to a higher overall risk to the successful operation of the system. Use these entries to test for interactions between higher and lower risk elements of the GUI. Use these elements to utilize criteria such as boundary value testing. Software Reliability © Aditya P. Mathur 2003 51

Questions n Are the tests generated using MOLS feasible? n How would you detect and handle infeasible tests? n How would you treat sub-menus? n Is pair wise testing sufficient? Software Reliability © Aditya P. Mathur 2003 52

GUI Testing Using Responsibilities n n A responsibility is an end-effect that the user of a GUI wants to achieve. We assume that the end-effect is observable in the surrounding environment that includes memory, peripheral device, application software, etc. Examples: Open a file. Copy a section of a file and paste it at another place within the same file. Select a sequence of vertically placed objects, align them at their left edges, and evenly distribute them vertically. Software Reliability © Aditya P. Mathur 2003 Establish communication with a pacemaker, and download data recorded in its internal memory during the past 24 hours. 53

Complete Interaction Sequences (CIS) n n A CIS is sequence of objects and selections made by a user to produce a desired response (or to fulfill a desired responsibility). Examples: [Show examples from Powerpoint. ] n A CIS might overlap with another, or be fully contained, or simply share GUI objects. . Software Reliability © Aditya P. Mathur 2003 54

Identification of Responsibilities and CISs n Responsibilities are identified using all available sources that include design documents. n A CIS is defined for each responsibility. n A finite state machine is constructed for each CIS. n n Several transformations are applied to each FSM to reduce its complexity and thus tackle the state explosion problem. These transformations abstract two types of components: n Strongly connected component n Structural symmetry component Each reduced FSM is used to generate tests for the corresponding CIS. Software Reliability © Aditya P. Mathur 2003 55

Strongly Connected Components n A sub. FSM is a strongly connected component if for any pair of states (s 1, s 2), there exists a directed path from s 1 to s 2. I 1 I 2 Q P R O 1 S Software Reliability © Aditya P. Mathur 2003 56

Design Tests n In the example below we need two design tests that match up the two inputs with the output and include all states. Design tests: {<I 1 Q, R, S, P, Q, R, O 1>, <I 2, P, Q, R, S, P, Q, R, O 1>} I 1 I 2 Q P R O 1 S Software Reliability © Aditya P. Mathur 2003 57

Implementation Tests n n Implementation tests include all transitions that do not occur in the design but do occur in the implementation. To obtain implementation tests determine all unused selections of each GUI object in the component under consideration. If these selections correspond to transitions outside of this component to other states of the CIS, then they produce new outputs of the component. Similarly, new inputs to a component are obtained by considering possible transitions to the states of a component. Software Reliability © Aditya P. Mathur 2003 58

Implementation Tests: Additional Transitions I 1 I 2 Q P R * S I 3 O O 2 A different GUI selection, not in the original design, produces this transition. Software Reliability © Aditya P. Mathur 2003 59

Implementation Tests n The revised state diagram leads to implementation tests. Implementation tests: : {<I 1, Q, R, S, P, Q, R, S, P*, Q, R, O 1>, <I 1, Q, R, S, P, Q, R, S, P*, Q, R, S, O 2>, } There are four more tests. Can you derive them? Software Reliability © Aditya P. Mathur 2003 60

Another example: Edit-Cut-Copy CIS A Select Text Strongly connected component B C Copy Cut E Software Reliability © Aditya P. Mathur 2003 Edit D Complete 61

Tests for the Select Text-Edit-Cut-Copy CIS n Two tests for this CIS are given below. <A, B, C, B, D, B, C, E>, <A, B, D, B, C, B, D, E> There are two more tests. Can you derive them? n n Notice that there are two different ways to get to E for Cut an Copy, respectively. . Further, for each of these two ways, there are different ways in which the sequence could be started, one with C and the other with D. Software Reliability © Aditya P. Mathur 2003 62

Another example: Account Withdrawal A Start Strongly connected component Select Account D E B C Print result Select other transactions Withdraw How many and what test(s) are required for this CIS? Software Reliability © Aditya P. Mathur 2003 63

Structural Symmetry Components: Definition n A component has structural symmetry if it has one input to state s 1, one output from state s 2, and one or more directed paths from s 1 to s 2. Here we assume that each directed path from s 1 to s 2 contains only one state other than s 1 and s 2. Structural symmetry components must also satisfy the following constraints: Software Reliability © Aditya P. Mathur 2003 64

Structural Symmetry Components: Constraints n n The path followed to get to the input of the component has no effect on the internal paths and states of the component. Any path traversed following the output of the component is independent of the path that was traversed within the component. Software Reliability © Aditya P. Mathur 2003 65

Structural Symmetry Components: Example I 1 Are the symmetry conditions met? Open File Dialog A Are they also met if UNDO transitions are added from B and C to A ? Select File by Name UNDO B D C Select File by Selection File Selected O 1 Software Reliability © Aditya P. Mathur 2003 66

Structural Symmetry Component: Design Tests n For the component in the previous example, without the UNDO transitions, we need two design tests: <I 1, A, B, D, O 1> I 1 <I 1, A, C, D, O 1> Select File by Name A Open File Dialog C B D Select File by Selection File Selected O 1 Software Reliability © Aditya P. Mathur 2003 67

Destruction of Structural Symmetry Component n n For implementation testing we need to discover additional transitions. The additional transitions will likely destroy the structural symmetry This will lead to an increase in the number of implementation tests. However, the discovery and abstraction of components reduces the number of tests required to test a CIS FSM. Why? This is because when testing the CIS FSM, we select only one of the several paths in each test of the FSM. Of course, we make sure that each path of the abstracted component is tested at least once. Software Reliability © Aditya P. Mathur 2003 68

Testing the Reduced FSM: Design Tests n n Each abstracted component in the CIS FSM is replaced by a superstate. To conduct a design test, construct a sufficient test set as follows: n n n Each test case must correspond to a sequence of GUI actions that begin at the start node and ends at the terminating node. The test set must cover all paths in the reduced FSM. each time a superstate is entered, an appropriate path of the corresponding component is selected. All design tests for each abstracted component must be exercised in the complete test of the FSM. Software Reliability © Aditya P. Mathur 2003 69

Testing the Reduced FSM: Implementation Tests n n n The reduced FSM for the implementation tests is likely to be larger in the its number of states than the one used in design test. . Some abstracted components might be invalidated and therefore replaced by their “complete” versions. A sufficient set of implementation test cases must: (a) cause all paths in the modified FSM to be executed and (b) include all the implementation tests for each abstracted component at least once. Software Reliability © Aditya P. Mathur 2003 70

Other Considerations n n The FSM does not model the effects of the GUI and therefore the result of traversing each path must be examined carefully. A cycle may need to be traversed different number of times at different points along a path. All distinct paths must be generated from the start to the terminating state. However, we can exclude any path where a cycle is traversed more than once. Note that this method is more exhaustive than simply covering each state and each transition once. Why? Software Reliability © Aditya P. Mathur 2003 71

Example n n This example shows how a CIS FSM can be reduced and tests derived from the reduced FSM. The CIS considered is an Edit-Cut-paste-Copy-Paste sequence that involves opening two files. We begin by developing the FSM for the given CIS. Next, we identify the strongly connected and structural symmetry components. n All such components are replaced by a superstate. n We can now develop tests for the reduced FSM. Software Reliability © Aditya P. Mathur 2003 72

FSM for Edit-Cut-Paste-Copy-Paste CIS Initial C 1 Open File Total tests required: 50 Name Select C 2 Select Edit Paste Software Reliability © Aditya P. Mathur 2003 Name HLT Cut Finish Open 2 File HLT Move Cursor C 3 Copy Ready/ Edit Move Cursor 73

Reduced FSM Total tests required: 8 Initial Move Cursor Paste Finish File C 1 2 tests C 3 C 2 2 tests 4 tests Move Cursor Ready/ Edit Software Reliability © Aditya P. Mathur 2003 74

Evaluation of the CIS-based Technique n n The GUIs of two applications, A and B, were subjected to CIS based testing technique. The primary objectives of the experiment were to determine (a) how good is the technique in detecting defects in GUI implementation and (b) the reduction in the number tests achieved through the reduction of the FSM. n Defect: Serious departure from the specified behavior. n Surprise: User recognized departure from expected behavior. Software Reliability © Aditya P. Mathur 2003 75

Experiment Details n n n Application A: MS Windows 98. n Application A 1: Arabic enabled n Application A 2: Arabic version n Application A 3: English version Application B: GVISUAL, a component of an object-oriented multimedia data base system. Application A: Sample CIS: Install an application, e. g. MS Word, from a CD. Software Reliability © Aditya P. Mathur 2003 76

Experiment Details [contd. ] n Sample responsibilities: n n n Application A: Install an application, e. g MS Word, from a CD. Application B: A condition box that forces certain constraints on the relations between objects. Sample defect seeded: Application B: n Creating a method without identifying its base class. Total of six defects seeded. Software Reliability © Aditya P. Mathur 2003 Results follow… 77

Results of the Evaluation of the CIS-based Technique # of Tests Application A 1 #CIS: 13 Application A 2 #CIS: 13 Application A 3 #CIS: 13 Application B #CIS: 52 Software Reliability © Aditya P. Mathur 2003 *Defects/Surprises Found 28 32 6/3 12/4 28 32 4/2 Design Implementation 28 32 4/1 6/1 Design Implementation 58 112 1/2 2/2 Design Implementation Seeded defects found 5/0 6/0 *Defects/surprises found by implementation tests include those found by Design tests. 78

Reduced versus all directed paths: # of Design Tests Reduced FSM All directed paths Application A 28 40 Application B 58 86 Software Reliability © Aditya P. Mathur 2003 *Defects/surprises found by implementation tests include those found by Design tests. 79

Conclusions n n n Though the design tests are useful, implementations tests are highly recommended. Consistency in the ratio [Tests/CIS] suggests that the number of tests could be estimated from the number of CIS. The savings in # of tests due to reduction is not impressive. However, this is due to the low complexity of the CIS used. More complex CIS will likely to to more savings. Software Reliability © Aditya P. Mathur 2003 80