Testing ObjectOriented Software Test Coverage Criteria Jeff Offutt

Testing Object-Oriented Software Test Coverage Criteria Jeff Offutt SWE 737 Advanced Software Testing

Testing OO Software Most of these ideas were developed after 2000 Few are widely used Most adapt graph-based testing SWE 737 © Jeff Offutt 2

OO Testing Outline 1. Example 2. The Yo-Yo Graph and Polymorphism 3. Categories of Inheritance Faults 4. Testing Inheritance, Polymorphism and Dynamic Binding 5. Object-Oriented Testing Criteria SWE 737 © Jeff Offutt 3

Example DU Pairs and Anomalies A -u -v -w +h() +i() +j() +l() Consider what happens when an overriding method has a different def-use def-set than the overridden method Method Defs A: : h () {A: : u, A: : w} A: : i () A: : j () B -x +h () +i () C -y +i () +j () SWE 737 {A: : u} {A: : v} A: : l() B: : h() Uses {A: : w} {A: : v} {B: : x} B: : i() DU anomaly {B: : x} C: : i() C: : j() A: : h() calls j(), B: : h() does not {C: : y} def-use © Jeff Offutt {C: : y} DU anomaly 4

OO Testing Outline 1. Example 2. The Yo-Yo Graph and Polymorphism 3. Categories of Inheritance Faults 4. Testing Inheritance, Polymorphism and Dynamic Binding 5. Object-Oriented Testing Criteria SWE 737 © Jeff Offutt 5

Polymorphism Headaches (Yo-Yo) A +d () +g () +h () +i () +j () +l () Actual type A B d() g() h() i() C i() j() l() k() j() l() Object is of actual type A A: : d () d() calls g(), which calls h(), which calls i(), which calls j() SWE 737 © Jeff Offutt 6

Polymorphism Headaches (Yo-Yo) A A +d () +g () +h () +i () +j () +l () g() B h() i() C A B +h () +i () +k () d() Actual B d() g() j() l() k() i() j() l() h() i() k() type C i() j() l() Object is of actual type B B: : d () SWE 737 © Jeff Offutt 7

Polymorphism Headaches (Yo-Yo) A A +d () +g () +h () +i () +j () +l () d() g() B h() i() C A d() g() j() l() k() i() j() l() h() i() B B +h () +i () +k () C A C +i () +j () +l () SWE 737 B Actual d() g() k() i() j() l() h() i() C i() k() j() l() type Object is of actual type C, C: : d () © Jeff Offutt 8

OO Testing Outline 1. Example 2. The Yo-Yo Graph and Polymorphism 3. Categories of of Inheritance Faults 4. Testing Inheritance, Polymorphism and Dynamic Binding 5. Object-Oriented Testing Criteria SWE 737 © Jeff Offutt 9

Potential for Faults in OO Programs • Complexity is relocated to the connections among components • Less static determinism – many faults can now only be detected at runtime • Inheritance and Polymorphism yield vertical and dynamic integration • Aggregation and use relationships are more complex • Designers do not carefully consider visibility of data and methods SWE 737 © Jeff Offutt 10

Object-oriented Faults n Only consider faults that arise as a direct result of OO language features: – inheritance – polymorphism – constructors – visibility n Language independent (as much as possible) SWE 737 © Jeff Offutt 11

OO Faults and Anomalies Acronym SWE 737 Fault / Anomaly ITU Inconsistent Type Use SDA State Definition Anomaly SDIH State Definition Inconsistency SDI State Defined Incorrectly IISD Indirect Inconsistent State Definition ACB 1 Anomalous Construction Behavior (1) ACB 2 Anomalous Construction Behavior (2) IC Incomplete Construction SVA State Visibility Anomaly © Jeff Offutt Examples shown 12

Inconsistent Type Use (ITU) No overriding (no polymorphism) n C extends T, and C adds new methods (extension) n An object is used “as a C”, then as a T, then as a C n Methods in T can put object in state that is s. push (“Steffi”); inconsistent. Vector for C s. push (“Joyce”); n -array +insert. Element. At() +remove. Element. At() Stack call SWE 737 +pop (): Object +push (): Object s. push (“Andrew”); dumb (s); s. pop(); // Stack is empty! void dumb (Vector v) { v. remove. Element. At (v. size()-1); } © Jeff Offutt 13

State Definition Anomaly (SDA) X extends W, and X overrides some methods n The overriding methods in X fail to define some variables that the overridden methods in W defined n W v u m() n() X x n() Y w m() SWE 737 • W: : m () defines v and W: : n() uses v • X: : n () uses v • Y: : m () does not define v For an object of actual type Y, a data flow anomaly exists and results in a fault if m() is called, then n() © Jeff Offutt 14

State Definition Inconsistency (SDIH) Hiding a variable, possibly accidentally n If the descendant’s version of the variable is defined, the ancestor’s version may not be n W v u m() n() X x n() overrides (hides) SWE 737 Y v m() • Y overrides W’s version of v • Y: : m() defines Y: : v • X: : n() uses v … getting W’s version of v For an object of actual type Y, a data flow inconsistency may exist and result in a fault if m() is called, then n() © Jeff Offutt 15

Anomalous Construction Behavior (ACB 1) Constructor of W calls a method f() n A child of W, X, overrides f() n X: : f() uses variables that should be defined by X’s constructor n W Calls When an object of type X is constructed, W() is run before X(). W() f() Overrides X Uses SWE 737 x X() f() When W() calls X: : f(), x is used, but has not yet been given a value! © Jeff Offutt 16

State Visibility Anomaly (SVA) A private variable v is declared in ancestor W, and v is defined by W: : m() n X extends W and Y extends X n Y overrides m(), and calls W: : m() to define v n W W -v Overrides, m() calls X -v Overridesm() SWE 737 X Y Overrides, m() calls Y m() X: : m() is added later X: : m() does not call W: : m() Y: m() can no longer call W: : m(), so v is not defined! © Jeff Offutt 17

OO Testing Outline 1. Example 2. The Yo-Yo Graph and Polymorphism 3. Categories of Inheritance Faults 4. Testing Inheritance, Polymorphismand Dynamic Binding 5. Object-Oriented Testing Criteria SWE 737 © Jeff Offutt 18

Coupling Sequences n Pairs of method calls within body of method under test: – Made through a common instance context – With respect to a set of state variables that are commonly referenced by both methods – Consists of at least one coupling path between the two method calls with respect to a particular state variable n Represent potential state space interactions between the called methods with respect to calling method n Used to identify points of integration and testing requirements SWE 737 © Jeff Offutt 19

Types of Def-Use Pairs def use A () intra-procedural data flow (within the same unit) A( ) B( ) def A() use B() F () object-oriented direct coupling data flow SWE 737 def A () last-def A () B () use first-use B () full coupling M () N( ) inter-procedural data flow A() M( ) def A() B() N() use B() F( ) object-oriented indirect coupling data flow © Jeff Offutt 20

Coupling-Based Testing (from Ch 2) n n n Test data and control connections Derived from previous work for procedural programs Based on insight that integration occurs through couplings among software artifacts SWE 737 Caller F x = 14 y = G (x) last-def-beforecall site print (y) first-use-after-call print (a) first-use-in-callee b = 42 last-def-beforereturn Callee G (a) return (b) © Jeff Offutt 21

Polymorphic Call Set public void f ( W o ) { Set of methods that can potentially execute as result of a method call through a particular instance context pcs (o. m) = {W: : m, Y: : m, X: : m} … j o. m(); … l o. l(); … k o. n(); } SWE 737 © Jeff Offutt 22

Example Coupling Sequence Client f W o bound to instance of W -v : -u : h def (o) i o. m() +n() +l() def (W: : v) j o. l() X Z Y -w : +m() +l() SWE 737 +m() +n() Coupling with sequence with respect to W: : u respect to W: : v l () def (W: : u) -x : +n() m () k o. n() n () use (W: : u) use (W: : v) © Jeff Offutt 23

Example Coupling Sequence (2) Client f W o bound to instance of Z -v : -u : h def (o) i o. m() +n() +l() def (Z: : x) j o. l() X Z -x : +n() +m() +n() Y +m() +l() SWE 737 l () def (W: : u) k o. n() -w : m () n () use (Z: : x) Coupling sequence with respect to Z: : x use (Z: : x) © Jeff Offutt 24

OO Testing Outline 1. Example 2. The Yo-Yo Graph and Polymorphism 3. Categories of Inheritance Faults 4. Testing Inheritance, Polymorphism and Dynamic Binding Testing Criteria 5. Object-Oriented Testing SWE 737 © Jeff Offutt 25

Testing Goals n We want to test how a method can interact with instance bound to object o: – Interactions occur through the coupling sequences n Need to consider the set of interactions that can occur: – What types can be bound to o? – Which methods can actually execute? (polymorphic call sets) n Test all couplings with all type bindings possible SWE 737 © Jeff Offutt 26

All-Coupling-Sequences (ACS) : For every coupling sequence Sj in f(), there is at least one test case t such that there is a coupling path induced by Sj, k that is a sub-path of the execution trace of f(t) n At least one coupling path must be executed n Does not consider inheritance and polymorphism SWE 737 © Jeff Offutt 27

All-Poly-Classes (APC) : For every coupling sequence Sj, k in method f(), and for every class in the family of types defined by the context of Sj, k, there is at least one test case t such that when f() is executed using t, there is a path p in the set of coupling paths of Sj, k that is a subpath of the execution trace of f(t) Includes instance contexts of calls n At least one test for every type the object can bind to n Test with every possible type substitution n SWE 737 © Jeff Offutt 28

All-Coupling-Defs-Uses (ACDU) : For every coupling variable v in each coupling Sj, k of t, there is a coupling path induced by Sj, k such that p is a sub-path of the execution trace of f(t) for at last one test case t n Every last definition of a coupling variable reaches every first use n Does not consider inheritance and polymorphism SWE 737 © Jeff Offutt 29

All-Poly-Coupling-Defs-and-Uses (APDU) : For every coupling sequence Sj, k in f(), for every class in the family of types defined by the context of Sj, k, for every coupling variable v of Sj, k, for every node m that has a last definition of v and every node n that has a first-use of v, there is at least one test case t such that when f() is executed using t, there is a path p in the coupling paths of Sj, k that is a sub-path of the trace of f() Every last definition of a coupling variable reaches every first use for every type binding n Combines previous criteria n Handles inheritance and polymorphism n Takes definitions and uses of variables into account n SWE 737 © Jeff Offutt 30

OO Coverage Criteria Subsumption All-Poly. Coupling Defs. Uses APDU All-Poly. Classes APC All-Coupling. Defs-Uses ACDU All-Coupling. Sequences ACS SWE 737 © Jeff Offutt 31

Conclusions n A model for understanding and analyzing faults that occur as a result of inheritance and polymorphism – Yo-yo graph – Defs and Uses of state variables – Polymorphic call set n Technique for identifying data flow anomalies in class hierarchies n A fault model and specific faults that are common in OO software n Specific test criteria for detecting such faults SWE 737 © Jeff Offutt 32

References 1. Testing Object Oriented Software: A Survey, Robert V. Binder, Wiley’s Software Testing, Verification and Reliability, 6(3 -4): 125252, December 1996 2. Testing Object-oriented Software Systems, Harry M. Sneed, 24 th European Conference on Object-Oriented Programming (ECOOP), Maribor, Slovenia, June 2010 (Anastasia) 3. Testing Coupling Relationships in Object-Oriented Programs, Roger T. Alexander, Jeff Offutt, and Andreas Stefik Wiley’s Journal of Software Testing, Verification, and Reliability, 20(4): 291 -327, December 2010 SWE 737 © Jeff Offutt 33