Using Logic Criterion Feasibility to Reduce Test Set

Using Logic Criterion Feasibility to Reduce Test Set Size While Guaranteeing Fault Detection Gary Kaminski and Paul Ammann ICST 2009 March 24 Version

Motivation Current logic criteria: – generate large test sets (Combinatorial) or – do not guarantee detecting logic faults (RACC) Goal: - generate smaller test sets while still guaranteeing fault detection Assumption: - restrict attention to minimal Disjunctive Normal Form (DNF) Boolean predicates tested in isolation

A Word About Infeasibility • Infeasible Test Requirements are a hassle! – They can bloat test sets – They can thwart subsumption hierarchies – Example: RACC and CACC • May be infeasible to satisfy RACC, but feasible to satisfy CACC • RACC subsumes CACC, yet for a literal in a predicate, CACC may yield a test case when RACC does not • Coverage Criteria for Detecting Logic Faults – – If all test requirements feasible, simple criteria are enough More complex criteria needed to fill in the gaps This paper analyzes feasibility at a “low” level Result: Minimal, fault-detecting test sets

Building Test Sets Guaranteed to Detect Faults (Current) Apply Criterion 1 T 1 Apply Criterion 2 Test Set = Predicate P: ab + a!c T 1 Apply Criterion 3 + T 2 + T 3 Apply Each Criterion to P, Component by Component • If criterion feasible on component, generate test • If criterion infeasible on component, satisfy as much as possible Result: Tests from all Criteria on all Components • Criteria are all necessary; but individual tests may be unnecessary

Analyzing Criterion Feasibility at Component Level Extract Components Criterion 1 Feasible? Yes Apply Criterion 1 T 1’ No Criterion 2 Feasible? Predicate P: ab + a!c No Yes Apply Criterion 2 Apply Criterion 3 T 2’ Test Set = T 1’ + T 2’ + T 3’ Criterion Feasibility Analyzed, Component by Component • If criterion feasible on component, generate test and FINISH • If criterion infeasible on component, partially satisfy and go to next criterion Result: Every resulting test has a reason for being there Note: Some details glossed over in this figure…

Minimal DNF • Terms separated by OR, literals by AND ab + a!c vs. a(b + !c) • Make each term true and other terms false ab + ac vs. ab + abc • Impossible to remove a literal or term without changing the predicate ab vs. abc + ab!c

Minimal DNF Logic Faults Original: ab + bc • • • Literal Insertion Fault: Literal Reference Fault: Literal Omission Fault: abc + b!c ab!c + b!c a!c + b!c b + b!c A test set detecting these faults also detects others

Lau and Yu’s Fault Hierarchy • A test set that guarantees detection of a source fault guarantees detection of a destination fault • Ignores effect of criterion feasibility LIF LOF LRF TOF LNF ORF+ TNF ENF

Unique True Points and Near False Points • UTP: An assignment of values such that only one term evaluates to true. ab + !ac: 110 and 111 are UTPs for ab • NFP: An assignment of values such that the predicate evaluates to false but when a literal is omitted, it evaluates to true. ab + !ac: 100 and 101 are NFPs for b

MUTP Criterion • Find UTP tests for each term such that all literals not in the term attain 0 and 1. • Detects LIF and if feasible, detects LRF • Inexpensive to satisfy • Feasible for ab + !ac ab – 110, 111 !ac – 001, 011 • Infeasible for ab + ac ab – 110

CUTPNFP Criterion • Find a UTP - NFP pair such that only the literal of interest changes value. • Detects LOF and if feasible, detects LRF • More expensive to satisfy • Feasible for b in ab + ac UTP for ab is 110 NFP for b in ab is 100 • Infeasible for b in first term of ab + b!c + !bc UTP for ab is 111 NFP for b in ab 100 (101 makes !bc true)

MNFP Criterion • Find NFP tests for each literal such that all literals not in the term attain 0 and 1. • Detects LOF and if feasible, detects LRF • Most expensive to satisfy • Feasible for a in first term of ab + ac 010, 011 • Infeasible for a in first term of ab + !ac 010 (011 makes !ac true)

MUMCUT Criterion • Combine CUTPNFP, MNFP, and MUTP - detects LIF, LRF, and LOF but expensive - without considering feasibility need all 3 criteria to detect LRF • Other criteria require less inputs but do not guarantee fault detection (RACC) • Can we reduce MUMCUT test set size while still guaranteeing LRF detection?

MUTP Feasibility and LRF If MUTP is feasible for a term: Black – Green - MUTP detects LRF - CUTPNFP not needed to detect LRF - MNFP not needed to detect LRF For Each Term MUTP feasible? For Each Literal In Term CUTPNFP feasible? Test Set = MUTP + NFP Test Set = MUTP + CUTPNFP MNFP Test Set = MUTP + MNFP

CUTPNFP Feasibility and LRF If MUTP is infeasible for a term but CUTPNFP is feasible for a literal in the term: Black – Red – Black - Green - MUTP does not detect LRF - CUTPNFP detects LRF - MNFP not needed to detect LRF For Each Term MUTP feasible? For Each Literal In Term CUTPNFP feasible? Test Set = MUTP + NFP Test Set = MUTP + CUTPNFP MNFP Test Set = MUTP + MNFP

MNFP Feasibility and LRF If MUTP is infeasible for a term and CUTPNFP is infeasible for a literal in the term: Black – Red – Black - MUTP does not detect LRF - CUTPNFP does not detect LRF - MNFP will detect LRF For Each Term MUTP feasible? For Each Literal In Term CUTPNFP feasible? Test Set = MUTP + NFP Test Set = MUTP + CUTPNFP MNFP Test Set = MUTP + MNFP

Minimal-MUMCUT Criterion • MUTP feasible MUTP detects LRF • CUTPNFP feasible CUTPNFP detects LRF • Both infeasible MNFP detects LRF Minimal-MUMCUT: • Always need MUTP tests to detect LIF • CUTPNFP tests only when MUTP infeasible • MNFP tests only when both are infeasible “Minimal” means that every test in the test set is needed to guarantee fault detection – not minimized

New Fault Hierarchy • Black arrow: relation always holds • Green arrow: relation holds if MUTP is feasible • Red arrow: relation holds if MUTP is infeasible and CUTPNFP is feasible LIF LRF TOF LNF ORF+ TNF ENF

Case Study • Analyzed 19 Boolean predicates in an avionics software system (Weyuker, Chen, Lau, and Yu) • Number of unique literals range: 5 to 13 • Determined MUTP feasibility for each term and CUTPNFP feasibility for each literal • Examined test set size for MUMCUT vs. Minimal-MUMCUT

Case Study Results • Minimal-MUMCUT size is 12% of MUMCUT size • Savings in test set size comes from 1) CUTPNFP feasible for all 853 literals: no MNFP 2) For 24% of literals, MUTP detects LRF: no CUTPNFP 3) 16 of 19 predicates had a MUTP feasible term

Test Set Size vs. Number of Unique Literals

Conclusion • Used criterion feasibility to reduce test set size without sacrificing fault detection • Modification of fault detection relations in Lau and Yu’s hierarchy based on criterion feasibility • Introduction of the Minimal-MUMCUT criterion based on minimal DNF • Applications for software testing of programs with large predicates