Automated Support for TestDriven Specification Edward L Jones

Automated Support for Test-Driven Specification Edward L. Jones IASTED SEA Conference Phoenix, Arizona October 14 -16, 2005 15 Nov 2005 Jones - IASTED/SEA 1

Outline Test-Driven Specification Paradigm n Background n An Example n Automation Scheme n Future Work n 15 Nov 2005 Jones - IASTED/SEA 2

The Paradigm: Test Driven Specification Software Development Assess Specification + Test Cases Assess Test Cases Software Testing Time 15 Nov 2005 Jones - IASTED/SEA 3

Background n Early Attention to Testing is Good V-Model of Software Testing (planning) n W-Model of Software Testing (verification) n Test-Driven Development (coding phase) n Requirements/Specification remain the hard part of software engineering n Verification options – prototyping, testing, model checking, automated test case generation n 15 Nov 2005 Jones - IASTED/SEA 4

Specifics of Our Work Black-box: focus is software function n Specification Language = Decision Table n Decision table-based test adequacy criteria n Automated conversion of decision table to analyzer program n Data-driven verification to reveal model anomalies and test data inadequacies n 15 Nov 2005 Jones - IASTED/SEA 5

Decision Tables Logic based, cause-effect specification n Within family of state-based specifications n Old technology from 1950 s-70 s n Automated programming systems n Basis for software testing n n Specification language for rule based systems n Vanthienen’ Prologa tool, other decision management tools for decision science 15 Nov 2005 Jones - IASTED/SEA 6

An Example Specification Calculate employee pay, including overtime paid at 1. 5 times the hourly rate of hourly employees for qualifying hours. The normal work week is 40 hours; the maximum number of paid hours is 80. Salaried employees earn over $30 per hour, and are paid overtime only for hours in excess of 60 hours. Salaried employees working less than 40 hours are paid for the normal work week. Hourly employees earn less than $30 per hour, and are paid for all hours worked up to the maximum; they are paid overtime for hours in excess of 40. 15 Nov 2005 Jones - IASTED/SEA 7

Stimulus-Response View of Specification hours rate 15 Nov 2005 Calculate Pay Jones - IASTED/SEA regular_pay overtime_pay 8

Decision Table for Specification (Incomplete) Conditions Computation Rule Decision Rules --------------hours > 40 | N N rate >= 30 | Y N ---------------overtime_pay = 0; | X X regular_pay = hours * rate; | X regular_pay = 40 * rate; | X --------------Actions 15 Nov 2005 Rules Jones - IASTED/SEA 9

Decision Table based Adequacy Criterion Possible adequacy criteria w. r. t. conditions, actions, rules n Functional Coverage n n Measures the thoroughness of testing w. r. t. decision table rules n FC = # computation rules satisfied # computation rules in DT 15 Nov 2005 Jones - IASTED/SEA 10

Automated Checking n Specification anomalies Ambiguous rules - test tuple satisfies multiple rules n Missing rule -- test tuple satisfies no rule n n Test data set adequacy Superfluous test data – rule satisfied by multiple test tuples n Incomplete test data – rule not satisfied by any test tuple n n Computation rule defines test oracle 15 Nov 2005 Jones - IASTED/SEA 11

Automation Generate Analyzer Build Analyzer Run Analyzer Decision Table Test Coverage Report 15 Nov 2005 Jones - IASTED/SEA Test Data Set Spec Coverage Report 12

DT Version #1 Missing Rules --------------hours > 40 | N N rate >= 30 | Y N ---------------overtime_pay = 0; | X X regular_pay = hours * rate; | X regular_pay = 40 * rate; | X -------------- 15 Nov 2005 Jones - IASTED/SEA 13

Analysis Reports - Version #1 SPEC COVERAGE REPORT TEST SET hours 39 41 40 45 15 Nov 2005 rate 9 29 30 40 VALID 35 9 315 0 MISSING 41 29 MISSING VALID 40 30 1200 0 MISSING 45 40 MISSING TEST COVERAGE REPORT Rules --> Test -- -Case 1 2 ==== -- -1 X 2 3 X 4 RULES NOT SATISFIED: FUNCTIONAL COVERAGE = 2/2 Jones - IASTED/SEA 14

DT Version #2 ---------------------------N N Y Y - - | hours > 40 - - Y - | hours > 60 - - - Y | hours > 80 N Y Y Y | rate >= 30 ---------------------------X - - - | regular_pay = hours * rate; - X X X | regular_pay = 40 * rate; X X - - | overtime_pay = 0; - - X - - - | overtime_pay = 1. 5 * rate * (hours - 40); - - - X X X | overtime_pay = 1. 5 * rate * (hours - 60); ---------------------------TEST SET hours: rate: 15 Nov 2005 35 9 41 29 40 30 45 40 70 40 90 50 Jones - IASTED/SEA 15

Analysis Reports - Version #2 TEST COVERAGE REPORT Test Case ==== 1 2 3 4 5 6 SPEC COVERAGE REPORT Rules --> -- -- -- -1 2 3 4 5 6 -- -- -- -X X X X X VALID 35 9 315 0 VALID 41 29 1160 43. 5 VALID 40 30 1200 0 VALID 45 40 1600 -900 AMBIGUOUS 70 40 1600 600 AMBIGUOUS 90 50 2000 2250 RULES NOT SATISFIED: FUNCTIONAL_COVERAGE = 6 / 6 15 Nov 2005 Jones - IASTED/SEA 16

No similar rule for salaried employee. DT Version #3 ---------------------------N N Y - - - | hours > 40 - - - Y Y - | hours > 60 - - N Y | hours > 80 N Y Y Y | rate >= 30 ---------------------------X - - - | regular_pay = hours * rate; - X X X | regular_pay = 40 * rate; X X - - | overtime_pay = 0; - - X - - - | overtime_pay = 1. 5 * rate * (hours - 40); - - - X X X | overtime_pay = 1. 5 * rate * (hours - 60); ---------------------------TEST SET hours: rate: 15 Nov 2005 39 9 41 29 40 30 45 40 70 40 90 50 Jones - IASTED/SEA 17

Analysis Reports - Version #3 TEST COVERAGE REPORT Test Case ==== 1 2 3 4 5 6 SPEC COVERAGE REPORT Rules --> -- -- -- -1 2 3 4 5 6 -- -- -- VALID 35 9 315 0 VALID 41 29 1160 43. 5 VALID 40 30 1200 0 MISSING 45 40 MISSING AMBIGUOUS 70 40 1600 600 AMBIGUOUS 90 50 2000 2250 X X X X RULES NOT SATISFIED: FUNCTIONAL_COVERAGE = 6 / 6 15 Nov 2005 Jones - IASTED/SEA 18

Conclusion n Have presented a black-box approach n Test-driven specification as a paradigm for interaction between developer and tester n Demonstrated benefits of automation n Claim that approach is applicable to existing decision based specification methods 15 Nov 2005 Jones - IASTED/SEA 19

Future Work n Extend to State-Based Specifications Mill’s State-Box model n SCR state tables n Pre-/Post-condition specifications n Consider random generation of test data used to verify specification n Use tool in software engineering courses n 15 Nov 2005 Jones - IASTED/SEA 20

Questions? 15 Nov 2005 Jones - IASTED/SEA 21

Motivation / References n Hoffman & Strooper’s 2001 paper “Prose + Test Cases = Specification” n Pamela Zave’s IEEE-TSE 1991 paper, a post -mortem on her 1980’s work on the Paisley operational specifications system “An Insider’s View of Paisley” 15 Nov 2005 Jones - IASTED/SEA 22