Requirementbased Testing A Presentation by Harry Sneed for

Requirement-based Testing A Presentation by Harry Sneed for the John von Neumann Society Budapest, 27 May 2009

The many Dimensions of Software Testing Test Objects • Units (Modules/Classes) • Components • Subsystems • Systems Test Types • Functional Testing • Performance Testing • Load Testing • Usability Testing Test Approaches • Random Testing • Creative Testing • Systematic Testing Test Bases • Code-based testing • Interface-based testing • Reference-based testing • Model-based testing • Require. -based testing

A Test is always a Test against Something Oracle Functional Specification (Oracle) Software Units Components Systems In 1978 Bill Howden introduced the notion of a test oracle in connection with Functional Testing. It means testing the System against the Requirement Specification (functional & non-functional Requirements)

Code-based Testing = Testing the Program against itself Expected results Expected paths Specification Specified Paths compare Results & Paths Potential Paths Test inputs Code Test results & Actual paths Corresponds to a Unit Test (White-Box)

Data-based Testing = Testing the Program against the Data Specification Specified Results Representative Values Test inputs Expected results compare Data Code Test results Interfaces/Files/Tables Interfaces/Databases Corresponds to an Integration Test (Grey-Box)

Requirement-based Testing = Testing the Program against the Specs Specification Specified Results Specified Inputs Test inputs Files/Tables/Panels Expected results compare Results Code Test results Files/Tables/Panels Corresponds to a System Test (Black-Box)

The Mother of all Test Tools = RXVP Requirement Evaluation & Verification Package Dr. Edward Miller from General Research Corporation FORTRAN Source Program with Assert Correct Syntax. Errors Correct Semantic Errors Correct Exec Errors Correct Struct Errors Syntax Analysis Static Analysis Execution Test Verf. Cond. Generator Indented Program Listing + Diagnostics Graph Checking + Call Checking + Units Consistency + Mode Checking + Asserted/Actual Use + Set/Use Checking Test Results + Execution Errors + Coverage Report + Assertion Exceptions Annotated Listing + VC'S + Symbolic Execution of Expressions + Simplified VC'S "Correct" Source Prog. Assertions taken from Specs(RSL) The idea was to put dirty software into a washing maschine and get out cleaned software.

Test Cases were extracted from the Code Test Object = Procedure or Module Test Case = Path thru Test Object P 2 2. Path Preconditions Before State Entry 1. Path P 1 Postconditions 3. Path P 3 4. Path 5. Path Exit P 4 After State

The Budapester Testlabor • The Budapester Test Laboratory was set up by Harry Sneed and Dr. Ed Miller from Software Research Inc in cooperation with the SZKI and Szamok in May, 1978. This was the first known attempt to outsource the testing activity. • The customer was Siemens, Munich, which was developing a large scale, distributed software system for the German Railroad – the Integrated Transport System (ITS).

Test Process in the Test Labor • Compilable Modules were delivered by Siemens every week on a magnetic tape. • Sources were statically analyzed to produce flow and structure diagrams. • Paths were marked thru the flow diagrams with magic markers. • Conditional variables were assigned values to confirm or negate every decision. • Values were assigned at test time to the test driver before each test case execution. • Results were logged to be checked later against the expected results.

Experience with the Test Labor • In the first half year 128 modules with 60881 SPL statements passed through the test labor. • 4378 code branches were tested with a coverage of 89. 7 % by 1544 test cases. • 192 program errors were discovered via dynamic analysis. • 1396 design and code deficiencies were discovered via static analysis. • The costs of the quality assurance remained below 4 DM pro statement. This was less than 10% of the development costs.

Summary of first 6 Months Comp Modules Stmts Test cases Branches Coverage Design & Program Code Defis Errors A 6 4029 B 37 7588 232 603 K 71 40735 1064 N 6 2847 S 8 Total 128 138 5 91% 130 22 2843 87. 5% 868 143 101 140 94% 110 14 5682 147 792 95% 150 6 60881 1544 4378 91. 9% 1396 190

Resume on Module Testing • In one year 281 Modules with 134 K stmts were tested with 3864 test cases at a cost of DM 75, - per test case = DM 289. 800 • The minimum branch coverage was 85% • 403 Errors were reported at a cost of DM 150, - per error = DM 60. 450 • Later in the system test by the customer, another 386 errors were uncovered. • The intensive module test had uncovered only 51% of the total errors found. • This demonstrates the fact that module testing of the code against itself is not sufficient. The software as a whole must be tested against the requirements.

RXVP progressed to test the systems against Requirements Nets R-Net Engine Monitoring Receive Signals ASSERT Report Signal Error Store Measure ASSERT Invalid Measure Validation Points Check Measure Valid Measure send Report Engine ok Alarm ASSERT Safe Range Out of safe Range

Extracting Test Cases from Requirement Nets R-Net-Entry Requirement Specification with RSL Measuring device Receive Signal invalid Measure Report Measure Error System Display 1. test case Engine Monitoring System or Valid Measure & and Store Measure Message Log Check Measure Engine ok or Engine problem Report ok 2. test case Alarm Nurse on Duty Alarm Display 3. test case

Extracting Test Cases from Business Rules Service Function Action Description If Account_Balance minus the withdrawal is more than the credit limit allow the withdrawal In case the withdrawal causes the Account_Balance to fall below the credit limit refuse the withdrawal. Conditional keywords are scanned. Test Case Generation Test Case 1 2 Account_Balance minus the withdrawal Y Y is more than the credit limit Withdrawal causes the Account_Balance Y N to fall below the credit limit Allow the withdrawal - X Refuse the withdrawal X - Each Condition is treated separately and combined with the other conditions in the same action to give the number of testcases = nr_conditions 2 3 4 N N Y N - - X -

Extracting Test Cases from Use Case Descriptions One Use Case can have one or more Test case Normally, there at least 2 Test Cases Positive outcome Negative outcome Invalid Card Use Case = Cash Withdrawal Wrong PIN Number Not enough cash Account overdrawn Test Cases = Use Case Paths Cash delivered

Extracting Test Cases from Requirement Documents Objects Test Cases Actions Functional States Non-Functional Conditions Requirements Text

Grammar types indicating Declarations to be tested Actions upon objects: The customer account should be deleted. The invoice should be sent within 30 days. Object states: The customer account is overdrawn. The customer is an employee. Rules: If the account is overdrawn a notice should be sent. Whenever someone fails to pay within 30 days, a reminder should be sent. Customer accounts should be deleted, if they are overdrawn more than 3 months. In case of a system crash, the recovery routine should be started.

Sample Requirements Text [RF 41] Notification can be divided into action and non-action (=information) notification. [RF 42] In case of an “action needed” notification, the system provides a link to the place where the action is needed. [RF 43] Notifications are added to a notification interface, which contains information about The reason of the notification The kind of the notification The type of the notification The date/time of the notification If action needed: A link to the place where the action is needed. [RF 44] The user is able “take” a notification (locked for others), do the action and mark the notification as “finished”. If the action is finished, the user name is stored by the notification. [RF 45] If at least one provider changed his feed data e. g. changing starting time, a notification is added to the notification table (see [RF 43]) so that each user can see the notification. The notification is only added to the notification table if the data is unequal to the current event data (see also [RF 30]).

Test Cases extracted from the Requirement Text Test. Case-Id Base. Type Requirement Relation Test. Type Test. Case Description ----------------------------------------------------------------------------DFI 0163 Require RF_43 perform action Notifications are added to a notification interface, which contains information about: The reason of the notification The kind of the notification The type of the notification The date/time of the notification DFI 0164 Require RF_43 checks rule If action is needed a link to the place where the action is needed. ----------------------------------------------------------------------------Test. Case-Id Base. Type Requirement Relation Test. Type Test. Case Description ----------------------------------------------------------------------------DFI 0165 Require RF_44 perform action The user is able “take” a notification (locked for others), do the action and mark the notification as “finished”. DFI 0166 Require RF_44 checks rule If the action is finished, the user name is stored in the notification. ----------------------------------------------------------------------------Test. Case-Id Base. Type Requirement Relation Test. Type Test. Case Description ----------------------------------------------------------------------------DFI 0167 Require RF_45 checks rule If at least one provider changed his feed data changing starting time, a notification is added to the notification table (see [RF_43]) so that each user can see the notification. DFI 0168 Require RF_45 checks rule The notification is only added to the notification table if the data is unequal to the current event data (see also [RF_30]). DFI 0169 Require RF_45 checks rule The user choose Provider A as primary source, so the event time is the time of Provider A. DFI 0170 Require RF_45 checks rule If Provider B updates his time to the time of the event, no notification is added. DFI 0171 Require RF_45 checks rule If Provider B sends a time different to the event time, a notification is sent.

Sample Non Functional Requirements The order entry system must be able to handle at least 1000 customer web pages at one time. A customer order must be processed within 2 seconds. The article database may contain up to 10. 000 articles. The system should be able to produce as many as 25. 000 bills in one billing run. A billing run should not take more than 2 hours. If the system crashes it should be back up and in operation within 15 minutes. Database backups should take place every 4 hours. Customers should be checked both by number and by password. Passwords should be rejected after 3 tries, customer number should be rejected after 2 tries.

Typical Non-Functional Test Cases 1. 999 customer orders at one time 2. >1000 customer orders at one time 3. response time per order < 2 sec. 4. response time per order >2 sec. 5. 9. 999 article records 6. > 10. 000 article records 7. 24. 999 bills in one billing 8. >25. 000 bills in one billing 9. run time of billing < 2 hours 10. run time of billing >2 hours 11. If system crash, recovery < 15 minutes 12. If > 4 hours, database secured 13. Submit 3 passwords where 3 rd password is correct 14. Submit 3 passwords where all 3 are incorrect 15. Submit 2 customer number where 2 nd number is correct 16. Submit 2 customer number where both are incorrect

Properties of a Test Case A Test Case: ü has a unique identifier ü serves a purpose ü refers to a Testobject ü refers to a requirement or use case ü has a precondition state ü has a postcondition state ü has a set of required input parameters ü has a set of expected results ü has a predecessor test case ü has a given environment ü has an owner ü has a status ü has a date of execution An international standard is required for the minimum Properties of test cases!!!

Functional Test Case Specification Test. Case ID: IC-Order-7 Test Case – Purpose: to test a fulfilled order Test-Object: Customer – Order Precondition: Ordered Article is on stock. Article. Amount is > Order. Position. Amount (Article. Amount - Order. Position. Amount ) > minimum – Amount Postcondition: Article. Amount = a Pre. Article. Amount – Order. Position. Amount No supply order Inputs: Order. Cust-Nr = 200100 Customer. Credit = ’A’ Order. Position-Nr = 2 Order. Position. Art – Nr = 4711 Order. Position. Amount = 3 Article. Amount = 10 Minimum – Amount = 7 Results: Text-Msg = „Article order fulfilled” Environment: MS – Internet, Windows – XP Usecase tested: Customer. Order. Processing Status: tested Date: 29. 8. 2005.

From Requirements via Test Cases to Test Scripts Tester enhances & refines Test Cases Text Analysis Text Analyzer generates Test Cases Repository Loader Test Scripts Tester executes Test Case DB Repository Unloader Test Script Generator generates Test Scripts for Test Automation

Text. Anal = Tool for extracting Test Cases from English and German Language Documents Text Pre Processor Object List Txt. Scan Text Txt. Anal Processor Object Model Objects - Use Cases - Actions - States - Rules Test Cases Text Excel XML Rule Violations Format Rules Rupp Rules Spec Metrics Size Quantity Complexity Repository Entries Sections - Use Cases - Objects - Actions - States - Rules

Requirements Analysis Demo • • Bet&Win Requirement Document Setting Key Words Scanning for Objects Analyzing the Document Extracting Test Cases Collecting Requirement Metrics Documenting Cross References Generating a test design

Experience with Requirement-based Testing of the Public Internet Portal for the state of Saxony Egovernment Web Site with three applications and circa 33 K Java Stmts 1844 Test cases were specified of which 1272 were generated 1495 Test cases were executed = 81% Test Case coverage 352 Test cases discovered at least one error = 24% Hit rate 450 Errors were reported by testers = 13 per KDSI 56 Errors were reported by users after release = 11% of all errors System test discovered 89% of all reported errors = test efficiency System test lasted 4 months from January to May 2005 392 Test days were booked against the project = 3. 8 Test cases per Testertag = 1. 1 Errors reported per tester day Customer was satisfied with the test result.

Conclusions • Test methods have not really improved since the 1970 s. • We are still using the basic approaches introduced back then. • Test tools have however improved significantly. • Today there are more and better testing tools. • The future lies in test automation. • Only through automation can we really reduce test costs and increase the test ROI. • Test ROI = (Test. Benefits – Test. Costs) / Test. Costs