Testing Considerations Past experience Methodology Options CAMP F

Testing Considerations (Past experience, Methodology, Options…) CAMP F 2 F Redwood City, November 2012 Jacques Durand Fujitsu Computer Systems 1

Testing serves different purposes • Validating a specification • Verifying an implementation See: https: //wiki. oasis-open. org/tab/Testing. Policy Fujitsu Computer Systems 2

Testing serves different purposes • Validating a specification • ambiguities, fuzziness • gaps (unspecified features) • inconsistencies / conflicts • Verifying an implementation • conformance • interoperability: two+ implementations interoperable (or conforming artifacts portable? ) • maintenance: is the implementation still conforming/interoperable over time? (changes in config, etc. ) See: https: //wiki. oasis-open. org/tab/Testing. Policy Fujitsu Computer Systems 3

Test assertions: a common test foundation Specification Normative statements Normative statement is Test assertions derived from addressing Actual Test suite Test Case Test Assertion: • A “measurable/testable statement of behavior for an implementation” • Blueprint for an executable test unit • see: https: //wiki. oasis-open. org/tab/Testability. Guide • see: https: //www. oasisopen. org/committees/download. php/44696/taguidelines-v 1. 0 -wd 02. pdf Fujitsu America, Inc. 4 Fujitsu Computer Systems

Example: a specification on “widgets” [requirement 1023] “A widget MUST be of cuboid shape”. Here is a test assertion addressing this requirement: TA id: widget-TA 100 -1 Target: widget Normative Source: “widget specification”, requirement 1023 Predicate: [the widget] has six facets, and each one of these is of rectangular shape. Prescription Level: mandatory ility b a g n i : t Worded dge of the tes expert nwle o o n n k a n y i b • le b a d n a t • unders Fujitsu America, Systems Inc. Fujitsu Computer 5

An example TA (CIMI) - The normative source is an interpretation (summary) of several normative statements in the original specification - The “SUT” tag identifies the larger entity “responsible” for conformance, while the “target” is a sequence of messages Fujitsu America, Systems Inc. Fujitsu Computer 6

An example TA (SCA) - If the “prerequisite” is false, the “target” is not qualified for further evaluation of the “predicate”. - The tags here are just keywords used to classify this TA. Fujitsu America, Systems Inc. Fujitsu Computer 7

What is to be tested Major CAMP entities or roles Black-box testing: through the testing of concrete test “targets” being exchanges: artifacts, messages App User#1 Messages Paa. S Provider Artifacts Fujitsu Computer Systems 8

Conformance aspects to verify: • (1) Message & Artifact well-formedness • Protocol aspects • schema validation, HTTP headers… • Request/Response basic consistency • (2) Consistency between exchanges and what is currently deployed on the Paa. S platform • Right set of resources created, after operations? • message responses consistent with platform state? Fujitsu Computer Systems 9

Benefits of Test Assertions (1): • Improve specifications • by “translating” normative requirements in precise implementation behaviors, discover : • gaps / ambiguities / inconsistencies • Align implementations (before testing) • lead to a common interpretation of the spec. • Facilitate Testing • starting point for Test Suites Fujitsu Computer Systems 10

When to write Test Assertions ? : t i f e n e b m u m i x c a e p m s r e o f h t t Bes writing e l i h w ? Implementation work Technical work On Specification ? ? POC work Product dev Test Suite developt Standardization (WD CS OS) specification work Fujitsu Computer Systems 11

A look at 3 Testing Set-ups: ( all useful ) 1. Isolation Testing - a single implementation of System Under Test (SUT) (CAMP) is being tested. - Primary goal: conformance assessment Test Driver Fujitsu Computer Systems Paa. S Provider (System Under Test) 12

A look at 3 Testing Set-ups: ( all useful ) 2. Matrix. Testing - a test scenario is repeated for each pair of implementations (or pair Paa. S Consumer-Provider) - main goal: interoperability assessment App User#1 User#2 App User#1 User#3 Paa. S Provider A Paa. S Provider B Paa. S Provider C Fujitsu Computer Systems 13

A look at 3 Testing Set-ups: ( all useful ) 3. Maintenance Testing - concerns in-production systems - main goal: check for regressions - 2 variants: (a) driven tests (b) real exchange monitoring Test Driver App User#1 (a) Paa. S Provider (System Under Test) (b) Fujitsu Computer Systems In-production Provider 14

Separating Test Operation from Test Analysis • allows for reusing analysis test tool in various set-ups Phase 1: Test operations Isolation testing Matrix testing Test Driver SUT #1 SUT #2 SUT #3 Maintenance testing users Phase 2: Test analysis Execution Trace Test Analyzer Test assertions Test Report In-production Provider S UT Fujitsu America, Systems Inc. Fujitsu Computer 15

Example of 2 -phase testing: Tools: overview WS-Interoperability. Testing Test SCENARIOS Phase 1 Client code MONITOR Interceptor Logger Execution report Phase 2 Web Service Message artifacts ANALYZER Test report Fujitsu America, Systems Inc. Fujitsu Computer 16

Benefits of Test Assertions (2): when implementing test suites • Single Test Tool/Suite for conformance logic • usable in all test set-ups • Visibility of conformance logic • in test suite (blueprint for test cases) • in test report (each report item refers to a T. A. ) Fujitsu Computer Systems 17