Conformance Testing Lynne S Rosenthal Software Diagnostic and

Conformance Testing Lynne S. Rosenthal Software Diagnostic and Conformance Testing Division (301)975 -3353 lynne. rosenthal@nist. gov

Outline Software Standards and Testing l Conformity assessment l Conformance Testing l – what is it – why bother – how it works

Software Standards l Promote portability & interoperability l Success depends on: – Efficiency of standard development process – Quality of resulting standard – Ability to ensure implementations conform to the standard

Software Testing - process of trying to find errors by means of experimentation l Testing of realistic system not exhaustive l – can show presence of errors not their absence l Testing is either – structural testing (white box) – functional testing (black box) • conformance testing

Conformance Testing l Determines which areas of the specification* are implemented correctly. – tests capture the technical description of the specification – testing is bound in scope by the specification – internal structure of product-under-test not accessible *publicly available standard

Why: conformity assessment l ISO Guide 2: “any activity concerned with determining directly or indirectly that relevant requirements are fulfilled. ” l communication between buyer and seller – buyers increased confidence – sellers substantiate claims

Why: Improves quality of software l Objective method for evaluating products l – ensures presence of required characteristics l Promote portability and interoperability

How: building tests l Falsification Testing – find errors by means of experimentation – show presence of errors not their absence – prove non-conformance l Tests based on: – required functionality – logical errors by misunderstanding requirements – errors from boundary conditions and divergence – common programming errors

Requirements Software Conformance Tests Test programs

When: l Develop in parallel with specification – feedback to specification for clarification, correction, etc. l Develop in partnership with product developers

Summary: Conformance tests l Determine if a product has required characteristics l Increases product confidence l Communication between buyer and seller è Improve quality of software