Open ADE ESPI Testing and Certification Considerations Phil

Testing Methods - Conformance • Conformance Testing – Testing that a product is conformant

Testing Methods - Interoperability • Interoperability Testing – Testing that a product will interoperate

Test for Compliance • Compliance Testing – May comprise Conformance Testing, Interoperability Testing or

Compliance Testing • Steps for Conformance Testing – Develop a pro-forma PICS – Develop

Options to Consider (1) • Open. ADE develops Test Plan • Who develops (and

Options to Consider (2) • How many test labs and do we restrict the

Things to decide • Number of labs? • Do we need a conformance test

Slides: 8

Download presentation

Open. ADE – ESPI Testing and Certification Considerations Phil Beecher phil@beecher. co. uk

Testing Methods - Conformance • Conformance Testing – Testing that a product is conformant to a specification – Implementation Under Test (IUT) is exercised by a Test Harness to verify its behavior to the specification – The Test Harness can behave in a way conformant to the specification, – It may also behave in a non-conformant way, allowing negative test cases to be executed.

Testing Methods - Interoperability • Interoperability Testing – Testing that a product will interoperate with one or more other implementations. – Implementation Under Test (IUT) is tested together with another (conformant) implementation to verify their combined behavior is interoperable. – Interoperability testing is ideally against a “Golden Unit”, i. e. an implementation which has been tested to be conformant to the specification. – The Golden Unit does not behave in a non-conformant way, as it is a conformant implementation, so negative testing is not possible.

Test for Compliance • Compliance Testing – May comprise Conformance Testing, Interoperability Testing or both. – Phil’s preferred method is both • Test for conformance using a test harness which can provide good coverage of both positive and negative test cases. • Test for interoperability using a known mature, conformant implementation.

Compliance Testing • Steps for Conformance Testing – Develop a pro-forma PICS – Develop a test plan – Develop test harness – who and how? – Verify test harness – Decide if test lab or labs are necessary • Assess lab capability and appoint lab(s).

Options to Consider (1) • Open. ADE develops Test Plan • Who develops (and maintains) Test Harness? – (Zig. Bee Alliance – each test lab develops its own test harness) – Wi-Fi Alliance – lead lab (Wi-Fi staff) develop test harness and supply to labs. • Do we use Conformance testing, Interop testing or both to assess product? – Zig. Bee uses a bit of a hybrid – Wi-Fi also uses hybrid – Bluetooth uses both: conformance followed by Interop with popular shipping devices

Options to Consider (2) • How many test labs and do we restrict the number? – Zig. Bee has 3 in 2 countries – Wi-Fi has 15 in 8 countries – UCAIug 61850 appoints any who meet criteria – Is there sufficient business for more than 1? – Do we need any, or can we use an accredited test harness, with self-test? • Who is the certification body? – Does UCA do it? – Does ISO 65 accredited body (e. g. NEMA) do it?

Things to decide • Number of labs? • Do we need a conformance test harness and interop test harnesses (Golden platforms)? • How do we decide who develops test harness? • How do we ensure availability of test tools / low bar for entry?