NISTdeveloped Test Suites David Flater National Institute of

NIST-developed Test Suites David Flater National Institute of Standards and Technology http: //vote. nist. gov 12/9 -10/2009 TGDC Meeting

Context n Status quo n n n August 2007 draft of 2. 0 (1. 9) n n n Labs have been testing to 2002, 1. 0 (2005) Lots of proprietary, custom tooling and review processes NIST defined test methods where appropriate for entire scope In case of omission or error, fall back to status quo is an option May 2009 draft of 1. 1 (1. 01) n n Test methods for new and changed material back-ported from 2. 0 (1. 9) Status quo prevails for everything else 12/9 -10/2009 TGDC Meeting Page

Materials 1. 1 (1. 01) 2. 0 (1. 9) Usability & accessibility ✔ ✔ Security test suite ✔ ✔ Votetest ✔ ✔ Volume testing guidance N/A ✔ Reliability, accuracy, … ✔ (In the draft) Hardware workmanship Operating temperature and humidity ✔ EMC N/A Coming soon Source code analyzer tool assessment guide and test suite … 12/9 -10/2009 TGDC Meeting Page

General principles n Ability to specify tests is limited by variability of design, unavailability of system-specific details, nature of requirements n n Measure when possible n n n Measurable quantity Benchmark If not measurable, specify procedure for evaluating n n Avoid creating new requirements via test suite Test narratives Checklists Pass/fail criteria … Assessment by qualified experts 12/9 -10/2009 TGDC Meeting Page

Coverage strategies n n The VVSG contains many different sorts of requirements… some more testable than others Coverage strategies vary "as appropriate" to what is being tested n n n One test per requirement, one requirement per test At least one test per requirement, many requirements per test Many tests for one requirement Incidental testing Derived requirements Tested by exception 12/9 -10/2009 TGDC Meeting Page

Usability & accessibility n n System-independent test narratives with pass/fail criteria CIF template & how-tos n n n Common Industry Format for reporting usability test results Highly structured process surrounding the usability test(s) Voting Performance Protocol (VPP) is only in 2. 0 n n Specified benchmarks Usability measured by test lab 12/9 -10/2009 TGDC Meeting Page

Security test suite n Procedures for evaluating n n n Secure configurations n n Cryptography Access control Event logging Other security features If X is used, then it should be configured thusly… Not OEVT 12/9 -10/2009 TGDC Meeting Page

Votetest n Scope: basic, essential voting system logic n n n Tests are intentionally simple… n n A volume test (mock election) is a significant test of all supported functions together …but they exercise the complete elections and voting process n n Ability to define elections, capture, count, and report votes Voting variations Election definition and test ballots in; reports out 92 tests formalized as SQL scripts 12/9 -10/2009 TGDC Meeting Page

Volume testing guidance n n A volume test is specified in Part 3 of 2. 0 Additional guidance for conducting that test has been published in a separate document 12/9 -10/2009 TGDC Meeting Page

Reliability, accuracy, misfeed rate n n Improved test method replaces material that was historically included in the VSS/VVSG… hence, included in drafts Now evaluated using data collected during all tests, rather than a single, isolated test 12/9 -10/2009 TGDC Meeting Page

Hardware workmanship n n General build quality Durability Maintainability Temperature and humidity n n Operating humidity requirement added to 1. 1 Transportation and storage 12/9 -10/2009 TGDC Meeting Page

Electromagnetic compatibility n n Coming soon for 2. 0 Undergoing NIST internal review 12/9 -10/2009 TGDC Meeting Page

One more thing n Source code analyzer tool assessment guide and test suite n n n Test labs use source code analyzers to help assess conformity to the VVSG Source code analyzers vary in quality This guide and test suite help to determine which are fit for purpose 12/9 -10/2009 TGDC Meeting Page

Possible futures n n n Voluntary use by test labs and/or manufacturers One input to a broader effort to reduce variability in testing Starting point for independent project(s) n n n …manufacturer consortium? …jurisdictional testing? …general open source community? n EAC requirements on test plans n … not for NIST to determine 12/9 -10/2009 TGDC Meeting Page