VVSG 1 1 Test Suite Status Mary Brady

VVSG 1. 1 Test Suite Status Mary Brady Manager, NIST Information Systems Group, Software and Systems Division, ITL http: //vote. nist. gov TGDC Meeting, July 2011

Background n n n Status quo: Labs have been testing to VVSG 1. 0 (2005) using proprietary, custom tooling and review processes In 2007 -08, NIST developed a set of public test suites for VVSG 2. 0, to be used as part of the EAC Testing and Certification Program In 2009, to support VVSG 1. 1, test methods for new and changed material was back-ported from 2. 0 test suites; status quo prevails for everything else TGDC Meeting, July 2011 Page 2

Test Validation n NIST test suites have been rigorously traced to VVSG requirements Reviewed by independent parties, including the EAC, VSTL labs, experts, and the public Procedural validation has been on-going n n Operating temperature and humidity -- complete Usability and accessibility – ongoing Core functionality, security under consideration Pursued obtaining in-house voting systems to aid in the evaluation of NIST test suites TGDC Meeting, July 2011 Page 3

Recent Activities n n n NIST met with each accredited VSTL to discuss possible integration of core functionality tests Recent activities regarding changes to VVSG 1. 1 will require changes to the NIST developed test suites; delayed status of VVSG 2. 0 warrants additional focus on improving testing for VVSG 1. 1, including existing methods for VVSG 1. 0 NIST and EAC met with both accredited VSTL’s to discuss ways to improve testing for the evolving VVSG 1. 1 and to integrate NIST-developed tests into the certification process TGDC Meeting, July 2011 Page 4

Preliminary Findings n n To ensure consistency across testing laboratories, agreed upon test methods must be developed for all of VVSG 1. 1 Face-to-face meetings revealed differences in interpretation of the term “readiness test”. Labs read as “acceptance test” whereas NIST definition was a “simple health check” Some NIST tests require new election definitions, which are problematic for the labs Labs define test data separately from the test case; NIST defines them together TGDC Meeting, July 2011 Page 5

Additional Findings n n Some NIST tests should be reworked to take into account practical test methods, with temperature and humidity cited as an example Source code analysis represents a large percentage of the cost of testing; tools remain difficult to use Reference ballots, including guidance on how ballots should be marked (pre-marked, hand-marked), election definitions/voting patterns to be used, and complexity of precincts/ballot styles, would be useful Should identify priority areas outside of VVSG 1. 1 TGDC Meeting, July 2011 Page 6

Action Plan n Monthly meetings among EAC, NIST, VSTL’s, with biannual face-to-face meetings: n n Focus on developing common test methods for new VVSG 1. 1, starting with information from EAC matrix, NIST test suites, and existing VSTL definitions Validate NIST test suites and integrate into certification process Expand effort on source code analysis tools and work with the VSTL’s to develop rules for VVSG 1. 1 Explore creating a set of reference ballots to be used by all labs TGDC Meeting, July 2011 Page 7

Discussion/Questions TGDC Meeting, July 2011 Page 8