VVSG 1 1 Test Suite Status Mary Brady

VVSG 1. 1 Test Suite Status Mary Brady National Institute of Standards and Technology http: //vote. nist. gov TGDC Meeting, Jan 2011

Background n n n Status quo: Labs have been testing to VVSG 1. 0 (2005) using proprietary, custom tooling and review processes In 2007 -08, NIST developed a set of public test suites for VVSG 2. 0, to be used as part of the EAC Testing and Certification Program In 2009, to support VVSG 1. 1, test methods for new and changed material was back-ported from 2. 0 test suites; status quo prevails for everything else. TGDC Meeting, Jan 2011 Page 2

Why Public Test Suites? n n To achieve consistency across testing labs and promote transparency of the testing process To review the VVSG for ambiguities, completeness, and correctness To assist manufacturers by providing precise test specifications To assist testing labs by lowering the overall cost of testing TGDC Meeting, Jan 2011 Page 3

Test Development Timeline • Internal • Voting Team VVSG 2. 0 Tests Summer 2008 • Expert • VSTL, Selected Experts VVSG 2. 0 Tests 2007 -2008 • General Public • Posted on Web Site VVSG 2. 0 Tests Fall 2008 VVSG 1. 1 Tests 2009 • Subset of VVSG 2. 0 Test Suites • EAC Comments • Nomenclature • Traceability • Expert, Public Review VVSG 1. 1 Tests June 2010 VVSG 1. 1 Tests Spring 2011 • VSTL Review • Work within VSTL’s • Procedural Validation VVSG 1. 1 Tests October 2011 December 2011: Integrated into Certification Process TGDC Meeting, Jan 2011 Page 4

VVSG 1. 1 Test Suite n VVSG 1. 1 test suite is based on VVSG 2. 0 test methods associated with back-ported requirements n n n Accessibility and usability Operational temperature and humidity Electronic records, security specifications, and VVPAT Core functionality, reliability and accuracy New test method developed for updated software setup validation requirement TGDC Meeting, Jan 2011 Page 5

Accessibility & Usability n n System-independent test narratives with pass/fail criteria Highly structured process surrounding the usability test protocols for the performance-based testing with test participants ISO Common Industry Format for reporting usability test results CIF templates and how-to’s for manufacturers and test labs TGDC Meeting, Jan 2011 Page

Accidental Activation: Input mechanisms SHALL be designed to minimize accidental activation EXAMPLE TEST CASE Covers requirements: 3. 2. 6 c Accidental Activation 3. 2. 6 c. i Size and Separation of Touch Areas 3. 2. 6 c. ii No Repeating Keys 3. 2. 6 c Input mechanisms SHALL be designed to minimize accidental activation 3. 2. 6. c. i On touch screens, the sensitive touch areas SHALL have a minimum height of 0. 5 inches and minimum width of 0. 7 inches. The vertical distance between the centers of adjacent areas SHALL be at least 0. 6 inches, and the horizontal distance at least 0. 8 inches. 3. 2. 6. 2. ii No key or control on a voting system SHALL have a repetitive effect as a result of being held in its active position. TGDC Meeting, Jan 2011 Page 7

Accidental Activation, cont. EXAMPLE TEST CASE • Test Method includes 7 test requirements, covering 2 pages • Excerpt: For touchscreen systems, The tester shall examine the touch areas for at least contests #4 (Governor) and #9 (County Commissioners). Using a ruler to measure distance and a stylus to perform the touching, the tester shall determine first that the touch areas used to vote for at least the first t candidates in each contest are separated as required. F => If any vertical distance between centers of adjacent touch areas for voting is less than 0. 6 inches, then, for requirement "Size and Separation of Touch Areas", the system fails. TGDC Meeting, Jan 2011 Page 8

Hardware n Operational Temperature and humidity TGDC Meeting, Jan 2011 Page

Operating Temperature and Humidity Covers requirements: EXAMPLE TEST CASE Volume I, Section 4. 1. 2. 13 Environmental Control - Operating Environment. Voting systems shall be capable of operation in temperatures ranging from 41 °F to 104 °F (5 °C to 40 °C) and relative humidity from 5% to 85%, non-condensing. For testing information, see Volume II, section 4. 7. 1. Volume II, Section 4. 7. 1 Operating Temperature and Humidity Tests. All voting systems shall be tested in accordance with the appropriate procedures of MIL-STD-810 D, "Environmental Test Methods and Engineering Guidelines''. TGDC Meeting, Jan 2011 Page 10

Operating Temperature and Humidity Covers requirements, cont. : EXAMPLE TEST CASE Operating Temperature All voting systems shall be tested according to the low temperature and high temperature testing specified by MIL-STD-810 -D: Method 502. 2, Procedure II – Operation and Method 501. 2, Procedure II – Operation, with test conditions that simulate system operation. Operating Humidity All voting systems shall be tested according to the humidity testing specified by MIL-STD-810 -D: Method 507. 2, Procedure II – Natural (Hot–Humid), with test conditions that simulate system operation. TGDC Meeting, Jan 2011 Page 11

Operating Temperature and Humidity, cont. EXAMPLE TEST CASE Test method includes 15 steps, covering 3 pages Excerpt: Step 8: Set the chamber to 104 degrees Fahrenheit and 85% relative humidity (see Comment 1), observing precautions against thermal shock and condensation (see Comment 2). Allow relative humidity and VSUT temperature to stabilize. All paper, including ballots, used by the system must be stabilized at the specified testing temperature and humidity levels prior to testing (see Comment 4). Step 9: Perform an operational status check. If the VSUT shows evidence of damage, or any examined function or feature is not working correctly, then record that the VSUT fails the Operating Temperature and Humidity test. End the test. TGDC Meeting, Jan 2011 Page 12

Security n n Electronic records Security specifications VVPAT New test method developed for updated software setup validation requirement TGDC Meeting, Jan 2011 Page

Electronic and Paper Record Structure EXAMPLE TEST CASE Covers Requirement 7. 9. 3 c Electronic ballot images shall be digitally signed by the voting system. The digital signature shall be generated using a NIST-approved digital signature algorithm with a security strength of at least 112 bits implemented within a FIPS 140 -2 validated cryptographic module operating in FIPS mode. Discussion: NIST approved is "An algorithm or technique that meets at least one of the following: 1) is specified in a FIPS or NIST Recommendation, 2) is adopted in a FIPS or NIST Recommendation or 3) is specified in a list of NIST approved security functions (e. g. , specified as approved in the annexes of FIPS 140 -2/3)". The security strengths of cryptographic algorithms can be found in NIST Special Publication 80057: Recommendation for Key Management - Part 1 General. TGDC Meeting, Jan 2011 Page 14

Procedure EXAMPLE TEST CASE Step 1: Obtain five electronic ballot images from the VSUT. Step 2: Verify digital signature on each of the ballot images individually. Step 3: If any of the digital signature verifications fails, record “The VSUT fails the Cryptographic Protection of Records test. ” End the test. Step 4: Execute the Sections 6. 1. 3 and 6. 2. 3 cryptographic tests for the digital signature cryptographic module used to sign the electronic ballot images: Step 5: If any one of the above tests fails, record “The VSUT fails the Cryptographic Protection of Records test. ” End the test. Step 6: Record “The VSUT passes the Cryptographic Protection of Records test. ” End the test. TGDC Meeting, Jan 2011 Page 15

Core Functionality n Votetest n Basic, essential voting system logic n n n 92 tests formalized as SQL scripts Tests are intentionally simple… n n n Ability to define elections Capture, count, and report votes Voting variations 89 use about 10 ballots, 3 use 100 ballots A volume test (mock election) is a significant test of all supported functions together …but they exercise the complete elections and voting process TGDC Meeting, Jan 2011 Page

Printed Report: Counted report of contest, including votes, undervotes, overvotes EXAMPLE TEST CASE Covers Requirement I. 2. 4. 3. d: All systems shall provide capabilities to produce a consolidated printed report of the results for each contest of all votes cast (including the count of ballots from other sources supported by the system as specified by the manufacturer) that includes the votes cast for each selection, the count of undervotes, and the count of overvotes. TGDC Meeting, Jan 2011 Page 17

Printed Report, cont. : Counted report of contest, including votes, undervotes, overvotes EXAMPLE TEST CASE General Procedure 1. Establish initial state (clean out data from previous tests, verify resident software/firmware); 2. Program election and prepare ballots and/or ballot styles; 3. Generate pre-election audit reports; 4. Configure voting devices; 5. Run system readiness tests; 6. Generate system readiness audit reports; 7. Precinct count only: 1. Open poll; 2. Run precinct count test ballots; and 3. Close poll. 8. Run central count test ballots (central count / absentee ballots only); 9. Generate in-process audit reports; 10. Generate data reports for the specified reporting contexts; 11. Inspect ballot counters; and 12. Inspect reports. TGDC Meeting, Jan 2011 Page 18

Printed report, cont. • Test Method includes 38 steps EXAMPLE TEST CASE • Excerpt: Step 26: Compute the absolute value of the difference between the reported number of votes for “Car Tay Fower” in the “President, vote for at most 1” contest in Precinct 1 and the value 4, and add it to the Report Error. If the needed value does not appear in the report, increment Report Error by one (1). . Step 34: For each spurious ballot count or vote total reported by the VSUT (e. g. , ascribing votes to a candidate that did not run in a particular contest or reporting one or more overvotes on a VSUT that prevents overvoting), increase the Report Error by one (1). Step 35: Record the Report Error. TGDC Meeting, Jan 2011 Page 19

Reliability, accuracy, misfeed rate n n Improved test method replaces material that was historically included in the VSS/VVSG… hence, included in drafts Now evaluated using data collected during all tests, rather than a single, isolated test TGDC Meeting, Jan 2011 Page

Test Validation n Rigorous traceability to VVSG requirements n Reviewed by independent parties n n n VSTL labs, experts, public EAC: updated to consistent nomenclature and traceability Procedural (on-going) n n Operating temperature and humidity – complete Other VVSG 1. 1 test suite components are under consideration and will be conducted in 2011 TGDC Meeting, Jan 2011 Page 21

Next Steps n n Continue procedural validation Round-trip with testing laboratories to discuss methods for integrating test methods into their workflow Success here will pave the way for the rest of the VVSG 2. 0 test suites Continue to work with all to improve the VVSG, manufacturer implementations, testing practices, and the test suites TGDC Meeting, Jan 2011 Page 22

Discussion TGDC Meeting, Jan 2011 Page 23