Session 8 Best State Practices Best Practices Quality

Session 8 – Best State Practices “Best Practices, Quality Systems, Customer Requirements and Testing as Applied to Voting” Jim Dolbear Director, Voting Systems Institute dolbear@votinginstitute. org 310. 899. 3800 ext. 163 9343 Culver Blvd. Culver City, CA 90232 1

Examples of Best Practices for Voting • Quality systems and product design from industry. – The ITA’s bring some of this background to voting. • Auditing procedure and reporting culture from FASB and corporate accounting. • Technical development models from academia and science. – NIST brings this experience to voting. • Inclusive, non-governmental standards setting from the internet and IETF (the “Internet Engineering Task Force”). • Performance testing and review from the automotive industry. 2

Defining Customer Needs or Requirements/Defining the Performance Objectives for a System • House of Quality concept. From the U. S. to Japan and back again. – Deming, Crosby and Nash. – Toyota Production System. Taichii Ohno and Shigeo Shingo. – Now they talk about “Six Sigma” projects. • Focus on the customer. • From customer needs to technical requirements. – An example from the car industry. – Defining requirements and then testing for voting systems is similar. 3

Defining and Achieving Customer Requirements Customer Needs (in English) Customer Requirements (technically defined) Product Requirements (technically defined) Process Requirements "Safe Car" 5 -star crash rating certain gauge steel in the front frame welding steel @ correct temperature Performance Testing of the System: Follow the National Highway Traffic Safety Administration's procedure on administering crash test. 4

Requirements, Specifications, Testing • What are we trying to test in voting? Talking about testing in a vacuum is putting the cart before the horse. – Need to know how we would want a voting system to perform before we can test it to see if it passes muster. • First, requirements tell us how we need a product, process or system to perform functionally. – Sometimes requirements must reflect practical compromises and don’t include everything we desire because of technical and resource constraints. – Minivans only go so fast. A requirement might be 0 -60 acceleration in 8. 5 seconds. A desire would be 0 -60 in 6. 5. • • 5 Specifications are typically technical definitions of materials, procedures, and structures that must be used to achieve requirements. Tests (and inspection) are done to evaluate whether the requirements are in fact being met by the system.

Testing and Product Design and Customer Requirements – a Virtuous Circle of Improvement • National Highway Traffic Safety Administration develops test and begins to test and report crash results. • Vendors focus on achieving better results. • Customers realize that safer cars can be made and begin to value safety in their purchase decisions. • Safety improvements over a 30 -year period result in much safer cars, higher profits for manufacturers of safe cars and pleased, safe drivers as customers. 6

Voting Product Design: User Interface Requirement and Testing • Define an objective test to measure good and bad performance. – Say 100 people interpreting 10 different ballot styles. – Measure their comprehension and error rate. • Apply the test to different systems -- paper-based and touchscreen. • Provide the basis for scientifically evaluating the performance of paper. • Report the results to support a choice based on cost and performance. 7

Product Design: User Interface Requirement and Testing 8

Absolute Requirements and Requirements which must be Traded off 9 • • Designing a Car Example/Absolute: Seatbelts are required by law. Example/Tradeoff: zippy acceleration vs. seating for seven • • Designing a Voting System Absolute Voting Requirement: Providing access for the blind. Absolute Voting Requirement: State ballot style laws. • Tradeoff: Exposure to vulnerability of computers vs. ease of automated tabulation and ballot creation • • Tradeoff: Auditing protocols vs. operational complexity Tradeoff: sometimes privacy vs. security

Importance of Formally Defining Requirements • A “Quality System” provides for the formal “archival” definition and documentation of customer requirements and corresponding technical product and process requirements/specifications. • It’s particularly important to document the dependency relationships between different, conflicting requirements. – Let’s get rid of this heavy bumper and crumple zone to make the car go faster. – Let’s use computers for voting so different ballot formats and tabulation will be much easier. • State code typically defines some or many of the customer requirements for voting. The testing protocols should provide for technical evaluation of achievement of these requirements. 10

The Future: Encouraging New Technological Developments • Meet absolute requirements more effectively at lower cost. • Reduce or eliminate the need for tradeoffs. • Lay out a plan for participating with development partners and encouraging innovation. • Example/Aluminum Crumple Zones: Lighter and Stronger! • Encrypted Receipts: Improved privacy and security? • Open Source Voting Systems: Cheaper and More Reliable? 11

Technical Design Improvements Customer Needs (in English) Customer Requirements (technically defined) "Safe Car" 5 -star crash rating certain gauge steel in the front frame 5 -star crash rating crumple zone with a cage of lighter steel 5 -star crash rating crumple zone with a cage of aluminum "Safe Car" Product Requirements (technically defined) Process Requirements welding steel @ correct temperature Performance or Evaluative Testing of the System: Follow the National Highway Traffic Safety Administration's procedure on administering crash tes 12

Performance Specifications or Requirements vs. Design Specifications • • Voting systems should perform at a certain level for each critical customer requirement. Testing this performance objectively is the aim. States should not specify or implicitly require particular designs. – 5 -star crash test does not test the gauge of the steel in the frame. – If a crumple zone gets 5 stars with lighter steel; it’s still 5 stars. • Voting System Performance Rating (www. vspr. org) – Bring together all stakeholders -- election officials and experts, vendors, computer and security professionals and social scientists and advocates. – Define performance attributes for voting systems. – Develop and define objective tests to evaluate performance of systems. • How secure • How private • How transparent 13

State Best Practices - Process • Publish and meet all absolute requirements for voting systems as laid out in Federal and State law. • Involve all stakeholders and make responsible, informed tradeoffs where a choice must be made between performance objectives. • Over-communicate the rationale for choosing particular solutions particularly where there are tradeoffs. – Why punchcards? Low cost. Secure. Auditable. – Why computerized voting? Access. Ballot creation. Tabulation. • Actively participate in and sometimes lead the development of higher performing/cheaper solutions. – Early voting, provisional voting and absentee voting as test beds. – Work with counties to develop, nurture and learn from different approaches. 14