-  Ben Bederson
     Paul Herrnson
     July 1, 2003

This report addresses the "Debugging Maryland Balloting" Op Ed piece that appeared in the Washington Post on May 12, 2002 and the "Usability Review of the Diebold DRE System for the Four Counties in the State of Maryland", both of which were authored by Professors Paul S. Herrnson and Benjamin B. Bederson of the University of Maryland College Park.

The major concern of the Maryland State Board of Elections (SBE) is that these professors did not test the machines that the State is putting into the hands of the voters. The machines used by the University of Maryland were actually still under the contractor's ownership and control. The machines used for testing had not yet gone through SBE's rigorous acceptance testing process. In addition, the demonstration ballot had been created by the contractor, was not sanctioned by SBE, and obviously did not meet SBE ballot standards.

A little context is needed here.  We (at UMD) were requested by the SBE to review the Diebold DRE machines.  While it is true that at the time of testing, they were still under the contractor's ownership, these were the machines that had already been purchased and delivered.  They were under a 30 day period of "acceptance testing" when we tested them, so while legally they may have been owned by the contractor, they were certainly very close to being owned by SBE, and were in fact the same machines that SBE deployed to voters.  Of course these machines hadn't gone through the SBE's acceptance testing process yet because our testing was part of that process. 

The SBE's claim that the demonstration ballot had been created by the contractor may be true, but it is irrelevant.  That is the ballot they gave us to test, and is the ballot they used to demonstrate the machines to members of the Maryland General Assembly and Maryland voters.  We asked several times for ballots that were more representative of the kind of ballots the SBE would actually use, but we were told that was not possible.

This report cites the six inaccuracies that exist in the op ed piece, in the order in which they were reported. The underlined parts are what are considered incorrect.  [Note: we do not have access to the formatted rebuttal with underlines - BB/PH.]

1: "The good news is that Maryland took action to avoid a disaster such as Florida had in 2000. At the legislature's direction, the state and the four counties with the oldest technology-Montgomery (it had the same punch cards that proved a disaster in Florida), Prince Georges's Allegany and Dorchester-chipped in to buy new machines."

SBE Response: Montgomery County did not use the same punch card system as Florida. They used the Datavote system, not a Votomatic-type card system that was used in Florida.

We stand corrected and apologize for the mistake (which ocurred during final copy-editing after we wrote the report correctly.)

2: "The system needs some reprogramming; it currently uses inconsistent terminology that could confuse voters."

SBE Response: The State Board of Elections has developed and established standard ballot terminology. SBE has instituted these standards on all ballots used on the new voting system. SBE does not have any information on the ballot used in the University of Maryland "Usability Review".

The ballot used by UMD was the one given to us by SBE, so the claim of no knowledge is unrealistic.  Our primary concerns were about the language in the Help section and in screen navigation.  These screens were not changed at all between the time we tested it, and when they were deployed in November, 2002.

3: "Help instructions are unclear and are not available throughout the voting process. A help button should be added."

SBE Response: The ballot standards used on the new voting system include comprehensive text and audio instructions. A function that will provide the ability for voters to retrieve instructions during any point in the voting process is not available in the current software version. SBE has requested that Diebold Elections Systems provide this function in a future software upgrade.

The specific language of the Help section was not updated between the time we tested it, and when it was deployed in November, 2002.

4: "No warning is given for over voting, and it is unclear how voters are to make a change once they have selected a candidate."

SBE Response: The ballot standards issued by SBE do not permit over voting on the new voting system. Preventing over voting is an option in the software. SBE does not have any information on the programming of the ballot configuration used in the University of Maryland review.

We mistakenly wrote "over voting" when we meant "under voting".  No warning if some, but not all the positions in a particular race are voted for.

We maintain that changing a selection once it is made is unnecessarily difficult.  The selection is performed with checkboxes, and if you decide to change your vote by pressing on another candidate, nothing happens.  The candidate does not change, and there is no warning.  Instead, the voter must first press again on the candidate they had already voted for to "uncheck" that checkbox, and then vote for a new candidate.  While this is standard in computer interfaces, this may be unfamiliar to voters not experienced with modern computer interfaces, and so we recommend feedback to the user that helps them with this process.

5: "The audio version designed for the disabled is confusing and hard to navigate."

SBE Response: Audio ballots are developed from the text-based ballots. If the text version of the ballot that was used was unclear, then it is likely that the audio version would also be unclear and confusing. Again, because SBE did not develop nor approve of the ballot that was used, SBE cannot respond further to this issue. Furthermore, the Diebold election system is the system preferred by the National Federation of the Blind (NFB).

Again, the SBE may not have approved of this ballot, but that is what they gave us to test and to introduce the machines to Maryland voters.

6: "On Election Day, in real polling places with a more representative range of voters, the problems and concerns could be even greater. That's why officials need to start now to launch a major voter education campaign in the four counties."

SBE Response: Pursuant to the State Election Law and the SBE regulations, SBE mandated in the December 2001 contract that Diebold provide an extensive voter outreach program.

As of May 14, 2002, approximately 10,000 voters have used the new voting system in a demonstration environment. In addition 56,320 students used the new voting system in a Montgomery County high school election in late April. The new voting equipment has been used in five successful elections in Prince George's and Montgomery Counties since early March. Allegany County has held demonstrations at one municipal election polling location and is scheduled to conduct demonstrations during four additional municipal elections. Dorchester County is demonstrating the voting units at schools, churches, and firehouses. A mock election is scheduled in Dorchester County on June 8, 2002.

In addition, 30 demonstrations are currently scheduled through mid-June in Montgomery and Prince George's Counties; with additional demonstration sessions added each week. Please refer to SBE web site, www.mdvotes.org, for up-to-date demonstration schedules.

This report comments on the University of Maryland review, "Usability Review of the Diebold DRE system for Four Counties in the State of Maryland."

The review cites problems mostly with the ballot layout that SBE has addressed through ballot layout standards. Additional observations of the review cite the following specific issues:

1.Inserting card was very confusing

SBE Response: Inserting the voter access card is not confusing with proper signage on the voting equipment. Voter education anecdotal evidence from actual elections indicates otherwise.

Our personal experience voting with these machines as deployed by SBE in November, 2002 was that the polling place workers inserted the card for us.  When asked why, they responded that many people had trouble with the cards, so they were helping out.

2. Concerns about reliability

SBE Response: Pursuant to SBE regulations, a full test of each unit must be conducted prior to acceptance and use.

3. Colors are not well chosen

SBE Response: SBE has established ballot standards that include colors that are easy to read and comply with the Rehabilitation Act Amendments of 1998 (Section 508, §1194.26).

The color selection overall, and particularly on the Help screen where it was most problematic, did not change at all between the time we reviewed the machines, and when they were deployed in November, 2002.

4. Font size could be bigger

SBE Response: Increased font size is available upon request of the voter. SBE can assume that the testing participants did not have a working knowledge of the equipment.

There was no indication to the voter that they could request a larger font size.  We expect that most voters would also have not known that they could request a larger font size, and so this feature, while nice, was likely not used to full advantage.

5. Layout of the ballot was confusing

SBE Response: SBE has developed a simple, straightforward ballot standard that is receiving widespread, positive reactions from voters that have used the equipment in elections or demonstration environments.

May 21, 2002