Monday, December 18, 2017

Dictatorship: the ghost that haunts re-election in Honduras

The tweet from @Codigo504 is the kind of mordant humor I think of as typically Honduran:
Después del informe de OEA y el tuitazo de Almagro los cachurecos deben calcular bien sus próximas acciones. ?

After OAS’ report and Almagro’s tweet, the Cachurecos need to think very carefully about their next steps ?

"What Would Carias Do?"

That's easy: hold on to power however he could. Tiburcio Carías Andino is the ghost hovering over presidential re-election in Honduras.

While the rhetoric used to justify the 2009 made re-election especially potent as a current political issue, the constitutional ban on re-election is not a long-established Honduran practice. It was inserted in the 1982 constitution that Oscar Arias famously called the "worst in the entire world" during his failed attempt to negotiate an end to the 2009 crisis.

The 1982 constitution was enacted as part of the exit from a long period of military rule, guided by the relationship of Honduras and the United States. One of the new features was the declaration in the Constitution that certain provisions could not be amended, including the prohibition on re-election and the definition of the term of the presidency as four years. These features have been described as "centerpieces" of the new 1982 constitution.

Why such an insistence that no future president would serve more than four years? It might seem at first that this was intended to forestall the kind of military dictatorships that had dominated Honduras from 1963 to 1982 (with a brief hiatus for an elected government lasting just over a year from 1971-1972). But that is too short a time frame to understand this provision.

Tiburcio Carías Andino is the ghost hanging over the understanding of the potential for a Honduran President to exploit electoral laws to hang on to power indefinitely. The Honduran people see Juan Orlando Hernández as seeking to follow the path of Carías, not of Policarpo Paz.

Carías Andino first took supreme executive control of Honduras in 1924, during a period of substantial political conflict. In 1932, he ran for election and started an unprecedented period of 16 years in that office. The constitution in force at the time prohibited consecutive terms as President, so Carías Andino initiated the writing of a new constitution. This allowed him to stay in office, and consolidate executive control.

Carías suppressed political opposition. His power ended in part due to protests that began in the capital city and in San Pedro Sula. In the latter case, the brutal suppression of the protests shaped a generation of political activists.

In the aftermath of his presidency, Honduras experienced a significant turmoil, starting with a presidential term by Carías hand-picked successor, Juan Manuel Gálvez. Toward the end of his term in office, a major strike against the dominant banana industry transformed the country, showing the power of labor.

The 1954 election for president did not produce a candidate with the then-required majority vote. (A majority is no longer  required by the 1982 constitution, leading to the election in 2013 of a president who received less than 38% of the popular vote, and contributing to the crisis of 2017. This non-majority clause can be seen as another haunting from the age of Carías Andino.)

In 1954, the resolution of the election was messy: the legislature was supposed to vote to decide who would be president, requiring a two-thirds majority. Two parties boycotted the required sessions, preventing this resolution. The Supreme Court was supposed to resolve a legislative failure to decide, but it was perceived as illegitimate due to being packed by Carías Andino. (Legislative packing of the Supreme Court by Hernández facilitated the decision that opened the way to his seeking re-election, another piece of the Carías playbook that he emulated.)

In the void of power, the vice president, Julio Lozano Díaz, took over, suspending the congress and instituting the writing of a new constitution. His extra-legal regime lasted two years, ending with a military coup. The candidate from 1954 who had received the most votes, Ramón Villeda Morales, was elected to a six-year term in 1957.

Villeda Morales initiated modernizing policies including modest agrarian reforms. By the end of his term, these led to opposition from conservative sectors of Honduran society. Under the constitution then in force, Villeda Morales himself was limited to one six year term as president. His party's nominee was expected to win, however, and to continue his policies.

That prospect was enough to initiate a military coup. The seizure of government initiated the long period that only ended in 1982 with the ratification of the current constitution. Its provisions about presidential election are shaped by the history that began with Carías Andino: a single term for president, without re-election, and no requirement for a majority, a run-off election, or any mechanism for resolving elections too close to truly be called like the one that once threw election to the Congress and then the Supreme Court.

The US government in 1963, under the direction of President John F. Kennedy, cancelled aid, withdrew US military from Honduras, and called the Ambassador to Honduras back to the US. None of these actions led to the return of control to civilian government. General Oswaldo López Arrellano, who held power until 1971 and again from 1972 to 1975, eventually initiated new agrarian reforms, before falling out of power due to a bribery scandal involving the United Fruit Company.

His two successors, also military officers holding extra-judicial power, consolidated the ideology of the military as a stabilizing force that led to their institutionalization in the current constitution as the guarantors of democratic processes. That constitutional role was cited by the military as motivation for their actions in the 2009 coup d'etat.

Hernández has worked to make the army loyal to him. He has also invested, with substantial US aid, in the creation of new militarized police whose role in the 2013 election already was seen as promoting repression. The history of presidential manipulation of the armed forces, too, can be traced back to the Carías dictatorship that is providing so much of the model for the current president.

So indeed, the question #whatwouldCariasDo appears to be the one that we all should be asking as we watch to see what tactics the modern successor to the authoritarian who scarred Honduran political memory might adopt.

Sunday, December 17, 2017

Statistics and fraud

We now have two separate statistical analyses of Honduran presidential voting data, and both conclude the same thing: there was something fishy that happened in the middle of vote counting.

To recap: The Economist looked at the percentages of votes that went to the two main candidates before and after the long break by the TSE in posting vote counts.

Before the break, with about 57% of the vote counted, Salvador Nasralla had a 5% lead. After that, the lead steadily declined to the final apparent margin of just over 50,000 votes.

The Economist specifically asked whether the explanation offered by the Partido Nacional-- which claimed that the later votes came from more rural locations more likely to favor their candidate-- could account for the shift. They compared vote counts before and after the break in counting within each municipio.

Rather than being differences between urban and rural places, their analysis compared vote counts within each locality. They found an average shift of 3.8% within the same locality. It didn't matter if the municipio was rural or urban-- they all shifted the same way.

One known difference: votes tallied after the break included a large number that were not scanned and transmitted from the polling places on the day of the election. Instead, these were trucked up to Tegucigalpa and scanned there. Much of the discussion about vote counting has centered on the treatment of these vote tallies, including concerns about some arriving in open, unsecured packages, and the rumor that some were scanned in a hotel (and thus potentially could have had substitutes).

The Economist also drew attention to the unusually high voter turnout reported in the late-counted votes, in particular, from three largely rural departments. This, they note, could reflect a better get-out-the-vote operation-- or ballot box stuffing. Here it is worth remembering that the sign-off on vote tallies is done by credentialed party members, and there has been reported fraud and sale of credentials by smaller parties, in 2013 and 2017,

There matters stood until the release by the OAS today of a report by Georgetown University Professor Irfan Nooruddin. His analysis identifies a point when 68% of the votes were counted where, across different regions, both the turnout level and support for the Partido Nacional increases sharply. Either of these would be unusual; both are very unusual.

Nooruddin uses the reported data to do something that the actual vote counting never did: he simulates what vote counts would have looked like if results had posted randomly. This has been a key problem throughout the process: it is unclear what order the TSE used in its vote counting; it was not statistically random nor selected to be a representative sample. The OAS in its initial report noted that the TSE shifted from counting as votes came in to some unexplained selection process. Nooruddin helps us see if the election would have been less confusing if the voting tallies were counted randomly.

The conclusion of this part of the analysis is that if the votes were accurate, and were counted randomly, the pattern seen could have happened, and not result from tampering.

Nooruddin doesn't stop there-- as he notes, this part of the analysis is only worthwhile if the vote counts were accurate. He continues with tests of this assumption, and finds that the differences between early counted vote tallies and later ones "are large and suspicious".

Every department showed the same pattern of early lead for the Alianza followed by a change in pattern. As in the analysis by The Economist, the universality of this pattern is not easily explained by innocent factors. There is nothing about early vs. late vote tallies that would account for this.

It is as if there were two elections being counted, with precincts in every department changing the same way.

The only way we can imagine to have this result would be if for some reason the TSE did a preliminary sort of actas and deferred counting those most favorable to the National Party until last. Needless to say, that makes no sense.

Nooruddin points out that the shift in turnout in the later-counted tallies would be expected less than one in one thousand times-- statistically a significant difference. He presents an in-depth analysis of the Department of La Paz that shows that even in a department that favored Hernández throughout counting, and has a higher-than-average turnout rate, the later vote tallies increased in both reported turnout and voting for the Partido Nacional. The turnout increase is statistically likely to occur only one time in one thousand. Nooruddin concludes "such a sharp increase in turnout in the same department is unusual".

He writes that these findings are "consistent with a hypothesis of tampering with the vote tallies that were counted last".

So what could have happened?

One way to produce such an effect is good old fashioned ballot box stuffing-- reporting more votes than actually took place, and attributing the extra votes to a preferred candidate. Once the acta was signed, no one went back to double check the voter rolls or ballots. As long as the math on the tallies added up, you could have a voting pool in whatever form you like. This might well be correlated with places where votes weren't transmitted online the day of the election, as the ballot stuffing could happen at many points.

A lot of anxiety around these late vote tallies revolves around whether fake actas were substituted on the way to Tegucigalpa, or even fake images of actas in the TSE database. These, again, would work, and would not produce any contradiction unless the full ballot box was opened and recounted.

Both statistical analyses allow for the possibility this was just a really unusual way votes came in and were counted. But in Honduras, there is little trust in the system and unusual has already translated into illegitimate.

What happens next will determine whether Hondurans can begin to rebuild trust in democratic processes.

OAS calls for new elections in Honduras

Today witnessed a series of press conferences in the contested Honduran election.

Shortly after the OAS Mission said it would be making a statement late today, the Tribunal Supremo Electoral announced its own announcement would be made earlier in the day.

Not surprisingly, given previous statements, the TSE's announcement was their conclusion that the presidential election had been won by Juan Orlando Hernández, of the Partido Nacional. Neither the Partido Liberal nor the Alianza formed by two opposition parties, the Partido Anti-corrupción and LIBRE, have accepted the vote tallies posted by the TSE, alleging a number of different kinds of fraud.

There is also a potential legal issue left unaddressed: whether the candidacy of Hernández was entirely legal. The current president ran for an unprecedented second term under a Honduran constitution that prohibited even talk of re-election, until a Supreme Court he shaped while head of Congress ruled otherwise. The Supreme Court ruling opened the door to re-election. But lawmakers in Honduras did not pass any legislation authorizing re-election. Technically, then, this is not just an unprecedented election outcome: it is one that took place outside any defined legal framework.

Both the European Union and the Organization of American States are on record as seeing the electoral process as problematic. While the EU released a statement today that many read as supporting the TSE's conclusion, the OAS today signaled more reservations, beginning with statements by Secretary General Luis Almagro on Twitter.

These were expanded in the OAS announcement this evening that the Secretary General of the OAS cannot provide certainty about the results of the election. The press release reiterates previous descriptions of the electoral process as "characterized by irregularities and deficiencies" and of "very low technical quality" and "lacking integrity".

The press release continues:
in the face of the impossibility of determining a winner, the only road possible for the winner to be the Honduran people is a new called to general elections, within the strictest respect for the rule of law, with  guarantees of a TSE that would enjoy the technical capacity and the confidence of the citizenry and the political parties.

This is followed by the appointment of a commission from the OAS of ex-presidents Jorge Quiroga and Alvaro Colom to "carry out the necessary work for a new electoral process and national democratic reconciliation in Honduras".

The full basis for this position is contained in the OAS mission's report to the Secretary General. It rehearses all the weaknesses in the electoral process. It calls allowing a run for re-election based on a court finding (without implementing legislation in place) a "bad practice...that revived the polarization generated by the coup and political crisis of 2009".

The OAS report also provides a new statistical analysis by Professor Irfan Nooruddin of Georgetown University addressing whether the sharp change in voting patterns noted after a break in counting could be explained in any innocent way.

This retraces some of the terrain covered by an analysis in The Economist that concluded that the shifts in voting seen were very unlikely.

Professor Nooruddin uses additional techniques, and concludes "on the basis of this analysis, I would reject the proposition that the National Party won the election

We will revisit these statistical analyses tomorrow, explaining what they do (and do not) show, and relate those observations to some of the known problems in the conduct of Honduran elections in general, and this one in particular.

For now, though, the question is: will Juan Orlando Hernández accept the OAS recommendation? Or does he think he can ignore the massive resistance to his re-election that has already led to almost two dozen deaths of protesters, and the closure of roads across the country?

Saturday, December 16, 2017

Processing an Acta: Rules and Procedures

Amidst allegations of voting fraud, the Honduran Tribunal Supremo Electoral does itself no favors with its incapacity to explain what it does. For many outside observers, it may be worth reiterating that the TSE does not directly count ballots; even when voting irregularities are charged, they mainly return to and re-examine the summaries of votes at each polling place, or MER.

Even those of us who have been following contested TSE procedures through the last three electoral cycles can get confused about how the TSE processes these vote tally sheets (acta in Spanish). Some confusion about how the political parties obtain actas has been evident in blog posts and other coverage. Although what follows is dense, it is an attempt to make this more transparent.

There are published rules governing how actas are supposed to be generated and transmitted to the TSE. The rules are contained in a document issued by the Tribunal Supremo Electoral on November 21 and published in La Gaceta of November 24, 2017-- just two days before the election. They are titled "Reglamento del Sistema Integrado de Escrutinio y Divulgacion Electoral (SIEDE)" and describe both the hardware and software environment for the processing of vote tally sheets for the three elections held on November 26, 2017.

Here's how it was supposed to work:

First, there is a physical space in a voting center where there are two different kinds of "digitization kits". This is the ATX, the "area of transmission (area de transmisión)". 

It contains a tablet kit, consisting of a tablet, a multifunction printer/scanner, and a GSM (cell phone) modem. It also contains the Operador de Mesa Receptora (OMR) kit, consisting of a tablet, multifunction printer/scanner, and 2 GSM modems, one for TIGO and one for CLARO, the two major cell phone providers in Honduras. 

Each OMR kit serves up to five MER (polling places). The modems are supposed to be connected to a Virtual Private Network (VPN) over the cell phone provider's data network, terminating in the TSE's computer center.

Each OMR kit is operated by a custodian who is designated and credentialed by the TSE.

On the day of the election, the TSE is supposed to reset its database, and all counts, to zero at 6 am and generate and sign a document that this occurred.

At 7am, the digitization kits are set up in the voting centers.  At that time the security envelope containing the login information for that specific kit is opened, and the operator logs in over the VPN to the TSE system to receive encryption keys, security certificates, digital signatures for each MER that that operator will support, and only for those MER. 

The operator then generates an "hoja de prueba" that is supposed to show that there are no images of actas stored on the tablet, print it, and sign it, then scan it and send it to the TSE.  This process is supposed to shut down the OMR kit, so that it cannot be used until after the voting center closes at 4 pm

At 4 PM, when the voting center closes, the software controls the transmission of the voting tally sheets. Voting tally sheets (actas) are generated at each MER for each of the three levels of election, in this order:  Presidential, Congressional, and then Municipal.  Actas are signed by representatives of each political parties present at the MER.

Then the President of the MER, along with any members who want to join in, take the vote tally sheets physically to the ATX area inside the voting center.

Once the acta is at the ATX, it is the responsibility of the ATX custodian to wake up the equipment, log in using the TSE supplied credentials, verify the ATX information (department, municipality, voting center, identification number of the ATX, number of the MER).

We can assume that all of this information is contained in the JSON information transmitted to the TSE.  We can also guess that this error-prone manual process is responsible for the actas in the TSE system today that have images of the tally sheet for a particular MER, but are filed within the system as if they are the tally for a different MER. This is acknowledging that there is the potential for operator error, which the system is supposed to have safeguards to prevent.  Once the information is entered into the tablet, the custodian is supposed to make sure the whole system is working (the procedure to do this is not specified).

Then the OMR custodian scans the actas in the ATX that serves the particular group of MERs.

The software works off a QR code on the acta and verifies it is for a MER assigned to this ATX and OMR kit.  Once scanned, the system displays the scan on the tablet for the custodian to verify the quality of the scan, that the information is legible, and that it is correctly scanned with no missing or obscured information.  If it is OK, they click a button on the screen to transmit it.  If its not OK, they click another button on the screen to rescan the acta.

Transmission occurs between the ATX and a receiving server in the TSE computing center, where it is then replicated to the servers of each of the political parties.

The political parties are responsible for installing a fiber optic network between their server and the TSE network. Each acta replicated is encrypted with a digital signature that guarantees its authenticity, as transmitted by the ATX.  The political parties and the TSE verify the digital signature of the acta to validate it.

There is a second check on poll tallies provided for the political parties. Back at the OMR, once an acta is transmitted up to the TSE, the custodian prints enough copies of the acta to give to each party's representative on the MER and stamps of the back of each one a rubber stamp that says it conforms to the original and is signed by the secretary of the MER.  This process is repeated for each of the OMR kit's MER for the actas for the Presidential, Congressional, and Municipal elections.

Obviously, if there isn't a party representative at a specific MER, this copy won't be received by the party. In general, when the parties cite their actas, they mean the ones transmitted by the TSE, but they may also have the paper copies.

Once transmission concludes for all actas, the custodian prints a receipt for his/her service as custodian and transmits the log files to the TSE.

Now here's one place where what happens introduces the fear of manipulation: all the actas scanned at the ATX centers are supposed to be scanned a second time in Tegucigalpa when the physical package of electoral materials (maletas in Spanish) arrives. The OAS report noted that some of these arrived without security, already open. Pictures of a truck backing up to a hotel in Tegucigalpa that appeared to show such packages raised the concern about some actas possibly being scanned outside the INFOP facilities. In both situations, there is concern that a substitute acta could have been inserted in place of the one scanned on election day at the ATX center.

The published rules make clear that at INFOP, as the documentation physically arrives, the actas are taken out and scanned a second time, and that those scans go into the TSE computers and are replicated to the party servers.

The scans produced in Tegucigalpa replace original scans transmitted from ATX centers. These scans are clearly done using different procedures with a different way of getting in to the system. INFOP does not use the ATX software. There are no documented security protocols to provide for the authenticity of the INFOP scans in the rules as printed by the government.

We presume that the scanner and software in the INFOP center is different than that used in the ATX centers. The images from scanning at the INFOP center (1) lack the time and date stamp at the top, and (2) don't clearly show the security tape applied to the acta to prevent alteration.

It is notable that the otherwise very specific rules from the TSE do a bunch of hand waving rather than documenting the scanning protocol at the INFOP warehouse. It is only by reading between the lines that we can infer that these scans replace the ATX transmitted scans in the TSE system.  A proper software/procedural audit would have questioned why there were no protocols described for this process, but the TSE didn't ask its audit firm, Audisis, for a pre-election audit.

What the published rules make clear is that each political party can receive both a physical certified copy of each acta from its representative on the MER, and a digitally transmitted, encrypted acta image from the ATX, replicated from the TSE receiving server. 

Each party also receives a scan of the acta made in the INFOP warehouse as each election package physically arrives back at the TSE warehouse and is opened and scanned. 

At no point does the TSE compare each of the scanned images with the paper original and the votes recorded in its computers to validate the results of the election. That simple procedure would detect some kinds of fraud that are suspected or rumored.

Friday, December 15, 2017

More Audit Irregularities

Thanks to the incredible people over at the Voto Social we now have another anomaly related to the Actas, that the timestamps added by the ATX system when it scanned the actas have been systematically removed from the images of all but 3,737 actas in the TSE system.

Just to refresh your understanding of this part of the voting hardware and software, each MER submits a vote tally sheet, an acta, as a paper form to be scanned and transmitted to the TSE.  The scanning software adds a time stamp in the upper left to the images of actas before encoding the image, its MER number, and other data into a JSON post to the TSE main computer system in Tegucigalpa.  Here's a sample, as originally downloaded by the Voto Social website from the TSE archive:
Note the small text above the TSE and 2017.

Here's how the same acta appears in the TSE system today:
Suddenly the ATX scanning software added text on the acta image is missing.  There is no conceivable reason for anyone to edit the image to remove that data, except that it provides a way to precisely audit when the images were added to the TSE database.  So someone removed information that could have been used to audit the addition and counting of actas in the TSE system.

In addition, the ATX scanners provided better quality scans so that you could actually see the security measures on each acta, such as the special tape that was put on the acta over the vote tallies to prevent alteration.  You can see the left edge of the tape on the top image, but its almost undetectable on the bottom one.  They noted that on some of the original scans you could even read the security watermark in the tape, while on the scans in the TSE system, it is invisible.  You can see the full images from which these details were extracted on the Voto Social website here.

They take all of the above as evidence that the TSE for some reason rescanned or edited actas that were transmitted on a non ATX scanner and entered them in some other fashion in the TSE database.  They note that there is no established protocol in the TSE that would require the rescanning or editing of ATX images.  They also note that there are no attempts on these actas to change any of the data....just the curiosity that original time stamped images were replaced with lower quality images that removed the time stamp information that could have been used to audit the arrival of each MER's acta.  No international observers report being present when acta images were re-scanned or edited.

But there are other curiosities with the images as well.  The ATX transmitted version of acta 13014 has no signatures, but the image stored in the TSE database does.  You can see side by side copies here. The Voto Social authors suggest it was transmitted this way to allow the TSE to add the signatures, though why they would want to do that is not examined.  This particular acta contains more votes for Nasralla than it does for Hernandez.

What does this mean?  It means that there is undocumented either rescanning or editing of ATX scanned actas as part of TSE procedures, neither of which appears as part of the governing rules of the Sistema Integrado de Escrutinio y Divulgación Electoral (SIEDE), the composite TSE vote tallying system and software., for this election.

Sunday, December 10, 2017

Auditing Update

This is an update to the previous post based on a memo issued by the Tribunal Supremo Electoral (TSE)  (not Audisis) on the crash of the TSE servers on November 29.  This update is a result of finding a technical appendix on another copy of the TSE memo.

First, lets make it clear the memo comes from the TSE, not Audisis as the press coverage implied.

The appendix describes the vote counting system as a group of clustered servers.  Scanned acta images are transmitted from the INFOP warehouse to a cluster of two receivers at the TSE.  Transmission is via JSON, a markup language, over a secure network connection.  From there, the JSON markup is converted to a database record, transitioning through an application software cluster of two servers, and a database cluster of two servers.  We learn for the first time what we had guessed; that the database software is MS SQL.  The database cluster talks to a single  database of 600 Gbytes on a SAN system which has 12 TBytes of free storage.  The transcription of the actas takes place on a series of desktops connected to the application software level over a network.  We know that there are at least 60 computers doing transcription today, and probably more.  The Technical diagram does not show the web dissemination of data a part of this process.

The appendix says that Dell AMC sized the original database at 600 GBytes based on recommendations of the vendor of the transcription software.

The tecnical appendix has confused and undoubtedly not correct set of times for events, and contradicts the main report of what happened with respect to the database.  We will use the times reported in the appendix here, but compare with the times reported in our previous blog post, which came from the body of the report.

Both the appendix and the body of the report agree that the servers ran out of space and went down at 9:47 am on November 29.  They shut down the transcription system and enlarged the database to 1.8 TBytes.  They repeat that it took 3 hours 20 minutes to enlarge the database and bring the transcription system back up, coming back on line about 1:10 pm the same day.  When they restarted transcription, the MS SQL database was unstable, needing to be restarted every 10 minutes.  No explanation of why it kept crashing or why they thought taking the next steps would resolve it.

At 5:30 pm they took down Node 1 of the database cluster, leaving Node 2 to do all database access. This supposedly restored some stability to the Transcription service.  At the time they took Node 1 off the cluster, they decided to install a new database instance on it, in case Node 2 started crashing.  This involved reinstalling MS SQL on Node 1, which they renamed SQL4, which they then gave a 6 TByte database.  They also configured another server, SQL5, as a mirror of SQL4, with a 2 TByte database.  They took a snapshot of the database on Node 2 at 9:47 pm? and restored it on to SQL4 and SQL5, finishing about 1:10 am? (the appendix says PM but that's not possible unless it took into the next afternoon, long after they'd restarted the transcription process).  The transcription server was then reconfigured to use SQL4 and Node 2 and its database were permanently retired. SQL5 collected a mirror of the database but otherwise was not part of the transcription process.

Audisis reportedly then audited what the TSE had done.  The TSE claims what they did was completely transparent.  This description of a reformat of the server and installation of a new, larger database on the SAN matches with what the Alianza reported when they stated that the TSE had formatted the server.  They did.

Just as a point of normal procedure, it should have been Audisis that released a report on the changes to the system, not the TSE, and it should have been Audisis in its own words reporting on the integrity, or lack thereof, of the systems after the change.  Instead the TSE chose to put words in Audisis's mouth.  That they managed to nearly fill up a 12 TByte SAN with databases is remarkable.

Saturday, December 9, 2017

Auditing a Failed Election

Auditoría Integral y Seguridad de Sistemas de Información (Audisis),the Colombian firm hired by the TSE on November 13 to audit the election results for the November 26th election, has released an unsatisfactory memo detailing what caused the TSE computers to fail on November 29,  and what modifications were made to the system to fix it.  In the process it raises more questions than it answers.

The Tribunal Supremo Electoral (TSE) announced it had reached a contract with Audisis to audit the election results of the November 26th election on November 13th.  David Matamoros Batson informed everyone at the time that they were lucky to have found such a qualified firm for only $700,000.  Matamoros stated that Audisis and the TSE had agreed on 7 points to be audited:
(1) Security of the actas, that the image scanned in the voting center is the same as the image received by the TSE to count.
(2) verify the software used by the TSE for counting and publicizing the tallies is what is being used.
(3) network security
(4) verify the functioning of the vote tallying software for all the elections being run on November 26.
(5) security of the database
(6) make sure transmitted actas are shared with the political parties
(7) evaluate the suitability of the technology being used to publicize the results.

So with the above in mind, Audisis released, through the TSE, a "report" of what happened when the system went down on November 29th.  According to Audisis, the system went down at 9:42 AM because it had filled up its database.  This, by the way, contradicts President Juan Orlando Hernandez who claims it never crashed or became unavailable, just slow. 

What the graphic in the report shows is two server instances running with a 12 Terabyte SAN storage network, but only a 600 Gigabyte database allocated, and apparently shared between the two servers, which are clustered for high availability.  It then took them 3.2 hours to expand the database to 1.8 Terabytes, bring up the servers, and perform a minimal data audit.  The servers were up for production again at 1:08 pm and they began adding actas again at 1:10 pm.

 They continued to observe problems with database performance and decided to bring the system down again at 6 pm the same day.  They increased the database size again; this time to 6 Terabytes.  It took them 5 hours 30 minutes to reconfigure the system to use the additional capacity.  They also added a 3rd server, this one configured with a 1.8 Terabyte database, to receive replicated data from the original database as a check of system integrity.  The system returned to production around 11:30 pm that evening, almost 9 hours after it was halted.

The first thing you do when you design a database is design the table structure, then make a good faith estimate of how much storage space the data will need.  You always give a healthy overestimate because you have to remove the database from use to increase its size.  When you're processing election data, you don't want that to happen.  With the kinds of data we are dealing with here, you should be able to make very precise estimates of how much storage space should be needed. Yet somehow they failed.

I can't fathom how they filled up a 600 Gigabyte database, even with all the acta images from Presidential, Congressional, and Municipal elections and a complete voter roll stored in the database, I estimated it would only take about 34 Gigabytes of database storage to process the results of the election.  I, after all, stored the complete results of the 2013 election in a database on my laptop without it taking up even 20 Gigabytes.  Even with every conceivable kind of transaction logging turned on, I'd be hard pressed to design a database requiring 100 Gigabytes.  What were they doing?

Replication put simply is the ability to have two or more databases stay synchronized to provide greater availability.  If they are in geographically different locations they can also be used for disaster recovery.  Since the databases need to remain identical, normally they would need to be of the same size.  So why would you replicate data from a 6 Terabyte database to a 1.8 Terabyte database?  Doesn't that mean you didn't need the 6 Terabyte database size?  Or even the 1.8 Terabyte database size?  Unclear from the report released by the TSE is whether the third server added already had a cloned database, so that only newly added data needed to be replicated, or whether the database was empty and needed to replicate all the existing data across a network.  While replication is designed by database vendors to minimize its impact on servers, it still has a measurable impact.

In many ways the report released by the TSE, supposedly compiled by Audisis, raises more questions than it answers.  It provides an excuse for why the TSE systems went down for almost 10 hours, but the reason doesn't seem credible.