When the computer is presumed right: testing system records

01A shortfall the system reported

The Post Office introduced its Horizon accounting system into branches from about September 1999. Branch managers, known as subpostmasters, were responsible for the cash and stock in their offices. When Horizon showed a shortfall, the Post Office generally treated it as a real loss and asked the subpostmaster to repay it. Some people were prosecuted for theft or false accounting on the strength of Horizon data.

Subpostmasters said the system produced false figures, and complaints began soon after the rollout. The Post Office's position was that the data was wholly reliable. Both sides were describing the same records, so the question was which account could be tested.

The case is not a simple story about one bad program or one dishonest branch. It asks how an organization decided which figures to believe, and what it did when a problem was reported.

02What a presumption of reliability does

A presumption of reliability is the default assumption that a system's output is correct until someone shows otherwise. It is reasonable for many machines. It becomes a problem when the output is the only evidence and the people who dispute it cannot examine how the figures were produced.

In the Post Office case, the Court of Appeal accepted that unexplained shortfalls were treated as undisputed debts, and that even a subpostmaster who had resigned still owed the disputed sum. The court said that by representing Horizon as reliable and refusing to countenance any suggestion to the contrary, the Post Office effectively sought to reverse the burden of proof. In its judgment of 23 April 2021, the court allowed the appeals of 39 of the 42 appellants and quashed their convictions.

Interpretation: a process that treats a disputed figure as a debt has already settled who must show what happened. The judgment itself decides only the appeals before it, and does not by itself establish the facts of every other conviction.

The Inquiry's first volume, ordered printed on 8 July 2025, states that many hundreds of people were convicted, wrongly, of criminal offences. The Inquiry is precluded by the Inquiries Act 2005 from determining criminal or civil liability, so that statement is its own finding rather than a legal verdict.

03What the courts found about the system

In the Horizon Issues judgment of December 2019, the High Court found that bugs, errors and defects could create apparent shortfalls in branch accounts, and that this had happened on numerous occasions in both Legacy Horizon and Horizon Online. The judge also said the precise extent of their effect could not be calculated.

Reliability differed by version. The judge described Legacy Horizon, used until 2010, as not remotely robust, found the robustness of the later online version questionable, and found HNG-A, in use from 2017, far more robust. A single verdict about the software would have been too blunt, because each version needed its own test.

The judgment also recorded evidence that remote access to branch systems could change transaction information without postmasters being aware. The judge found the occasions were not isolated incidents, but could not reach a concluded view on how often they happened, so the frequency cannot be read from the record.

04A warning that was not used

On 25 February 1991, a Patriot battery at Dhahran, Saudi Arabia, failed to track and intercept an incoming Scud missile, which struck an Army barracks and killed 28 American soldiers. The General Accounting Office later reported that Army officials had received Israeli data showing a 20 percent range-gate shift after eight consecutive hours of operation. They modified the software, but the modified version reached Dhahran only on 26 February.

The mechanism was a clock that drifted with uptime. The count of tenths of a second was converted using 24-bit registers, so each conversion carried a small error that grew as the count grew. The GAO calculated a timing error of 0.3433 seconds after about one hundred hours, which corresponds to a range-gate shift of roughly 687 meters. These are calculated values from the stated precision, not measured track data.

Interpretation: a warning that was received but not used to decide how long the system could run is a records problem as much as a software problem. The GAO also noted that rebooting, which resets the clock, could have prevented significant shifts.

05Reproduce the failure under the real conditions

Between June 1985 and January 1987, six known accidents involved massive overdoses from the Therac-25 radiation machine, with deaths and serious injuries. After the Tyler overdose of 21 March 1986, the manufacturer stated that an overdose was impossible and that no other accidents had occurred. Early reports were treated as implausible before the mechanism was understood.

The clue was Malfunction 54. At Tyler it appeared when prescription data was edited rapidly. AECL engineers could not reproduce the error at first, and produced a similar malfunction only after a physicist explained that the procedure had to be done quickly. Reproduction turned a disputed report into a checkable fact.

Interpretation: the useful test was a reproduction under the timing the machine actually allowed, not a test under the slow sequences operators would normally use. The case points to three checks: the machine's logs, user data and reproduction tests.

06Shared software and removed checks

The Therac-25 relied more on software for safety functions than its predecessors did, and AECL chose not to duplicate all the existing hardware interlocks. Software from earlier models was reused in part, and concurrent tasks shared variables without real synchronization. The result was that a timing sequence could produce states the designers had not anticipated.

Knight Capital's 2012 incident had a similar structure. New code reached seven of eight router servers, and an order flag used by the new program activated old functionality on the eighth. The SEC identified 97 automated emails and inadequate controls over outgoing orders, aggregate exposure and incident response. The SEC reported losses exceeding $460 million.

Interpretation: the Knight case suggests that the deployed state has to be verified on every instance, not only on the majority. A check that confirms seven of eight servers says nothing reliable about the eighth.

07Testing a system's records

Investigators test a system's records against something the system does not control. Useful questions are whether the system keeps an audit trail of changes to its data, whether an independent source recorded the same event, whether the output matches what users and operators saw, and whether the error can be reproduced under the timing, load and uptime the system really experiences.

Test each version and each operating condition separately. The Post Office judgments found different reliability across versions, the Patriot error grew with uptime, and the Therac-25 failure depended on timing. A single statement that the system is reliable hides all three.

Interpretation: when a system is the only witness to its own records, the person who relies on a figure is the one best placed to show that the figure is right, and the person who disputes it is the one least able to test it. The Court of Appeal's reasoning in the Post Office case points that way.