Software timing failures: priority, drift and inherited code

01A bus manager that stopped running

During its mission on Mars in 1997, the Pathfinder spacecraft began resetting itself. Engineers on Earth had to work out why a system that looked healthy kept restarting. The software ran on the VxWorks real-time operating system, which schedules tasks by priority: the highest-priority task that is ready to run gets the processor.

The resets came from a high-priority information bus task that stopped being scheduled. A lower-priority meteorological task had taken a shared resource, a mutex, and was holding it. The bus task needed the same resource and was blocked. A medium-priority task then ran ahead of the low-priority task holding the resource, so that task could not finish and release it. The bus task waited. After a while a watchdog timer noticed that the bus task had not run for some time and initiated a total system reset.

02Priority inversion, in plain terms

Priority inversion is a situation in which a high-priority task must wait for a lower-priority task, while medium-priority work keeps the lower task from running. The priorities look correct on paper, but a shared resource turns the arrangement upside down. The important task does not get its turn because the task holding the resource does not get the processor.

The engineers reproduced the behavior on a replica and analyzed a total trace of system events, which recorded context switches and uses of synchronization objects. The trace revealed the inversion. The fix was an option rather than a new algorithm: when a VxWorks mutex is created, a parameter enables priority inheritance, so that the task holding the mutex temporarily takes the priority of the highest task waiting for it.

The general lesson is that a scheduler's behavior depends on its resource rules. A system that shares a lock among tasks of different priority needs a documented answer to inversion, such as inheritance or a ceiling protocol, and that answer must be tested under load.

03Clocks that drift with uptime

On 25 February 1991 a Patriot battery at Dhahran failed to track an incoming missile, and 28 American soldiers were killed. The GAO analysis describes a clock that counted in tenths of a second, with an accumulated error that grew with the time the system had been running. After more than 100 hours, the error was about one third of a second, and the computed range gate had shifted by roughly 687 meters.

This is a timing bug of a special kind. Each tick was only slightly wrong, but the errors added up, and they grew with uptime. A system restarted often can hide such a bug, and a test that runs for minutes will not reveal it. Clock drift belongs in the design as a measured quantity with a stated limit, not as a rounding detail to be checked later.

A helpful step is to translate a drift into the units of the mission as early as possible. A timing error expressed in seconds means little to a designer until it becomes a distance, a missed target or a position error. Translating the error turns an abstract defect into a requirement that can be tested and given a limit.

04Reused code and inherited assumptions

The first flight of Ariane 5, on 4 June 1996, ended shortly after liftoff. The inquiry board found that the cause lay in specification and design errors in the software of the inertial reference system. Some of that software carried assumptions inherited from an earlier launcher, and a conversion of a number produced an overflow whose error was passed on to the flight control system.

Two inertial reference systems were installed for redundancy, but they ran the same software and met the same flight conditions. A backup with the same flaw protects against nothing that the flaw can cause. The corrective measures included switching off or inhibiting the alignment function after liftoff, a function no longer needed on this flight that continued to run. The board also found that reviews and tests had not adequately analyzed and tested the inertial reference system under the complete flight control system.

Reuse is not the problem in itself. Reused code is valuable because it has been exercised. The risk lies in the assumptions a reuse brings with it, which may be undocumented and may no longer hold. A reuse review should list those assumptions, the environment in which each was valid, and the tests that exercised them in the new environment.

05Testing timing and inherited behavior

The cases suggest a method. Start with the event trace: which tasks ran, which resources they took, and when. A replica lets engineers reproduce a fault without risking the flight unit. Test with realistic schedules and worst-case loads, and run long enough for drift to appear. Review every inherited function and ask whether it is still needed in the new environment. Check numerical conversions at the edges of their range, using real trajectories or workloads rather than nominal values.

A reset or an exception is a symptom, not an explanation. The useful question is which sequence of events would produce the symptom, and which test would show that sequence in the trace or the replica. In each case the explanation became actionable only once it could be reproduced and mapped onto the system's own design.

06Asking whether a program still works where it runs

None of these practices is a complete defense on its own. Together they change the question from whether a program works in the laboratory to whether it still works in the conditions it will actually meet. Those conditions include the schedule of competing tasks, the time since the last reset, the range of numbers the code will convert, and the history carried into the new system by reused code.

Investigators should also ask what the system was designed to tolerate, and whether that tolerance was tested. A backup that shares the same software tolerates a hardware fault but not a software fault, and a timing margin that is never measured is a margin nobody can defend.

Teams often ask whether a fault could happen again. A more useful question is whether the system would detect it if it did, and how quickly. Detection and recovery are part of the design, and they can be tested like any other function.