Mars Pathfinder software resets investigation

01 · The briefing

Begin with the question.

Mars Pathfinder launched in December 1996 and landed on 4 July 1997 at Ares Vallis, carrying the small rover Sojourner. Its flight software ran on the VxWorks real-time operating system, with tasks given fixed priorities so that time-critical work could take precedence over science and housekeeping. A bus scheduler ran at a high rate, and all of the tasks shared a single processor. During the surface mission the computer reset. On a spacecraft a reset is serious: it stops the current activities, reinitializes the system and delays the next day plans. Engineers had to decide whether the cause was a faulty part, a memory problem, an error in new code or a timing conflict built into the design. The question was harder because the failure occurred only under some conditions and was difficult to reproduce. Your investigation has to separate the hardware and data paths from the scheduling logic. Activities that day were postponed to the next day while the team worked on the cause. Identify which task was late, what it was waiting for, and why lower-priority work could delay it.

The timeline

1996-12-04
Mars Pathfinder launches from Cape Canaveral aboard a Delta II.
1997-07-04
The lander touches down at Ares Vallis, Mars, with the Sojourner rover aboard.
1997
During surface operations, the computer resets; the day activities are completed the following day.
1997-09-27
The end of the surface mission is recorded by NASA.
1997-12-15
The software team lead, Glenn Reeves, sends a first-hand account of the failure, diagnosis and fix to colleagues.

A researched reconstruction for learning. These reasoning steps simplify a complex record; no score can capture the full history.

02 · The evidence

What does the record show?

Small six-wheeled rover with a solar array and an upright antenna on reddish, rocky ground. — On the evidence table1997 The Sojourner rover photographed near a rock called The Dice, as shown in NASA image PIA01122. Photo: NASA · Public domain · Wikimedia Commons.

Read each exhibit before settling on an explanation. The source links below every exhibit lead to the record behind it.

Exhibit 01

The priority order

Reeves describes the bus scheduler as the highest-priority task after the VxWorks executive, the bus distribution task third, and the meteorological task lower still, with several medium-priority tasks running in between. This structure made it possible for a low-priority task to block a high-priority one. The account does not give the full task table or the scheduler configuration.

Source record s1web.archive.org

Exhibit 02

The blocked resource

Reeves reports that the meteorological task called select() and was preempted while it held a mutual-exclusion semaphore inside the pipe mechanism. The distribution task then blocked on the same semaphore when it sent data, and medium-priority tasks ran while the low-priority holder could not finish. This explains the inversion. It does not show how often the sequence occurred, since the failure was rare.

Source record s1web.archive.org

Exhibit 03

The reset logic

At the next cycle the bus scheduler found that the distribution task had not finished, which the spacecraft treated as a failure, and the computer was reset. Reeves notes that data in memory were recovered as long as power was kept, but the remaining activities for the day were postponed. This shows the consequence designed into the system. It does not identify the cause, which is the inversion.

Source record s1web.archive.org

Exhibit 04

Reproduction on the ground

Reeves reports that the flight software contained trace and log facilities left in by design, which helped the team find the sequence. The team ran the same activities in the laboratory with the flight software unchanged, and reproduced the failure in less than 18 hours. This shows that flight-configuration instrumentation made diagnosis possible. It does not show that the timing was understood before the reproduction.

Source record s1web.archive.org

Exhibit 05

Mission background

The JPL history page confirms the December 1996 launch, the July 1997 landing and the airbag-based landing with a small rover. It establishes context rather than cause, since it does not describe the software fault.

Source record s2web.archive.org

03 · Your reasoning

Follow the strongest explanation.

Choose an interpretation, then compare it with the evidence. You can revise your choice. The goal is understanding, not a perfect score.

Reasoning step 01 of 04

The reset followed a missed deadline by the bus distribution task. Which observation most directly explains why that task was late?The bus distribution task was the most processor-intensive task in the system.The account does not say this; the task was third in priority, and a heavy load would not by itself produce the specific blocking sequence.A lower-priority task held a lock the higher-priority task needed, while medium-priority tasks kept running.This is the priority inversion Reeves describes, and it explains why the late task could not progress.The spacecraft radio was transmitting when the reset occurred.Radio activity is not identified in the account as the cause of the blocking sequence.

Evidence-based explanation

The distribution task blocked on a semaphore held by a preempted lower-priority meteorological task, so medium-priority work kept running. Processor load or the radio alone would not produce this specific inversion.

Weigh it against

Exhibit 01The priority order Exhibit 02The blocked resource

Reasoning step 02 of 04

Which test would best confirm the inversion before the software was changed on the spacecraft?Re-run the flight software on the ground with the trace facility enabled, and record whether the same lock sequence and deadline miss occur.This reproduces the flight configuration and captures the sequence, which is what confirmation requires.Count how many times the radio transmitted each day.Transmission counts do not show lock ownership or the order of task execution.Replace the bus hardware with a spare unit and wait for the next reset.Waiting for a reset is slow and uninformative, and a hardware swap would not show the scheduling sequence.

Evidence-based explanation

Reproducing the flight configuration with the trace facility shows the lock sequence directly. The team did this, and the diagnosis rested on that reproduction rather than on hardware replacement.

Weigh it against

Exhibit 04Reproduction on the ground Exhibit 02The blocked resource

Reasoning step 03 of 04

The fix was to enable priority inheritance on the semaphore. What is the strongest reason to test its side effects before uploading the change?Priority inheritance requires new hardware that the spacecraft does not have.The account does not describe any such requirement; the change was made in software.The patch must be uploaded as a full copy of the flight software, not as differences.The account describes sending the differences between the onboard and ground copies, so this is not the risk.The change alters how every semaphore created after that point behaves, so its effects on other tasks and performance must be assessed.Reeves says the global option changed all select semaphores created afterwards, which is why the team analyzed and tested it.Priority inheritance makes the bus task run before the science tasks, so no test is needed.This misstates the change and would dismiss the testing the team carried out.

Evidence-based explanation

Reeves describes a global option that changed every select semaphore created afterwards. The team asked Wind River to analyze performance and behavior impacts and tested the system extensively before changing the spacecraft.

Weigh it against

Exhibit 02The blocked resource Exhibit 01The priority order

Reasoning step 04 of 04

Which lesson is best supported by the Pathfinder case?Remove diagnostic code from flight software, since it can only add risk.The team kept the trace facilities by design, and they made the diagnosis possible.Assume that a commercial operating system defaults are safe, since vendors test them.Reeves states the opposite lesson: know how a commercial component works before flying it.Because the fault was rare, no test changes are needed after a reset.A rare fault still reveals a gap in test coverage, which the account identifies.Test the flight configuration under heavier and more varied loads than the nominal case, and keep the diagnostic tools that made diagnosis possible.Pre-launch tests covered best-case data rates, and the diagnostic tools were what made the sequence visible.

Evidence-based explanation

Reeves states that pre-launch testing was limited to best-case data rates, and that the team kept diagnostic tools in the flight software by design. The lesson is supported by that account, though it comes from one first-hand source.

Weigh it against

Exhibit 04Reproduction on the ground Exhibit 02The blocked resource

04 · The findings

The chain behind the outcome.

The reset was caused by priority inversion. The bus scheduler ran at a high rate and needed the distribution task to finish before the next cycle. The distribution task was blocked on a mutual-exclusion semaphore inside the select mechanism for a pipe. That semaphore was held by a low-priority meteorological task, which had been preempted in the middle of releasing it. Medium-priority tasks then ran for long enough that the holder could not finish, so the higher-priority task waited, missed its cycle and triggered the reset. The team found the sequence with trace facilities that had been left in the flight software by design, and reproduced it on the ground in under 18 hours without changing the software. The fix enabled priority inheritance on the semaphore by changing a global creation option, which temporarily raises the holder priority while a higher-priority task waits. The change affected every select semaphore created afterwards, so the team analyzed and tested it before uploading the software differences. The established points are the sequence, the reset logic, the reproduction and the patch method, as described by the software lead. The interpretation lies in why the problem was not caught earlier. Reeves states that pre-launch testing covered best-case data rates and that the team saw the problem before landing without reproducing it. The durable lesson is that a correct design can hide a timing dependency that nominal tests do not expose, and that a team must know how the commercial components it relies on actually behave.

How it ended2 Oct 1996 Technicians in clean-room suits work on the Mars Pathfinder lander, closing its petals, on 2 October 1996. Photo: NASA · Public domain · Wikimedia Commons.

Contributing causes and conditions

Fixed priorities that let a medium-priority task run while a low-priority task held a lock
A shared semaphore inside the select mechanism for a pipe used for communication between tasks
A high-rate bus cycle with a hard deadline that the blocked task could not meet
A reset policy that treated the missed deadline as a failure and restarted the computer
Pre-launch tests that covered best-case data rates rather than the heavier case that exposed the sequence

OutcomeMars Pathfinder software resets

Take the method with you

A higher-priority task can be delayed by lower-priority work when both share a resource.
Keep diagnostic facilities in flight software when they help explain an anomaly, and test the flight configuration itself.
Know how a commercial operating system defaults behave before relying on them, since one global option can change many objects at once.

You can read findings at any time. Exploring every reasoning step marks this case as complete in your notebook.

Open another investigation

A reset from a waiting task