Automation surprise: when a system does something else

01A gap between the model and the machine

Automation surprise happens when an automated system does something its operators did not expect, and they must work out what it is doing before they can act. The usual source is a gap between the operators' picture of the system's mode and the mode it is really in, which is called mode confusion. A related weakness is single-sensor dependency: one input decides how the system behaves, and nothing checks it against a second source.

Interpretation: a surprise is not the same as a fault. An automatic function can do what it was built to do and still catch a crew that was not shown the conditions it responds to. The useful question is which signal the crew had.

02Control handed back at altitude

On 1 June 2009, Air France flight 447, an Airbus A330-203, was cruising at 35,000 feet over the Atlantic. At 2 h 10 min 05 the autopilot and autothrust disconnected, the flight control law changed to alternate law, and the speed indications became inconsistent. The BEA's final report names ice crystals obstructing the Pitot probes as the probable starting point, a blockage that is usually temporary.

The stall warning depended on valid angle-of-attack data. Those values were declared invalid when the measured airspeeds fell below 60 knots, so the warning stopped and restarted as the readings failed and recovered. The copilots identified the loss of airspeed indications, but neither used the unreliable airspeed procedure or formally identified the stall.

Interpretation: the sequence turned a sensor problem into a change of control. The copilots had had no in-flight training at high altitude for that procedure or for manual handling, and the report weighs icing, training and coordination together without ranking them.

03One sensor, repeated commands

On the Boeing 737 MAX, MCAS could push the nose down by moving the horizontal stabilizer in certain high-angle conditions. The House committee report states that MCAS acted on input from a single angle-of-attack sensor and was not classified as a safety-critical system.

On Lion Air flight 610 and Ethiopian Airlines flight 302, the crews struggled against nose-down trim for much of the flight. The House report, summarising the Indonesian investigation, describes MCAS acting more than 20 times on the first flight. Pilots were not told MCAS existed, and the AOA DISAGREE alert that could have flagged a faulty sensor was inoperative on most aircraft.

Interpretation: the surprise was not a mode the pilots could read. The system acted on one value, and the crew had neither a description of the function nor a working disagreement alert to show the value was suspect. The House report, the Joint Authorities Technical Review and the Ethiopian report all treat single-sensor reliance as a central weakness.

04A display that showed a command

After Three Mile Island Unit 2 shut down on 28 March 1979, its pilot-operated relief valve opened as designed when pressure rose, but it did not close when pressure fell. The control-room indicator showed only that a close signal had been sent. The indicator had been installed in 1978, after the relief valve stuck open a year earlier at Unit 2.

Operators relying on the light did not heed other indications. They read rising pressurizer level as a sign of a filling vessel and were concerned that the plant was going solid. Injection was cut from about 1,000 gallons per minute to under 100 for extended periods, and the loss of coolant lasted more than two hours. The commission judged that core damage would have been prevented, despite the stuck-open valve, had injection not been throttled.

Interpretation: the indicator answered a different question from the one the operators needed answered. The case's lesson is that operators need independent indications when the position of a safety valve is uncertain.

05Shedding load under pressure

During Apollo 11's powered descent on 20 July 1969, the guidance computer issued five program alarms, starting with a 1202 at 102:38:22 mission elapsed time. Alarm codes 1201 and 1202 meant the executive had run out of resources for scheduled jobs. The mission report attributes more than 10 percent of the computer's capacity to counter interrupts from the rendezvous radar, whose mode switch was in Slew, not under computer control.

The restarts cleared some of the load but not all of it. Adding landing-site processing in P64 left less than 10 percent duty-cycle margin, and three alarms came within 40 seconds. Eyles, a member of the guidance software team, writes that the restart design was built for hardware glitches, so its load-shedding under this overload was a side effect. His account is a participant's memoir, not a contemporaneous log.

Interpretation: here the trust was supported. Houston replied Go after the 1202 was reported, on the judgment that the overflow concerned radar pointing data the computer was programmed to treat as secondary. That judgment rested on evidence about how the computer behaved, although the team had not seen this alarm in simulation. The crew then eased the load by moving to attitude hold and then semi-manual P66. This is the clearest contrast with MCAS: the system shed work, and the people responsible had a basis for trusting it.

06A time reference that drifted

On 25 February 1991 a Patriot battery at Dhahran failed to intercept an incoming Scud, which struck an Army barracks and killed 28 American soldiers. GAO traced the tracking error to the system clock. The clock counted tenths of a second as an integer, and converting that count used 24-bit registers, so the conversion error grew with continuous operation. GAO's calculated timing error was 0.3433 seconds after about 100 hours, a range-gate shift of about 687 meters, derived from the stated precision rather than measured track data.

The warning came early. On 11 February 1991 the Patriot Project Office received Israeli data showing a 20 percent range-gate shift after eight consecutive hours of operation. The modified software, released on 16 February, reached Dhahran on 26 February, the day after the incident. GAO also reports that rebooting, which resets the clock, could have prevented significant shifts.

Interpretation: this is less a mode change than an input that drifted with uptime, and the battery relied on one internal clock, a shape close to single-sensor dependency. The summaries do not reconstruct the track at the moment of attack, so the case establishes the mechanism and the warning chain, not the exact effect on that engagement.

07Reading the surprise

Four questions help when reading any automation surprise: which mode or input the system was using, which signal showed that state to the crew, whether a second, independent check existed, and whether the condition had been tested before it mattered.

Interpretation: automation did not simply work in one case and fail in another. In each example it acted on the inputs it had. What separated them was whether the crew had a truthful picture of the system's state, and whether anyone checked the input before the system acted on it.