Competing hypotheses in practice: list the rivals, then test them

01A matrix, not a verdict

Richards J. Heuer Jr.'s Psychology of Intelligence Analysis, published by the CIA's Center for the Study of Intelligence in a 1999 edition, devotes a chapter to what he calls analysis of competing hypotheses. The method is simple to state. List the plausible explanations. List the evidence. For each item of evidence, ask how consistent it is with each explanation, and focus on the evidence that rules hypotheses out rather than on the evidence that merely fits the favorite.

Heuer is candid that the method will not always yield the right answer. Its purpose is to counter shortcuts that analysts share with everyone else: fixing on an early explanation, discounting evidence that does not fit, and weighting what is easily remembered. The value is a discipline that keeps the rivals alive until the evidence has had its say. The same logic applies in astronomy, cryptanalysis and geology, as the cases below show.

In practice the matrix can be as simple as a table with hypotheses in columns and evidence in rows, each entry marked for consistency. The most useful entries are the ones that rule something out. A table full of consistent marks may show that the hypotheses are not being separated by the evidence at all.

02A residual with several possible causes

By the mid-1840s, astronomers had a problem with Uranus. Its observed motion did not fit the calculated orbit closely enough. A residual, the gap between observation and prediction, signals that something is wrong, but not yet what. The error might lie in the observations, the model of forces might be incomplete, or an unseen body might be pulling on Uranus.

Urbain Le Verrier in Paris and John Couch Adams in Cambridge, England, independently calculated where a further planet would have to be to account for the disturbance. The important feature of Le Verrier's work was that it produced a position in the sky, not merely a claim that something existed. Johann Gottfried Galle at the Berlin Observatory observed a new planet only about one degree from the calculated position.

The prediction did not eliminate the rivals so much as rank them. An unseen planet made a risky claim that could fail, and the observations and the model still contained errors. The case shows that a successful prediction does not make every parameter of the model exact.

Notice the sequence. The prediction came first and the observation followed. A rival that predicts a position can be tested by pointing a telescope at it. A rival that only explains an existing residual can be tested only by finding new data that the original fit did not use.

03Cribs as hypotheses about plaintext

At Bletchley Park, codebreakers faced machine ciphers whose settings changed daily, which created an enormous number of possibilities. The productive method was not to guess an entire message. It was to test a guessed stretch of plaintext, called a crib, against intercepted ciphertext. A crib makes predictions. If a phrase sits at a given position in a message, certain relationships in the ciphertext must follow, and settings that violate them can be rejected.

The Bombe, the machine that searched settings against those constraints, is described in Bletchley Park's teaching notes as working from links between letters in the ciphertext and a crib, a guess at part of the message's content. Surviving candidates still had to be checked. The work was a pipeline. Interception, cryptanalysis, machine operation, decryption, translation and the distribution of intelligence were distinct stages, each with its own errors.

For the method of rival hypotheses, the lesson is that a guess becomes useful when it generates falsifiable constraints. Rejection is informative. Confirmation needs a second, independent check.

The same method applies after a crib succeeds. A candidate setting that survives one check is not yet a solution, and it must survive further, independent tests. Each surviving candidate is a rival hypothesis that has earned another round of testing.

04Two rivals for one boundary layer

The iridium anomaly in the boundary clay, reported in 1980, had more than one possible interpretation. A purely local source was one rival, and an extraterrestrial impact was the other. The geographical spread of the anomaly challenged the local explanation, but the impact proposal was still only a testable hypothesis. The buried Chicxulub structure in Mexico had been interpreted as volcanic, and its outline alone did not settle which process had made it.

The rivals made different predictions about material that could be examined. An impact should leave shocked minerals and impact melt, and a deposit formed by ejection should look different from one formed by ordinary volcanic activity. The Lunar and Planetary Institute's account of a thick deposit on Hispaniola describes it as dominated by impact melt spherules and rich in shocked quartz, both indicating ejection from an impact crater.

Rival explanations are most useful when they predict different things. The boundary case shows that the best test is not the most famous observation, but the one in which the rivals diverge.

05Rivals decided by a pattern of ages

For much of the twentieth century, the debate over moving continents rested on the fit of coastlines, on fossils and on evidence about ancient climates. Alfred Wegener assembled such evidence, but a convincing physical mechanism remained difficult. Later measurements of the seafloor offered tests that did not depend on how neatly the continents fit together.

Magnetic patterns recorded in ocean-floor rock form stripes that match on opposite sides of mid-ocean ridges. The US Geological Survey explains that rocks at the ridge crest are the youngest and grow older away from it, and that the youngest rocks carry present-day magnetic polarity. A stationary ocean floor and a floor of new crust moving outward make different predictions about those ages and stripes. The seafloor-spreading explanation survived because it predicted patterns that could be checked.

The rivals were compared on a dimension where they diverged, not on the number of resemblances each could claim. That is the discipline the matrix is meant to enforce.

The contrast with the older debate is instructive. A proposal that fits the shapes of coastlines can be argued for a long time, because the shapes can be read in more than one way. A proposal that predicts the ages of rock on the seafloor can be checked against an independent record, and that is why such predictions carry more weight.