Fair tests before statistics: how comparisons earn their weight

01Twelve sailors in six pairs

In May 1747 the surgeon James Lind, working aboard HMS Salisbury, took twelve men suffering from scurvy and divided them into six pairs. Each pair received a different addition to the ship's regular diet, so the six comparisons ran side by side under the same conditions. Lind's account of the trial appears in his Treatise of the Scurvy, first published in 1753.

The design is worth studying for its features. The pairs shared one ship and one illness, which kept many conditions common to all of them. The treatments were compared at the same time and in the same place, so that time and location did not separate one comparison from another. A single factor was meant to vary between the pairs. One pair received two oranges and a lemon each day, which is the citrus group the account describes.

A caution belongs here too. Twelve men in six pairs is a very small comparison, and one trial does not establish a general rule. What it demonstrates is that a deliberate, parallel comparison can produce a clear contrast that the observation of a crowd could not. The contrast still required interpretation and independent confirmation before it could guide practice.

02Two wards with different outcomes

At the Vienna General Hospital, two obstetrical divisions had very different mortality. The First Division had the higher rate of deaths from childbed fever, and the Second Division, where midwives were trained, had the lower. Ignaz Semmelweis was troubled by the difference, and the contrast between the two wards became the starting point for his reasoning.

From the second half of May 1847, Semmelweis introduced handwashing with a chlorinated lime solution. He set out his argument in his 1861 book, whose title names the etiology, the concept and the prophylaxis of childbed fever. The book is the primary record of his reasoning, and it should be read directly for the details of his evidence.

The lesson is about the limits of a single difference. Two wards differ in staff, routines and training as well as in the practice being studied. A difference between them is a clue. The stronger test is whether a change in one practice reproduces the change in outcome while the other differences are held as nearly constant as circumstances allow.

A useful comparison can be built from routine records, provided the records are kept the same way in both places. Consistent definitions of a death, a case or an exposure are a condition of any fair comparison, and they deserve checking before the numbers are trusted.

03Following the water

In the 1854 Broad Street outbreak in Soho, London, John Snow found many deaths near the Broad Street pump and then asked where the affected people obtained their water. The neighborhood pattern was the starting point, not the conclusion.

Two groups mattered. Snow reported that the brewery close to the pump had no severe cholera among its workforce, and that its proprietor said the workers did not use the street pump and had other supplies. Distant people who drank water brought from Broad Street were affected. Location alone did not explain the pattern, while exposure did.

Snow was also candid that the outbreak had diminished before the pump handle was removed. His own account acknowledges that the before-and-after sequence cannot measure the intervention's effect precisely. A fair comparison therefore has to be paired with honesty about what it cannot show.

Snow's approach is worth stating explicitly. He moved from the pattern on the map to the question of exposure, and from exposure to a test that could fail. If the water was the channel, people who drank it should be affected wherever they lived. The distant water drinkers were that test.

04Bacteria that an accepted explanation missed

In 1983 J. Robin Warren and Barry Marshall reported curved bacteria on the gastric epithelium in patients with active chronic gastritis. A 1984 paper by Marshall and Warren described curved bacilli in biopsies from 58 of 100 patients. The Nobel Prize summary explains that the work used technologies generally available at the time, including fiber endoscopy, silver staining of tissue sections and culture techniques for microaerophilic bacteria.

The comparisons were specific. Warren observed that signs of inflammation were present wherever the bacteria were seen, which pointed to a link between the organism and disease rather than to an unrelated presence. In 1982, stress and lifestyle were considered the major causes of peptic ulcers, and the bacterial explanation challenged that view. The Nobel summary notes that the link was then established through studies of volunteers, treatment trials and epidemiological studies.

Treatment provided another kind of test. The Nobel summary states that peptic ulcer disease could then be cured by a short regimen of antibiotics and acid secretion inhibitors. A cure that follows from removing a suspected cause is a different kind of evidence from an association, and the case shows that several fair comparisons, not one, carried the conclusion.

The discovery also shows the risk of an accepted explanation. Stress and lifestyle were reasonable candidates in their time. A fair comparison has to include the explanation that the prevailing view does not, and then test it in the same disciplined way.

05What statistics can and cannot repair

Statistics quantify the uncertainty of a comparison. They tell you how large a difference would be by chance alone, given the way the data were gathered. They cannot rescue a comparison in which the groups differed systematically from the start, because the difference being measured already belongs to the design.

Good design includes three ideas these cases anticipate. A control group shows what happens without the factor under study. Random assignment, where it is possible, spreads unknown differences between groups by chance rather than by someone's judgment. And exceptions should be examined: the brewery workers and the distant water drinkers were the observations that separated a water explanation from a neighborhood explanation.

Before reaching for a formula, ask who chose the exposure, what else differed between the groups, and what the comparison would look like if the proposed cause were absent.

A fair comparison is also a matter of reporting. A study should state how its groups were formed, which comparisons were planned and which were not. Readers can then weigh the evidence rather than trusting a single striking result.