Too good to be true: warning signs before anyone replicates

After reading, you will be able to name the red flags that make a dramatic claim worth slowing down for, and to tell a claim from a measurement. You will also know which questions to put to a result before anyone spends time or money trying to reproduce it.

01Three claims that looked too good

In July 2023, two preprints said a copper-doped lead compound called LK-99 carried current with no resistance and levitated at room temperature and ordinary pressure. Within weeks, laboratories were trying to make it, and the most striking image was a single small flake hanging at a tilt over a magnet.

Bernard Madoff's firm reported steady returns for many years from a strategy described as a split-strike conversion, which combines stock trades with options. Theranos, according to the SEC's later allegations, led investors to believe its analyzer could run comprehensive blood tests from finger drops of blood.

These claims share a shape. Each is an extraordinary claim, each rests on figures that are hard to see behind, and each has people positioned to check it who are not the people presenting it. That combination is where a careful reader begins.

02Ask for the raw measurement

A headline figure is a conclusion built from a measurement and a calibration. In the 1989 cold fusion case, the heat claims depended on calibrating each cell, and the review panel said that this method is intrinsically prone to overestimate heat. It judged that an error of a few percent in the assumed coefficient would invalidate nearly all reported excess heat.

The Madoff case shows the same habit with returns. A 2005 memorandum noted that the largest one-month loss was only 0.55 percent and called a Sharpe ratio of 2.55 unbelievably high. Those are summaries too, but they point back to a monthly series that anyone can recompute, so ask for the worst month, the number of losing months and the source of each figure.

The OPERA neutrino claim of 2011 needed the same discipline. Its 60.7 nanosecond early arrival depended on clocks at CERN and at the Gran Sasso laboratory, about 730 kilometres apart, so the time figure meant little until the clocks, cables and survey were known.

03Ask who checked, and with what

An independent check is the step that is most often skipped. The SEC Inspector General found that the agency never verified Madoff's trading through an independent third party, which the Inspector General's expert called the most critical step. The examiners did not seek records from the Depository Trust Company, and they asked Madoff for copies of his own records instead.

The same question applies to a laboratory test. The Theranos laboratory's own 2016 submission said it had failed troponin proficiency testing in both events of 2014 and could not determine the cause. It then voided all patient test results from March 2014 through October 2014. A proficiency test compares results with reference values, so an unexplained failure is a check the method had not passed.

Ask which records a claim depends on, and whether the person presenting it controls them. A check that relies on the subject's own records tests the subject's word rather than the underlying activity.

04Test the ordinary explanation first

Before accepting an extraordinary explanation, list the mundane explanations and find a test that separates them. The LK-99 flake is a good example. A magnet can hold a small magnetic flake partly up. A study by Guo, Li and Jia attributed half-levitation in small flakes to soft ferromagnetism combined with shape, and reported no Meissner effect and no zero resistance in its samples.

The second LK-99 question concerned the sharp drop in resistance. Zhu and colleagues argued that the drop was most likely due to a structural transition in copper sulfide, an impurity in the samples, at about 385 K, and they measured pure copper sulfide as well. Jain pointed to a copper sulfide transition at 104 °C and argued that a valid test needed LK-99 made without copper sulfide. The sources gave different temperatures, and the case does not resolve the gap.

Interpretation: the mundane explanation does not always win. In the Madoff case, the claim that a skilled and unusual strategy produced steady returns was the ordinary-sounding reading, and the record did not support it. What separated the readings was whether the trades existed, and the examiners did not verify that with an independent party.

05Who has a reason to look?

Outside analysts and regulators are often the first to test an unusual claim, and their reasons differ from those of the people presenting it. Harry Markopolos, a chartered financial analyst and certified fraud examiner, sent the SEC several versions of his complaint about Madoff between 2000 and 2005. The third version, presented in October 2005, described about 30 red flags and called a Ponzi scheme highly likely.

Regulators bring a different kind of check. The CMS letter of July 2016 described a 2015 survey that found the Theranos laboratory out of compliance with five CLIA condition-level requirements, and it imposed sanctions that included revocation of the laboratory's CLIA certificate. According to the SEC's allegations, the company's claims about military use and about 2014 revenue were false.

Regulatory silence can also mislead. The Inspector General found that investors were reassured because the SEC had examined Madoff or entities that did business with him and found no evidence of fraud. A previous examination that found nothing is not a test of the claim in front of you.

06Red flags to raise before replicating

Some warning signs can be checked before a replication begins. The first is publicity ahead of method. The 1989 cold fusion announcement of 23 March reached the public before the paper, which is dated 10 April, so early judgments rested on press descriptions rather than on full methods.

A second flag is missing products. The 1989 panel found that groups claiming excess heat did not find fusion products at levels that matched the heat. A third is an unanswered question about the chain of measurement. The OPERA abstract did not by itself show whether the timing chain had been fully checked, and the later corrections traced the anomaly to two candidate hardware faults with opposite effects.

Good replication questions follow from these flags. What is the raw measurement, and who can recompute it? Which known impurity, cable, calibration or instrument could produce the same effect? What single observation would separate the ordinary explanation from the extraordinary one? The answers decide whether a replication is worth its cost, and they decide what the replication should be designed to test.

Terms in this note

Extraordinary claim: A claim that would overturn well-tested knowledge or that ordinary processes do not easily produce. It needs stronger evidence and more independent checks than an everyday result. Glossary
Mundane explanation: An ordinary cause, such as an impurity, a loose connector, a calibration assumption or a magnet, that could produce the observation without the extraordinary claim being true. Glossary
Red flag: A feature of a claim or its evidence, such as an unexplained failure, missing products or unavailable records, that calls for a specific check before the claim is accepted. Glossary

Test the idea

The case files behind this note

The open record

Sources & further reading

Written for Faultory by Yunus Emre Vurgun, with AI-assisted research and drafting. Editorial standards · Report an issue.