Radar Perene / Archive / science

Confidence interval: what a number says — and what it withholds

◦ Index methodology v2.2 (working papers with DOI). See the methodology.

Science

Every historical market average travels through the news like a passenger with no luggage: "stocks returned such-and-such percent a year", full stop. The number looks solid because it arrives alone. But any average computed over a limited past carries a second piece of information, almost always left at the check-in counter: the precision with which that number was measured. Two identical averages can hide opposite realities — one nailed down by a century of regular data, the other guessed from two turbulent decades. The reader who receives only the number has no way to tell the ruler from the guess.

A confidence interval is the range of values compatible with the observed data, given the method used: instead of a single number, one declares the margin within which the true measure would fall in the great majority of repetitions of the same exercise. It is the number accompanied by its own uncertainty — the luggage the headline checks and never claims.

The concept is taught in any introductory statistics course and yet almost never appears where it would matter most: in everyday market reading, where averages, "typical" returns and probabilities circulate naked.

What the range says

The intuition needs no formula. Whoever measures the same thing many times — an index's average return, the frequency of a pattern — obtains slightly different results with each sample, because each sample is a cut of chance. The confidence interval answers the honest question: given the size and variability of what was observed, how far could the true value reasonably sit from the measured value?

Three practical consequences follow, and all three contradict newsroom habits. First: short samples produce wide ranges — and financial series, even those spanning decades, are short relative to how much they swing, so that celebrated average returns carry embarrassingly wide margins. Second: a difference between two numbers is only informative if it exceeds the imprecision of both — "fund A beat fund B" can be pure noise. Third: when the range around an estimated effect includes zero, the data cannot distinguish the effect from nothing at all — and that is, quietly, the situation of a good share of the patterns announced in market research.

What the range does not say

The confidence interval is also the victim of a misreading so widespread it deserves its own paragraph. A "95%" interval does not claim there is a 95% chance the true value sits inside that specific range — it claims that the procedure generating such ranges hits the target in roughly nineteen out of every twenty repetitions. The difference sounds pedantic and is not: the first reading treats the range as prophecy; the second, as the declared performance of an instrument. Instruments fail once in twenty — by construction, and without announcing which time.

There is a direct kinship with the trail's neighbor: the same "one in twenty" that defines the significance convention is what p-hacking exploits by repeating tests in silence. Whoever understands the range as instrument performance understands why running it twenty times and showing the best result destroys the guarantee.

How the house writes uncertainty

This house handles the problem through an editorial route: uncertainty goes into the sentence, not into the footnote. When an essay in the archive describes the spread of historical outcomes of a regime, the preferred form is plain frequency language — "nine out of every ten episodes stayed below such a mark" — because it carries the margin inside its own syntax, demanding no technical vocabulary from the reader. And when the estimate permits no claim at all, the house rule is to say so in as many words.

The most public case of that discipline is a study of the house's own: the working paper The Tactical Ânima Index asked whether extreme readings of a house indicator offered a tactical edge — and the published conclusion records that the edge under test does not hold. It is the language of someone who looked at the whole range instead of falling in love with its center: an effect whose uncertainty swallows its own size is not a reportable effect. In numbers: the study is public on Zenodo, with an active DOI and the adverse conclusion in the body of the text — verifiable by any reader.

Frequently asked questions

Are a confidence interval and a margin of error the same thing?

Almost. The margin of error is half the interval's width — the shorthand that election polls popularized. The interval declares the full range; the margin, the distance from center to edge.

Does a wide interval mean the study is bad?

No — it means the available data support little. The honest wide range is superior to the illusory single number: it states exactly how much is still unknown. What deserves suspicion is the short-sample study with a range that is too narrow.

Why do market reports rarely show intervals?

Wide ranges devalue headlines — "expected return somewhere between a sizable loss and a sizable gain" sells no report. The omission is usually commercial, not technical: the uncertainty was computed and left in the drawer, or never computed.

With 95% confidence, can I act as if the true value is in the range?

The range describes what the data support; what to do with it involves costs, risks and context that statistics does not decide. This article describes the tool — the decision belongs to whoever makes it.

The range measures uncertainty inside a stable world. But what about when the world itself swaps rules mid-sample — and yesterday's average starts describing a regime that has already ended? That is the subject of the next step: Regime switching: when the rules change mid-game →

House readings: today's note, in the Daily · the precedents, in the Atlas.

Computing the honest range around a specific number the reader has in hand is workbench material — the kind of exercise the house conducts on request.