Radar Perene / Archive / science

Beating the trivial benchmark: the test every house index must pass before it exists

◦ Index methodology v2.2 (working papers with DOI). See the methodology.

Science

Every new indicator arrives in the world with an unfair advantage: it gets compared with the void. Whoever builds an index of sentiment, risk or liquidity tends to present it against the alternative of having no index at all — and against nothing, anything looks like information. The honest question is a different one, and it is uncomfortable: what does this index say that a dumb ruler would not?

A trivial benchmark is the simplest imaginable alternative for the same task — the historical average, yesterday's value repeated, a one-line rule anyone would apply without any model. An index only justifies its own existence if it informs more than that ruler; beating it is not an indicator's final trophy, it is the entry requirement.

Why the dumb adversary is so hard to beat

Trivial rulers are humiliating because they carry, for free, almost everything a series has of its most predictable behavior. "Tomorrow looks like today" gets a lot right in any persistent series. "The usual average" gets a lot right in any series that oscillates around a center. A sophisticated indicator has to pay for its own complexity: every additional parameter is a promise that extra information is being captured — and promises are collected against the cheapest adversary, never against the void.

The forecasting literature has known this embarrassment for decades: elaborate models lose to naive rules with a frequency nobody enjoys publishing. The reader who only sees indicators compared "with the market", or with nothing, is seeing the flattering half of the exercise.

What the house did with its own indices

The house keeps five indices in production, and none of them was exempted from this ruler. In an internal adversarial evaluation exercise, all five were deliberately confronted with trivial alternatives — the one-line rulers any skeptical reader would propose first. The spirit of the exercise was that of a devil's advocate with full access: if an index does not inform more than the dumb rule, it does not deserve the complexity it carries.

The exact design of the confrontation and the numerical results stay on the bench — they are working material, not shop-window material. What can be said is what matters to the reader: the exercise was run, it was run against all five at once, and it was run with the declared intention of tearing them down. An index that survives that kind of grilling is not proven; it is authorized to keep existing until the next round.

The test comes before, not after

There is an inversion of order that separates this habit from the decorative use of benchmarks. In the decorative use, the index is born, gains an audience, and only then does someone ask whether it beats the dumb ruler — and by then too much reputation has been invested to accept a "no". In the practice of this house, the trivial comparison is the first hurdle, not the last: an index candidate that fails to cross it never gets a name.

That ordering is directly related to the honest testing of strategies, which we explain in Honest backtesting, step by step: in both cases, the right adversary is defined before looking at the outcome, and the flattering comparison is treated as the vice it is. The difference is the object — there, a strategy; here, an indicator's right to exist.

What this filter does not guarantee

Beating the trivial ruler does not turn an index into an oracle. It can inform more than the dumb rule and still inform little; it can win in the tested period and tie in the next; it can be useful for one reading and irrelevant for another. The filter eliminates the most numerous category of indicators — the ones that only look informative because they never faced the right adversary — and that is all it promises. The rest of the validation work continues after it, in layers other texts in this series describe.

Frequently asked questions

What is a trivial benchmark, in one sentence?

It is the simplest possible solution for the indicator's own task — repeating the last value, using the historical average, applying a one-line rule — used as the mandatory adversary of any more elaborate model.

Why not compare the index directly with the market?

Because "the market" is not the real alternative of someone who would use the index. The real alternative is the dumb rule that makes the index unnecessary — and it is against that rule that the information gain must show up.

Is an index that loses to the trivial ruler always discarded?

In the house's habit, a candidate that loses does not enter production. An existing index that lost in a new round would enter review — the decision process another text in this trail describes.

Does this apply only to market indices?

It applies to any model that claims informational content: forecasts, classifications, scores. The trivial ruler changes shape with the task; the obligation to face it does not.

---

Continue the trail: When the house kills its own hypothesis: the track that did not survive validation →

House reading: the indices that crossed this ruler feed the Diário every day; the patterns that survived retesting are in the Atlas.

Putting a third-party indicator through the same adversarial grilling is work the house does on request — the conversation starts with the ruler, not with the model.

Characters: Method