Radar Perene / Archive / science

Five indices, one single test: why testing jointly changes the result

◦ Index methodology v2.2 (working papers with DOI). See the methodology.

Science

Evaluating indicators one at a time looks like the careful way to work — each gets exclusive attention, its own report, an individual verdict. And yet that is the most forgiving way. Five separate evaluations produce, with remarkable ease, five approvals; one joint evaluation of the same five produces something else: an ordering, exposed redundancies, and at least one embarrassment. The house chose the second way for its own indices, and this text explains why.

Testing a set of indicators as a block, under the same conditions, the same period and the same adversaries, reveals what isolated tests hide by construction: which indicators say the same thing under different names, which only shine because they were tested on the terrain that suited them, and how much of the set would survive if all were held to the best one's ruler.

What isolated evaluation forgives

The first pardon is the chosen terrain. Each isolated evaluation tends to inherit the natural choices of the indicator being evaluated — the period in which its data is best, the reference against which it was designed. None of this requires bad faith: it is the path of least resistance. In the joint test, that comfort disappears, because the conditions are one and the same for all — and an indicator that only works in its own backyard is exposed by the simple change of backyard.

The second pardon is invisible redundancy. Two indices can pass separate tests comfortably and, placed side by side, reveal that they carry nearly the same information — two names for one signal. Isolated evaluation has no way of seeing this; the question "what does this index add to the other four?" only exists when the five are at the same table.

The third pardon is arithmetic: whoever runs five separate evaluations, each with its own attempts and adjustments, accumulates chances of finding good results by luck — and none of the five accounts for the attempts of the others. The block test forces the bookkeeping to be one. It is the same principle that sustains honest backtesting: the number of attempts is part of the result, and hiding it across five separate reports does not make it disappear.

The exercise the house ran

The house's five production indices went through an adversarial exercise designed like this: all together, same conditions, same trivial adversaries, same ruler. The opening text of this trail described the spirit of the confrontation against dumb rulers; what this text adds is the joint design — the decision that no index would be evaluated on its own terrain, and that the question of redundancy among them would be asked head-on.

The individual results, and in particular the ordering that emerged — which index crossed comfortably, which scraped through —, stay on the bench. Publishing the internal ranking would turn an audit instrument into a league table, and the point of the exercise is different: ensuring that the published set informs, with no shadows between components. What can be said is that the joint design produced what is expected of it — questions the isolated tests would never have asked.

A set is a claim, not a sum

There is a conceptual consequence that goes unnoticed: whoever publishes five indices is not making five independent claims — they are claiming that there exist five distinct readings deserving their own names. That claim is testable, and the test is necessarily joint. If two of the five carry the same content, the set is inflated; if one of them adds nothing to the combination of the others, keeping it is an editorial decision, not an informational one — and needs to be defended as such.

That is the difference between a collection and a system. Collections grow by accumulation: each piece got in because, alone, it looked good. Systems grow by relative justification: each piece got in because it adds to what was already there. The joint test is the toll that separates one from the other.

What the reader can demand

Facing any house that publishes a family of indicators, the useful question is not "was each one validated?" — the answer will be yes, it will be true, and it will not be enough. The useful question is: have they been tested together, under the same conditions, with the redundancy among them measured head-on? Indicator families that never faced each other are collections awaiting an audit. The answer may exist and be internal — as it is here —, but the existence of the exercise is the minimum one can demand of whoever claims to publish a system.

Frequently asked questions

What is a joint test of indicators?

It is the simultaneous evaluation of a group of indicators under identical conditions — same period, same adversaries, same ruler —, including the measurement of what each adds to the others.

Why is the ordering of the results not published?

Because the exercise is an internal audit instrument, not a championship. The public commitment is to the existence and design of the test; the detailed score is the house's working material.

Doesn't testing jointly punish indicators with different purposes?

The design accounts for that: equal conditions do not mean equal tasks. What the joint test eliminates is the advantage of the chosen terrain — each index is still held to its own promise, but under the same weather.

Does this apply to someone using third-party indicators?

Directly. Whoever assembles a dashboard with indicators from various sources has assembled, without noticing, a set never tested as a block — and the redundancy between components from different sources tends to be larger than imagined.

---

Continue the trail: A null result is also a result: the study that found no statistical difference →

House reading: the set that crossed this test is what you read, in production, in the Diário and the Atlas.

Auditing a third-party basket of indicators as a block — with redundancy measured head-on — is a service the house designs on request.