Radar Perene / Archive / science

k nearest neighbors: how the house chooses which crises to compare with today

◦ Index methodology v2.2 (working papers with DOI). See the methodology.

Science

When a commentator compares the current moment with 2008, he has made two choices he almost never declares: he chose how many episodes to compare with — one — and he chose which. Both choices shape the conclusion before any analysis begins. Had he chosen 2015, the moral of the story would be different; had he chosen five episodes instead of one, there might be no moral at all — just a fan of diverging outcomes, which makes a worse headline and better information.

In a formal historical analogy, the choice of how many and which episodes enter the comparison is a methodological decision declared in advance — not an intuition exercised afterwards. Statistics calls the family of methods "k nearest neighbors": one defines how to measure proximity between two moments, one defines the number k of neighbors that enter the count, and from there the selection of episodes is a consequence, not a choice. The house's historical analog engine belongs to that family, and what this article describes is the principle — the values used in practice are bench matter.

Intuition picks the neighbor that confirms

The method exists to correct a documented vice of memory. Left free, intuition does not seek the most similar episode — it seeks the most available one: the most recent, the most traumatic, the one that props up the thesis already held. It is neighbor selection by narrative convenience, and it explains why the same market week is "just like 2008" to the pessimist and "just like 2016" to the optimist. Both are right about some resemblance and neither declared a criterion — the argument is unsolvable because it is not about data; it is about which analogy each wants to be true.

Formalizing the choice inverts the order. First one decides what counts as proximity: which variables describe a "moment" and how the difference between two of them is measured. Then one decides k. Only then does one look at the archive — and the neighbors that come out are the ones that come out, including when they contradict the story one wanted to tell. The method does not eliminate the choices; it moves them to before the result, where the result cannot contaminate them.

The k dilemma: few lookalikes or many strangers

The k carries a genuine dilemma, with no free solution. A small k selects only the very close episodes — but few cases produce an unstable split of outcomes, where each individual episode weighs too much. A large k stabilizes the count — but at the cost of admitting ever less similar neighbors, until the "analogy" becomes an average of everything with everything, which says nothing about the specific moment.

This dilemma is an old acquaintance of statistics in different clothes: it is the trade-off between variance and bias, the same one that makes confidence intervals widen when the sample shrinks. Few neighbors: a faithful portrait of what is rare, with an enormous margin. Many neighbors: a narrow margin around a question that is no longer yours. Choosing k is the declared administration of that trade-off — and the word that matters is declared: the house's k is fixed by design, not adjusted case by case until the output pleases.

What "close" means is the heaviest decision

Before k, there is an even more influential decision: the proximity ruler. Two moments can be neighbors by the behavior of interest rates and strangers by the behavior of the currency; alike in market mood and opposite in fiscal condition. There is no proximity in the abstract — there is proximity in the chosen variables. Change the ruler and the neighbors change; change the neighbors and the split of outcomes the analogy returns changes with them.

That is why the ruler is the part of the method demanding the most discipline: chosen after looking at results, it becomes an instrument of confirmation wearing the costume of mathematics. In the house's design, the variables that describe a moment and the way distance between moments is measured are fixed beforehand and documented internally, and the temptation to "adjust the ruler because the neighbors came out strange" is treated as what it is: the sophisticated version of picking 2008 because 2008 is convenient. Which variables and which metric is content of the draft study; the commitment to fixing them in advance is what the house can state in public.

Frequently asked questions

What k does the house use?

The value is an internal design decision of the study and is not public. What is public is the property that matters: k is fixed before any query to the archive, not calibrated afterwards to improve the answer.

Do older episodes count less than recent ones?

That depends on the declared ruler, not on preference. An old episode can be the closest neighbor of the current moment; excluding it for age would smuggle in the hypothesis that the distant past carries no information — a hypothesis that deserves a test, not a decree.

What if the archive holds no genuinely close neighbor?

That is a legitimate answer of the method — and one of the most valuable. When the current moment does not resemble anything on record closely enough, the honest analogy is to declare the void, not to force the least distant neighbors. The trail returns to that case two articles ahead.

Isn't "k nearest neighbors" a machine-learning technique?

The family of methods is the same one used in statistical classification, decades older than the current label. In the house's use it does not "learn" or predict: it selects comparable episodes and returns the outcomes that followed them.

---

Continue the trail: Range, not point: how to read the output of a historical analogy model →

House reading: the episodes that are candidates for neighbors live in the Atlas; the moment one wants to compare, in the Diário.

Applying a formal neighbor selection to a specific episode — ruler and k declared — is bench work the house conducts on consultation.