Radar Perene / Archive / science

A null result is also a result: the study that found no statistical difference

◦ Index methodology v2.2 (working papers with DOI). See the methodology.

Science

There is a study underway at the house whose central finding, so far, is a sentence no marketing department would approve: we found no significant difference. No turning point, no hidden pattern, no jaw-dropping chart. And the study remains in the pipeline — being worked on, revised and moved along like any other. For much of the market, that is stubbornness. For the method, it is the correct behavior, and the reason deserves a text.

A null result is the outcome in which the test does not find the effect it was looking for: the expected difference does not appear, or appears too small to be told apart from chance. It answers the question asked — the answer is "no" or "it cannot be affirmed" — and an answer to a well-posed question is knowledge, whichever way it falls.

Why "nothing came of it" almost never gets published

The publication system — academic and commercial — pays in attention, and attention is drawn by effects, not absences. The study that finds something becomes a headline; the one that does not becomes a drawer. That filter has a name, publication bias, and a perverse consequence: the literature that reaches the reader is an edited slice of what was tested, in which the "yeses" are over-represented and the "noes" barely exist.

The practical effect is worse than it sounds. When ten teams test the same idea and one finds an effect by luck, that one is the one that publishes — and the reader concludes the idea works, never learning about the other nine. The shelved null result is not neutral: it is the missing piece for reading the positive result correctly. Every closed drawer distorts what sits on the open shelves.

The house study, at the level that can be told

The study in question started from a hypothesis the house considered plausible — plausible enough to deserve a serious test. The test came, and the effect did not show up. What the hypothesis proposed and how the test was designed stay on the bench, by the same rule as every text in this trail; what matters here is the editorial decision that followed. There were three roads: shelve it in silence, torture the analysis until something publishable appeared, or treat the null as a finding and follow the normal maturation rite. The house took the third.

The second road deserves naming, because it is the market's most traveled: reformulating the question after seeing the data, slicing the period, swapping the ruler — until some cut delivers the effect the honest version did not. It is the vice we describe in What is p-hacking, and the null result is its greatest silent victim: almost all p-hacking begins as a null someone refused to accept.

What a well-made null informs

A serious null result does not say "there is nothing here" — it says something more precise and more useful: under the tested conditions, with the available data, the sought effect cannot be told apart from chance. That formulation carries the two pieces of information the reader of market research almost never receives: the size of effect that would have been detectable, and the uncertainty around what was measured — the territory we explore in Confidence intervals in the market. A null with those two coordinates closes doors usefully: it spares whoever comes next from testing the same thing the same way, and recalibrates confidence in whoever affirms the effect with worse data.

There is also an internal value, less visible: keeping nulls in the pipeline changes the incentive of whoever does the research. If only the positive effect becomes a text, every researcher in the house learns, without anyone saying it, that finding beats testing. If the null also becomes a text, the incentive returns to its place: the quality lives in the question and the test, not in the luck of the outcome.

The contrast the reader should find strange

Apply the absence test to any producer of market analysis: over recent years, how many times has this source published "we tested and found nothing"? If the answer is never, two possibilities remain — either every tested idea worked, which any researcher's experience contradicts, or the nulls were suppressed, and the public record is the flattering half of a larger archive. The house prefers the discomfort of publishing the "it did not work" to the comfort of an archive that only smiles.

Frequently asked questions

What is a null result, in one sentence?

It is the outcome in which the test does not find the sought effect — the expected difference does not appear or cannot be told apart from chance — answering "no" to a well-posed question.

Does a null result prove the effect does not exist?

No. It proves that, under the tested conditions and with the data used, it was not detected. A small effect can exist below the test's detection capacity — and an honest null declares that boundary instead of hiding it.

Why publish such a study, if it changes no reading?

It does change: it closes a road that looked promising, informs the maximum plausible size of the effect and spares redundant tests. What it does not change is the headline — and the headline was never this series' criterion.

When will the house study be published?

When it crosses the same maturation ruler as any other — the null outcome neither speeds up nor delays the rite. What is already public is what this text records: it exists, the central finding so far is the absence of a difference, and it remains in the pipeline.

---

Continue the trail: Multiple passes before publishing: internal review when there is no journal →

House reading: the effects that survived testing — and the ones that did not — shape what appears in the Diário and the Atlas.

Assessing whether an outside "nothing came of it" deserves publication — and what it informs despite appearances — is a methodological conversation the house has with those who seek it.