Radar Perene / Archive / science

Reproducibility audit: rerunning your own scripts to see whether the result repeats

◦ Index methodology v2.2 (working papers with DOI). See the methodology.

Science

Ask anyone who works with data whether their own numbers are reproducible and the answer will be yes — with the serene conviction of someone who never checked. Belief in one's own reproducibility is nearly universal; testing it is rare. Between the version of the code that generated the table and the version sitting on disk today, months of small adjustments pile up, and the memory that "this runs" grows ever more similar to faith.

A reproducibility audit is the exercise of re-executing one's own pipeline from scratch — from raw data to final result — to verify whether the published numbers repeat. Done once, it is an event; done in a cycle, with a recorded date, it is a discipline. The difference between the two is the difference between tidying the house for visitors and living in a tidy house.

Why results stop reproducing on their own

Nobody breaks their own reproducibility on purpose. It rots through banal mechanisms: an updated dependency that silently changed a behavior, an intermediate file left behind that masks the failure of the step that used to generate it, a parameter fixed by hand on a Friday and never returned to the code, a data source that revised its own historical series without notice. Each of these events is too small to be noticed on the day it happens. Their sum is a pipeline that produces numbers — just no longer the same ones.

The uncomfortable point: none of this shows up while nobody reruns everything. The production system keeps working, the reports keep coming out, and the divergence sleeps where nobody looks — in the distance between what was published and what today's code would generate.

The house's cycle

The house treats this risk with a deliberately unglamorous instrument: an internal reproducibility audit cycle, in which its own pipeline is re-executed from scratch and the result is confronted with what is published. The most recent cycle was closed in July 2026, with a record — it was not the first, and the design is that of a recurring habit, not a commemorative event.

The scripts, the outputs and the content of what was verified stay on the bench, as in every text of this trail. What this text records is the existence and the cadence: the house reruns its own production, in a closed cycle, with a date. Anyone who has attempted the exercise knows that this short sentence hides considerable work — and that the permanent temptation is to postpone it, because it is never urgent and the cost of not doing it is invisible until the day it stops being.

Reproducing is not the same as being right

One distinction prevents wrong expectations: the reproducibility audit verifies whether the number repeats, not whether the number is true. A study can be perfectly reproducible and methodologically fragile — the pipeline re-executes without error and produces, once again, the same poorly grounded conclusion. The question about the soundness of the finding belongs to another instrument, the robustness test, which alters the conditions on purpose to see what survives. The two disciplines complement and do not replace each other: robustness interrogates the finding; reproducibility interrogates the workshop.

In the field's broader picture — which we cover in Reproducibility in finance —, the fragility starts earlier than one imagines: many published results do not reproduce even with the original data and code. The internal audit attacks exactly that first layer, the cheapest to verify and the most embarrassing to fail.

What a closed cycle buys

The habit pays off on three fronts. Grounded confidence: when the house states that a published number is reproducible, the statement carries a verification date, not a hopeful tone. Early detection: the rot described above is found while it is small and cheap to fix, not years later, fossilized under layers of new code. And a disciplining effect on the present: whoever knows the pipeline will be re-executed from scratch writes the pipeline differently — fewer manual adjustments, fewer steps that only exist in someone's memory.

That last effect is probably the most valuable one. The audit does not only improve the verified past; it changes the behavior of whoever builds the future.

Frequently asked questions

What is a reproducibility audit, in one sentence?

It is the complete re-execution of one's own pipeline, from raw data to final result, to verify whether the published numbers repeat — done in a recurring cycle and with a recorded date.

What is the difference between reproducibility and robustness?

Reproducibility asks whether the same procedure generates the same number; robustness asks whether the finding survives when the procedure is changed on purpose. A study can pass one and fail the other, in any combination.

What if the rerun does not match what was published?

Then the audit worked: the divergence exists either way — the audit merely decides whether it will be found by the house or by a reader. A found divergence becomes a correction with a record, and the protocol follows the same rite described in this trail for any finding that does not hold.

Isn't this simply good software engineering?

There is overlap, and it is welcome. The difference in emphasis: engineering asks whether the system runs; the audit asks whether what was published under the house's signature is still what the system produces.

---

Continue the trail: Seven criteria that do not change: what stays fixed when an index changes version →

House reading: the numbers this discipline protects are the ones appearing, every day, in the Diário and the Atlas.

Setting up an audit cycle like this for another team's production — from design to the first rerun — is a service the house provides on request.