Reproducibility
StableThe standard
A number in this archive is reproducible when a competent person with the repository and no access to the author can obtain it again. Not “the author could probably redo it”. That distinction is the whole content of this page.
What every reported number requires
| Requirement | Recorded in |
|---|---|
| An experiment record | /experiments/EXP-nnn |
| A full configuration file | archive/experiments/EXP-nnn/ |
| A pinned environment | The experiment’s environment page |
| Identified input data | A DATA- record with a version |
| Pinned analysis code | A CODE- record with a commit |
| A regeneration command | The experiment’s reproduction page |
| A success check | The same page |
The success check is the one most often missing, and the one that matters most. Without it, a reproduction attempt that completes and produces nonsense is indistinguishable from one that worked.
Pinning
| Thing | Pinned by |
|---|---|
| Code | Commit hash |
| Container | Image digest |
| Dependencies | Lockfile |
| Data | Version and checksum |
| Randomness | Seed |
A version range is not a pin. >=2.1 reproduces differently in a year.
What is allowed to be irreproducible
Some things genuinely are: a measurement on hardware that no longer exists, an external service that has changed, a model that is no longer served. These are recorded as such on the experiment’s limitations page, with what was pinned and what was not. An honestly labelled irreproducible result is usable; an unlabelled one poisons everything built on it.
Checking
Reproduce one result at random each month, from a clean checkout, without consulting memory. The failures found this way are always in the steps that felt too obvious to write down.