Reproducing a Repo's Headline Benchmark Before You Trust It

You found a library whose README claims a large speedup, and before you build on it you need to reproduce that number yourself. This guide is for engineers who have to return a defensible answer, not a feeling. It carries one performance claim all the way through a reproduction attempt - the clone, the missing harness, the version pin, the mismatched hardware, the raw runs, and the wrong turns - and ends with a three-way verdict you can attach evidence to.

Most guides on this topic stop at a checklist and tell you benchmarks "can be misleading" in the abstract. That is true and useless. What you need is to know the exact forks where a claim falls apart, and what the surviving number actually proves once it does. The short version, borne out by every documented case below: the claim usually dies at provenance, long before you ever measure anything.

What a reproducible benchmark actually requires

A benchmark is reproducible when a second engineer can rebuild your result from recorded facts, not when it is merely fast. Multiple current acceptance-criteria specs converge on the same minimum list, so this is not a matter of taste.

To call a performance number reproducible, you need, at minimum, to record exact software versions, the git SHA, and the hardware environment; define your timing boundaries (setup versus solve versus transfer); specify warmup, repetition count, and seed policy; and report dispersion, not just a single figure. A fuller protocol adds dataset split and sample IDs, the model or checkpoint and runtime identity, batch and concurrency, sequence lengths, the memory-measurement method, and preprocessing overhead. Gernot Heiser's canonical list of "systems benchmarking crimes" names two offenses that matter here directly: missing platform specification, and reporting relative numbers only.

Translate that into two questions you ask of any README claim:

• Can I regenerate it? Is there a committed script that produces this number, with the versions and rig it was produced on?
• Can I compare it? Is the number absolute and bounded, or is it a bare multiple with no baseline, no hardware, and no date?

If the answer to either is no, you already have most of your verdict.

The claim I am going to reproduce

The worked example here is a real pattern, not a toy. The repo I will carry through ships a README that cites a GPU speedup, specifically numbers measured on an RTX 4090. That is the claim. The job is to land a verdict of holds, hardware-inflated, or unreproducible, and to leave behind artifacts a colleague could rerun.

Those three verdict categories are this guide's framework, not a named public standard, so treat the labels as load-bearing editorial. What is documented is what each verdict needs attached to it: raw result artifacts with hashes, normalized JSON or CSV, a reproduction command, median and p95, toolchain, OS, machine, and commit. The verdict is my judgment; the evidence is not negotiable.

Step one kills most claims: provenance first

Before you install anything, find the script that produced the number. In the documented repo failures, the breakage was absence, not a wrong algorithm, which means your first twenty minutes decide most verdicts.

Three real cases make the point. In one repo (kvcompress #13), the README cited specific GPU numbers, but the only committed result files were labelled CPU-only and measured on macOS, and the reproduction script imported a module that did not exist. The claim was not wrong so much as undefended - there was no artefact in the tree that produced the headline figure, and the one script that pretended to was broken. In another (LogosLang #73), the headline speedups "came from ad-hoc runs": no benches folder, no cargo bench target, no script that re-measures them. In a third (ThinkRL #134), the benchmark entry point pointed at a module that did not exist.

Here is the concrete provenance pass for my RTX 4090 claim:

1. grep -r "4090" and grep the exact numeric figure across the tree. If the figure appears only in README.md and nowhere in a results file or a script, that is the first red flag.
2. Find the results directory. Open the files. Read the labels. In the kvcompress pattern, the committed results said CPU-only while the README said GPU - a README-versus-results mismatch you catch only by reading both.
3. Find the harness. Look for a benches/ folder, a cargo bench target, a scripts/bench.sh, or a Makefile target. If none exists, the claim has no re-measure path and you are already at a verdict.

For my example, assume provenance survives: there is a benches/ target, a committed results file whose labels match the README, and a documented command. That is the minority case, and it is the only case where the rest of this guide applies.

Pin the build to the exact commit

Pin to a full commit SHA, not a version tag. A full SHA guarantees the exact code is used even if the tag or the code is moved later, which is the whole point of reproduction.

Resolve the README's claimed version to a full SHA, clone, and check out that worktree. Then confirm git rev-parse HEAD matches the pinned SHA and write down the versions of everything you installed in a small pin manifest. CI, if you have it, should fail when the manifest disagrees with what is actually resolved.

The subtle trap is that there can be two commits, not one. The benchmark suite and the runner that drives it may live at different revisions. In one documented case (graalvm #9972), between a pinned suite commit and a runner commit one week later, the analysis code "moved substantially." If you record only one SHA, you cannot tell which code produced the number. So record both benchmarkSuiteCommit and runnerCommit, and make every execution build its source worktree from that exact commit, never from the launcher's current HEAD.

claim_source: README.md line, quoted verbatim
benchmarkSuiteCommit: <full 40-char SHA>
runnerCommit: <full 40-char SHA, or "same as suite">
library_version: <tag> -> <resolved SHA>
baseline_version: <name> <version> -> <resolved SHA or package digest>
dataset: <name> :: <split/sample IDs> :: <digest>
seeds: corpus=<n> native=<n> provider=<n>
toolchain: <compiler/runtime + version>
os: <distro + kernel>
cpu: <model> | gpu: <model>

Note the separate seed fields. A reproducibility spec (sts2-harness #125) requires typed fields for corpus, native, and provider seeds and tells you to never substitute one for another. Collapsing them into one "seed=42" is how two honest engineers get different numbers and blame each other.

Reconstruct the harness, then do a sanity pass

Rebuild the environment from the manifest until the benchmark runs one clean trial, then run it on hardware close to the README's rig before you trust your own. Reproducing the author's number first tells you whether your setup is sane; skipping it means every later deviation is ambiguous.

For the RTX 4090 claim, this means finding a 4090 - a cloud instance will do - and running the committed harness there first. One of two things happens. Either you land inside the claimed band, and you have a working baseline to carry to your own hardware, or you do not, and you now have a concrete deviation to explain rather than a vague "I couldn't get it to work." Both are progress. The deviation is data.

This is also where you confirm the measured unit is the product path and not startup. In the aside-codemode #42 case, a tool reported "5.6x slower," but re-measuring on the real path turned it into 13 ms versus ripgrep's 16.7 ms. The reported gap was the harness: process startup and transport counted inside the unit of work. Before you report any ratio, check that the thing being timed is the thing you would actually run.

Run the measured protocol, and know your harness's defaults

Run with warmup discarded, multiple repetitions, fixtures and IO excluded from timing, and record every raw run so you end with a distribution and dispersion rather than a single number. The default settings of your benchmarking tool decide, on their own, whether a claim looks proven or noisy.

This is the quiet trap. The same claim, measured honestly, can read either way depending only on which harness the author reached for. Criterion.rs ships at 100 samples with 3 seconds of warm-up, 5 seconds of measurement, 100,000 bootstrap resamples, and 0.95 confidence, and it reports confidence intervals and standard deviation by default. Google Benchmark ships with repetitions defaulting to 1, which means no dispersion at all unless you explicitly ask for more.

Practitioner specs put the floor at five repetitions and insist you report p50, p95, and p99, not a mean. So if the README's number came from Google Benchmark at defaults, the author never had dispersion to begin with, and "reproducing" a single run tells you nothing. Set repetitions to at least five, discard warmup, and record the raw values. For my 4090 claim I run the harness with explicit repetitions on both the author's rig and my own, and I keep every raw number in a file I will later hash.

Once you have a distribution on your hardware and the author's, you have done the honest part. Finding the people who do this work well, rather than the people whose title merely says they do, is its own problem - and one worth solving before you staff a reproduction effort.

The same-hardware control is the deciding test

To separate algorithmic speedup from hardware, put the library and the baseline it claims to beat on one machine and measure the ratio there. A cross-machine table in a README proves nothing about the algorithm, because performance is strongly dependent on factors unrelated to the code.

The documented control is explicit: include at least one same-hardware comparison to separate algorithmic from hardware effects. The deep-learning efficiency literature says the same thing in stronger terms - ensure the methods being compared use identical hardware and software to the greatest extent possible, because timing is strongly dependent on hardware and software factors that have nothing to do with the algorithm. A 2026 replay study fixed 64 vCPUs and 256GB of memory across four cloud profiles and varied only the processor, precisely so the processor was the only free variable. The GSO benchmark sidesteps absolute speedups entirely by comparing generated optimizations against expert implementations in the same environment.

Why this is the deciding test, not a nicety: the same config does not travel. One speculative-decoding setup delivered 3.5x on NVIDIA H100s and only 1.1x on AMD MI300X with no code change. A README that shows a big number on one accelerator and a baseline on another is measuring the gap between two machines and calling it an algorithm.

For the RTX 4090 claim, the control is concrete: run the library and its stated baseline on the same single GPU, measure the ratio, and compare that to the README's cross-machine figure. If the same-hardware ratio collapses toward 1.0 while the README showed a large number, the gain lived in the hardware comparison, and the verdict is hardware-inflated regardless of how fast the absolute number looks.

Watch the quieter hardware confounds too. On virtualized hosts, being allocated 2 vCPUs says nothing about how fast those cores tick, so pin the CPU governor, record the scaling driver, and confirm your vCPU entitlement is not the variable moving your numbers.

The reproduction procedure end to end

Here is the whole attempt as an ordered procedure. Each step has a done condition so you know when to move on, and the early steps are the ones that end most attempts.

How this goes wrong: the failure modes

Most false verdicts come from trusting an artifact that looks authoritative but is not. Each failure mode below has a specific check and a documented case, because a signal is only useful when you also know what it looks like when it lies.

• Harness absent. Lies as: a results folder that looks authoritative but holds synthetic or CPU-only numbers under a GPU README. Check: grep for the exact figure in-tree and confirm a committed script regenerates it (kvcompress #13).
• Ad-hoc provenance. Lies as: a plausible number with no re-run path, so a future regression is invisible. Check: is there a bench target or a scripts/bench.sh; if not, the claim is unverifiable (LogosLang #73).
• Wrong commit measured. Lies as: you reproduce a number from current HEAD and assume it matches the release. Check: build the worktree from the pinned SHA and record runner versus suite commit separately (graalvm #9972).
• Single-run trust. Lies as: one fast run inside normal variance reading as a win. Check: require repetitions and dispersion; Google Benchmark's default of 1 gives you none.
• Cross-hardware inflation. Lies as: a README's cross-machine table reading as algorithmic gain. Check: run the same-hardware control; 3.5x became 1.1x across accelerators with no code change.
• Frequency and turbo confound. Lies as: faster numbers that are really a higher clock or a turbo window. Check: pin the governor, record the scaling driver, and confirm vCPU entitlement is not the moving variable.
• Harness-not-algorithm gap. Lies as: startup or transport overhead counted as the unit of work. Check: align the measured unit to the product path; a 5.6x gap was the harness (aside-codemode #42).
• Drifting baseline. Lies as: a stale "2x faster" that only held against an old reference. Check: date the comparison and re-run the current baseline; a 2x can fall to 1.4x against a faster successor with no code change.

That last one is worth dwelling on, because it means even a correct number ages out. An undated multiple is structurally unreproducible: the reference it was measured against has moved on, and you have no way to reconstruct what it beat. When a README gives a bare multiple with no date and no baseline version, the honest verdict is often unreproducible even if the code is excellent.

Grade the claim and attach the evidence

Assign one of three verdicts and attach the artifacts that let someone else check your work. The verdict is your judgment; the evidence is what makes it defensible.

• Holds. The same-hardware control shows a real algorithmic gain within the claimed band, with dispersion that does not swallow it. Attach the control ratio and the distributions.
• Hardware-inflated. The absolute number may be real on specific hardware, but the same-hardware control collapses the ratio, or the README compared across machines. The gain is the machine, not the code. Attach both the cross-machine and same-hardware numbers.
• Unreproducible. The harness is absent or broken, provenance is ad-hoc, or the claim is an undated bare multiple with no baseline. Attach the provenance findings - the missing script, the mismatched results folder, the grep that came up empty.

Staffing this work and keeping it current

Reproduction is a specialist skill, and the title that names it is noisier than it looks. In Refolk's index, "Performance Engineer" resolves to 1,119 current profiles in the US, 243 in the UK, and 84 in Germany - but the label conflates software performance engineers with motorsport ones, where the top UK and German employers include Red Bull Racing, Aston Martin F1, and TOYOTA RACING. Treat those counts as upper bounds for the software reader.

The practical consequence: sourcing on title alone over-counts the people who can actually reproduce a software benchmark. You want people who publish dispersion numbers and have handled cross-accelerator work, not people who share a job title with a pit-lane engineer. Describing exactly that in plain English - and excluding motorsport - is where Refolk removes the friction that a title filter cannot. When you need a reviewer for a contested claim, asking for "performance engineers in Germany who do software tuning, exclude motorsport" returns the right people the first time.

To keep a verdict current, re-run two things on a schedule: the same-hardware control against the current baseline version, since a 2x can drift to 1.4x against a faster successor with no code change, and the pin manifest, since the upstream tag may have moved off the SHA you measured. A verdict without a date is a claim that will quietly expire. Attach the date, the commit, and the baseline version, and your reproduction stays defensible long after the README's number stops being true.