Does Backtesting Really Work? Here Is the One Thing to Check.

What we granted, and what we actually tested

Granted, not measured: that the backtest a viewer is being shown is almost certainly true — real code, real data, it reproduces, and nobody faked it. The failure named here has nothing to do with lying, and we are not testing whether anybody lied. We also grant that a criterion written after the fact can still be perfectly honest. It simply cannot be evidence, because it cannot lose.

Tested: only our own corpus. 14 lab directories under validation/ as of 2026-08-15, 13 of them carrying a pre-registration, about 143 registered items enumerated from the pre-registration side rather than from whatever the results happened to discuss, and 27 rows hand-scored with a line reference on both sides. Zero new computation and no market data: this is a file-reading census of our own paperwork.

§1 · The pre-registration — and what it cannot prove

The falsifiable question

Across every lab in this repo: how many priors and goalposts have we registered, how did they score — and when we deviated, did the deviation run in the direction that flattered us?

• A yes looks like: at least 60% of direction-assignable non-HIT rows understate our own case, and the overall hit rate lands inside the registered 65–85% band.
• A no looks like: flattering deviations outnumber understating ones, or the hit rate falls outside that band. We wrote down what a death would mean before the count: a hit rate above the band would say our priors are written loose enough that they cannot lose, and a flattering direction would say the discipline is being practised as decoration in our own lab — which is the exact thing this episode condemns.
• Ambiguous residue: the registration never said which set of rows it would be scored against. That is where the direction prediction actually landed, and it is a defect in our registration rather than a property of the corpus.

Our stated prior: our deviations would cluster toward understating our own findings

It split, and the split indicts the registration rather than rescuing it. On the 27 hand-scored rows the direction bar clears at 62.5% (5 understated against 3 flattered) — over a 60% bar by two and a half points, close enough that one row moving flips it. Add the labs' own post-publication corrections and unregistered deviations and it collapses to 35.3% (6 of 17).

We did not register which set of rows this would be scored against, so we are not permitted to pick the tier that scores better. Both are published in §5. That is an unbounded prior — the precise defect one of the labs in this very table had already written down as a lesson — and we committed it in a prior about our failure to bound priors.

One more thing we owe you about this prediction: we do not claim this was a blind prediction. It generalises a finding one of these labs had already published about itself, and the pre-registration says so, in its own contamination section, written before the count. It is still a real test — that lab's finding was about that lab, and there was no reason it had to hold across the corpus — but it was not blind and may not be described as though it were.

The primary configuration (locked)

Deliberately NOT tested — named in advance

1. Any outside creator's pre-registration practice. There is no external specimen in this report and there is not supposed to be one: the only lab whose pre-registrations we can open, quote and score line by line is ours. We looked for a qualifying outside specimen and did not find one, and inventing a defendant to fill the slot would be the retrofit our own rules forbid. There is no defendant here but us.
2. The two channel-performance pre-registrations at the root of the validation folder. They are not lab pre-registrations. The exclusion was registered before the count rather than discovered after it.
3. Whether any prior was truly written before its run, corpus-wide. That is the version-control question, and it is answered per lab: §6 answers it for the 6 registrations the episode leans on, and this lab does not clear it for itself.
4. Whether our priors were any good. A hit rate is a forecasting score, not a rigor score. A lab can hit every prior it registered and still be worthless.
5. Re-running any lab, or re-verifying any lab's underlying figures.

The goalposts — all required to proceed

2 of 4. Two hits, one miss, one split — and the miss and the split are printed above the hits, here and in the lab file, deliberately. The clean failure is the last one: we guessed the priors convention was a recent addition and that fewer than half the labs would carry a block. It is 13 of 13. The convention is older and more uniform than we assumed, and the real gap in this corpus is not registering — it is scoring what was registered.

§2 · The headline arithmetic

§3 · What the instrument could not see

The instrument here is a file reader, so the stress test is a question about reach: what could it not see?

The mechanical count is a floor, not a census, and it is published as a floor. The enumerator finds 92 registered items by pattern-matching labelled bullets, numbered lists and bold table rows inside a detected block. The hand pass finds about 143 (112=12 · b01=27 · held lab 1=11 · d03=11 · d05=8 · d16=18 · held lab 2=25 · held lab 4=31). The gap is not an error in either: several labs register much of what they register as free prose inside a paragraph, and no regular expression is going to find that.

The one green the mechanical pass produced is narrow, and must not be over-read. Its registered-but-never-mentioned check came back clean on 12 of 12 scorable labs. That check asks only whether a named label appears as a token somewhere in the results file. It cannot tell mentioned from scored, and it cannot speak at all to prose-form registrations. Every unscored prior below was found by hand, not by that check. A green gate answers only the question it measures.

At least 9 registered items nobody ever scored — the ones we found

The corpus moved underneath the count, and that is disclosed rather than smoothed over. A concurrent session was writing this repo while the census ran: HEAD advanced 0a05136 to 575c72b in the first four minutes, and one lab's results file did not exist at the start of the enumeration and did exist at the end. That lab's row is a snapshot of a lab mid-build. Everything on this page is dated 2026-08-15 and is a snapshot, not a current count — the validation folder has grown since, and re-running the census today would answer a different question about a different corpus.

§4 · The controls

The enumerator has a negative control, because a guard only ever seen green is untested. Its self-test removes a registered label from a results file and asserts the diff goes red, then asserts the untouched corpus stays green for that same label. It passes both. So the mechanical check works — it is simply asking a smaller question than the finding needed, which is the paragraph above.

This report has one too. Running its generator with --selftest perturbs a single scored row and asserts the integrity gate refuses to write the page. That gate is what stands between the headline above and the table below: it recomputes the outcome tally, the direction tally, the hit rate and all four verdicts from the 27 rows, checks them against the counts the lab file already published, and refuses to emit a byte if they disagree. On a report about results that cannot lose, a page that could not reproduce its own headline from its own ledger would be self-refuting.

Independent corroboration, read back from the labs' own published scorecards: 4 hits, 1 miss; 6 hits, 1 near-hit, 1 UNSCORABLE, 0 clear misses; Prediction scorecard: 6 hit / 1 partial. Those lines corroborate the hand scoring; the 20-of-27 figure itself is assembled from the §7 rows and not from these scorecards (one of them belongs to a lab with no row in the 27), and each was opened and re-read against its own file rather than carried forward from memory.

§5 · The best objection, pre-registered and answered

“The people grading these priors are the people who wrote them.” That is the right objection, and there is no way to answer it fully from the inside. Three things are done about it, and none of them is a promise to be careful.

One: every row is evidenced on both sides. All 27 rows in §7 carry a pre-registration line reference and a results line reference, printed so the check is mechanical rather than a matter of trust. Until the lab repository is published (§6 says why it is private today) those references are our attestation, which is a weaker thing than what this page asks of everybody else; the CSV below carries every row, so the arithmetic at least is yours to check now.

Two: the direction column is declared a judgement, in advance. The pre-registration says the flattered/understated axis is read off quoted text and is never a computed quantity. It is labelled an opinion because it is one.

Three: we were not allowed to choose the set of rows that scored better. Here is the split, both halves.

The 9 rows Tier 2 adds, each verified against a line:

8 of the 9 run in the direction that had flattered us. That is the sentence worth more than either percentage: our corrections are not random-signed. When a published result of ours moved after publication, it moved toward the version that had made our position look stronger. One of these labs says it plainly in its own results file — that the correction moved the result in the product's favour, and that we published a verdict our own corrected lab does not support.

§6 · Our own terms — there is no defendant but us

There is no outside specimen in this report, at any level, and that is by construction rather than by omission. This channel's standing rule is that the practitioner whose work is under test gets named. Here nobody outside is under test: the subject is our own registration corpus. We did go looking for an outside specimen — somebody who publishes a timestamped bar and then publishes a loss against it — and the nearest candidate applies its correction after the search rather than before it, which is the whole distinction this report is about. Naming somebody anyway, to fill a slot, is the retrofit our own rules forbid. So the slot stays empty and the bounty in §8 stays open.

This ledger carries the same defect it scores other labs for, and said so before the count. Its pre-registration was written at 2026-08-15 17:52 PDT, against repo HEAD 0a05136, before any priors block or scorecard had been opened for counting. But no commit lock exists for it: the authoring session had no commit authority, so the pre-registration and the results were not separated by a commit, and version control cannot witness the ordering. What stands behind the timestamp is a written attestation, not cryptographic evidence. That is exactly the limitation this report flags on one of the labs in its own table, and it is stated here rather than discovered later. Filling the gap with a hash would be the defect this episode exists to name, committed on the page that names it.

And two sessions built this lab at once. A second session registered a competing pre-registration for the same question two minutes after this one, and deliberately left it unrun. Two registrations and one corpus is a real problem; that registration's blind-arm design is better than this lab's; and reconciling them is a merge neither session should perform unilaterally. It is recorded, not resolved.

Question three — where is the clock? The 6 registrations this episode leans on, read from version control

Our work lives in git. Every commit carries a timestamp and the exact files that went with it. A commit date can be set by whoever writes it, so this is not a notarised clock — it is an ordering, and a history that cannot be quietly rearranged once somebody else holds a copy. The rack below was read from the log by the generator that writes this page, not typed, and the build refuses if what git says stops matching what the episode says. Commit hashes are printed; file names are not, because counts are the claim and one registration commit’s subject line names a person this report has no business naming.

4 of 6 have a clock behind them, and the timestamps line up to the second. And here is the part that costs us: our lab repository is private, so not one of those 4 is a link you can open today. You have the hashes and you have our word, and that is a weaker thing than what this report asks of everybody else. On the other 2 we cannot prove the order at all, in either direction: the registration, the model and the results entered history in a single commit, at the same second, so the ordering exists only in the file itself and in our telling you about it. The case this episode leans on hardest — the prediction we lost on camera — is one of those two.

Envelope two was then edited 5 more times over the next 4 days (b09f1ff, 0a4e68d, b11bc18, 499fe45, bf3f0c8), every one of them after the results already existed. Each edit is in the history with its own timestamp, which is the only reason we can tell you about them. A bar you can rewrite is still a bar, but only if the rewriting is as visible as the writing.

Envelope four is the study whose registered bar caught a sizing defect in our own engine; it has its own episode and its own report, here. It is the strongest ordering we have and the one furthest out of your reach: that repository is not going to be published at all.

Envelope one is the lab whose kill our own errata reversed (§5). Its bar is as checkable as anything on this page, and it still did not save the verdict, because a bar can only check the number it was handed, and the number it was handed was wrong. A timestamp is the floor, not the ceiling.

§7 · The hand-scored ledger — all 27 rows

§8 · What would change our mind

1. Somebody who publishes a timestamped bar and then publishes a loss against it. We feature them by name, on camera, as the counterexample this report could not find. It takes all four: a verifiable pre-result timestamp; a published loss against that criterion; a criterion bounded on both sides; and a real funnel, so the bar had something to cost.
2. Evidence that pre-registration costs accuracy, not just comfort — a demonstration that bars fixed in advance produce systematically worse decisions on the same data than post-hoc analysis does. That is a real argument against the discipline and we would run it.
3. This ledger re-run against the blind arm the concurrent session registered. That registration covers the labs this pass had already observed and registers predictions over the remainder. A blind re-score that contradicts the direction finding in §5 would change the closing claim of this report.
4. A registered criterion of ours that could not have failed. We had one candidate and it stopped qualifying: its own results file retracted the argument that made it unfailable, and the test now passes 9 of 1,008. Finding another one in our corpus remains an open bounty against us, and we will publish it.

§9 · What survives

The portable version is three questions, and each takes about five seconds on any backtest anybody shows you.

Where is the bar, and when was it written? Not whether the result is impressive — whether a criterion existed before the result did. If the success criterion cannot be dated earlier than the number it blesses, the number is a description, not a test.

Is the bar bounded on both sides? A criterion with an open end cannot lose in the direction the result actually lands. This is the one we failed on ourselves: our own direction prediction never said which rows it would be scored against, so it scored 62.5% one way and 35.3% the other, and we had no principled way to choose.

Which way do the corrections run? Everybody publishes errata. The question is whether they are random-signed. Ours are not: 8 of the 9 post-publication corrections and unregistered deviations we found moved toward the version that had flattered us.

And the thing worth carrying out of our own numbers: 13 of 13 of our registered labs wrote the bar down, and at least 9 registered items across 6 of them were never read back. Writing it down is the cheap half. The discipline is not self-executing — not theirs, and not ours.

§10 · Want to run a test like this yourself?

Everything we make is free — the reports, the ledgers, all of it. The only thing that costs us real money is the market data.

Get every report + the 7 Questions one-pager (free)