The AI Said It Followed the Spec — The Spec Disagreed

What we granted, and what we actually tested

Granted, not measured: that the AI in this story wrote the pre-registration, wrote the bar, wrote the bug, found the bug, fixed the bug and published the correction. We wrote the bug. There is no clever reviewer in this report and there must not appear to be one. The protagonist is a four-line paragraph in a markdown file.

Tested: whether a plausibility band written before the code existed caught a specific instrument defect on one study. It did: the band was breached by 2.4983 and 1.9342 in realized beta, and the pre-registration named the diagnosis in advance — sizing bug — which is the diagnosis that turned out to be true.

§1 · The locked pre-registration

Committed 6beaa55 on 2026-07-02 12:26:24 -0700, before the probe was run — the wager you cannot move after seeing the cards.

The falsifiable question

Did a plausibility bar, written down and locked before the code existed, actually reject a result that a reader would otherwise have believed?

• A yes looks like: the locked bar is breached by the first run's own printed diagnostic, and the corrected answer differs from the flattering one.
• A no looks like: the bar is inside its band on the first run, or the correction leaves the verdict where it was — in which case the bar cost attention and bought nothing.
• Ambiguous residue: the bar is breached but the corrected verdict is unchanged: the instrument was wrong and the answer survived it anyway. That is not where this landed.

Our stated prior: the option arm should cost MORE, not less

The locked pre-registration states it in its own words: “LEAPS leverage costs MORE than margin at 1.33×”, with an expected drag of 1–4% a year. The first run said the opposite by 9.85 and 14.51 points a year, and the prior is not what caught it. A violated prior only says surprising; the reader still has to decide whether to believe the surprise. The band said sizing bug, which is a different kind of sentence — it named the defect and the action in advance, so there was nothing left to argue about. The corrected drag — 1.37 and 1.43 points a year against the option arm — does land inside the prior's 1–4% range, at its low edge.

The primary configuration (locked)

Deliberately NOT tested — named in advance

1. Whether bars like this one work in general. No survey, no false-negative rate, no comparison against projects without one. One caught bug is an existence proof, not a rate.
2. A mis-sizing small enough to land inside the band. Not constructed, not tested. It is the obvious way to weaken this report and it is listed below as a standing invitation.
3. The original console output. This is a fresh run that agrees with the committed account of the incident, not a copy of what was on screen that night.
4. The wall-clock timing of the original session. Not recoverable from any committed artifact. The only timing figure here is the git window between two commits, which anyone can check.
5. A second defect the same session. It exists, it was found by ordinary debugging, and no gate caught it. No line of it survives in any committed artifact, so it is described here and carries no number, no timestamp and no chart.
6. Re-opening the live options API. The terminal was offline; the cached archive was replayed instead, which is why the control run below exists.

The goalposts — all required to proceed

3 of 5 — but the tally is the least interesting line in the table, because the row that FAILED is the finding. Row 1 is the catch: the run was rejected by a bar written before the engine existed. Row 3 is the honest company it keeps — the pre-registration listed two plausibility bars, not one, and the other also fired, on the corrected run, and was inspected and cleared. Row 4 is the kill this reconstruction was run to face, and it did not fire.

§2 · The headline arithmetic

§3 · Was it the instrument? The stress tests

What else was on the same screen. The rejected run's option arm also re-derives a worst peak-to-trough fall of 78.70% on the index fund and 89.42% on the volatile name, against the margin arm's 41.84% and 63.37%. Those are a second implausibility and they are not a second check. The pre-registration lists exactly two plausibility bars — realized beta, and implied financing — and the drawdowns are not one of them. Nothing becomes a bar retroactively, least of all on a page whose subject is a bar that was written first.

How big the error actually was, and why it did not crash. Dividing by 1.0 instead of by spot asks for 1.33 share-equivalents per dollar of equity rather than 1.33 dollars of exposure per dollar of equity. At a $250.49 spot that is an over-order of about 250×. It did not produce a 250× position, because the engine's own affordability check caught the infeasible order and capped it at all-in. So the arm quietly became spend one hundred percent of equity on deep-in-the-money calls — delivered entry exposure between 2.16× and 3.03× across the 13 entries, against a standing order of 1.33×. The check that capped the position is what made the bug survivable enough to produce a plausible-looking answer. A bug that crashes is not the dangerous kind.

Where this bar is blind. It answers one question — is realized beta inside [1.1, 1.5] — and nothing else. It catches sizing errors large enough to push realized beta outside that band. It cannot see a defect that leaves beta inside it, and it did not see the other defect in that same session, which was found by ordinary debugging.

§4 · The controls

The control that earns the reconstruction. Before the re-derived run was trusted for anything, the fixed engine was put through the identical offline harness and compared cell for cell against the committed results file: 107 of 107 cells match, 0 failed, at 1e-9 relative tolerance. The scope of those 107: six metrics on each of three arms for each of two underlyings, plus the drag, the minimum equity ratio, the entry count, and the date, expiry, strike, spot and ask of every one of the 13 entries.

The control that shows only one arm moved. The margin arm and the buy-and-hold arm are bit-identical between the two runs — every metric, both underlyings. The defect lived entirely in one of the two things being compared, which is itself the reason the output looked like a finding rather than a fault.

The control that makes an offline replay legitimate. Contract selection reads only the symbol, the expiration list, the date and the spot — it never sees position size. So both runs issue the same data requests, and they did: 13 of 13 entries are identical across the two runs on date, expiry, strike, spot and ask, with 128 cache hits and 0 misses and the network never opened.

The control on our own claim about the instrument. We say the bar was enforced by a human, not by the program. That is measured, not asserted: the committed engine and runner (160 + 70 lines) contain 0 assert statements between them and exactly 1 raise, at line 98, about a missing contract. Nothing keyed to the band. A person read a printed number and stopped.

§5 · The best objection, pre-registered and answered

“The damning numbers are a reconstruction. You are grading your own homework, twice.” It is the right objection, and it is why the reconstruction was built the way it was rather than typed from memory.

1. The engine copy is generated, not hand-edited. A script reads the committed source, asserts its two anchor strings appear exactly once, writes the copy and emits a machine-produced unified diff carrying the sha256 of both files. The whole behavioural delta is one statement; the rest of the 17 changed lines are a docstring and one import that routes reads to an offline cache.
2. The control ran first. 107 of 107 cells, against a file committed six weeks earlier.
3. The reconstruction could have killed this report and we ran it anyway. If the re-derived drag had missed the pre-registered “Drag ≤ 1%/yr (CAGR gap, LEAPS − margin): LEAPS = viable substitute” band, the committed account of the incident would have been wrong. It did not miss it: −9.85% and −14.51%, both inside, both returning the same rejected readout — against a corrected +1.37% / +1.43% once the one line was fixed.
4. What it does not recover is stated rather than glossed. Not the original console output. Not the wall-clock timing of that session. Not the second defect, which has no surviving line to restore. This run agrees with the original; it is not a copy of it, and every figure from it is labelled re-derived 2026-08-15, not the original console output.

§6 · On our own terms

Three things run against this report, and they are the reason it exists rather than a footnote to it. An episode about our own rigor is structurally an advertisement for our own rigor, and the failure mode is that you leave impressed instead of equipped.

The bar answers exactly one question. Is realized beta inside [1.1, 1.5]. That is all it asks and all it can see. A mis-sizing that lands the position at 1.45 clears it without a sound, and there was a second defect in that same session that no gate caught — found by ordinary debugging, and it carries no number here because no committed artifact records one.

The bar was not clever. It was early. Its entire power is that it was committed alone, before the engine existed, in a commit containing 1 file and 83 lines. Nothing about writing it required skill you do not have. If you cannot, from this page, write the same kind of bar for your own next backtest, the page has failed regardless of what it says about humility.

We wrote the bug. The same agent wrote the pre-registration, the bar, the defect, the fix and the correction. There is no outside reviewer in this story.

And one boundary, recorded so it cannot later look concealed. On the corrected run MU's LEAPS beta is 1.0996 — 0.00036 below the band floor. The tool printed beta to two decimals, so 1.10 is genuinely what was on screen and genuinely inside the band. The honest form: 4 of 4 reported betas print inside the band at the precision the tool displayed, and 3 of 4 are inside at full precision.

§7 · The complete ledger — all 13 rows

§8 · What would change our mind

1. A plausible mis-sizing that lands inside the band. The band is [1.1, 1.5] against a 1.33× target. Construct a units error that puts realized beta near 1.45 and the claim weakens from a bar like this catches the defect to a bar like this catches the defect at this magnitude. That is already all this page claims, but it would change which sentence carries it. This is the live one and it is cheap to attempt.
2. Evidence the bar was not load-bearing — that the beta would have been inspected anyway without a pre-registered band. As stated it is unfalsifiable, and its concrete form is already conceded above: the other defect that session, which nothing caught.
3. A defect in the reconstruction itself. The offline replay is legitimate only because contract selection is independent of position size; the control tests that empirically at entry level, 13 of 13. Show us a path that couples contract selection to position size and the re-derived rows fall — not the committed floor, which stands on git alone.
4. The pre-registered kill, still standing. Bounty one was that the re-derived drag might miss the prereg's viable band, which would make the committed account of the incident wrong. It was run and it did not fire. It stays on the books because the reconstruction is two commands and anyone may re-run it.

§9 · What survives

The transferable part of this is not that we are careful. It is that the disqualifying diagnostic was printed next to the flattering headline, by a bar written before the code, and that anyone holding any machine's backtest can demand exactly that of whoever built it.

So the question worth carrying is the one the bar is an answer to: what number would have to be true for this result to be impossible — and did you write it down before you ran? A prior only tells you a result is surprising, and you can talk yourself out of surprised. A bar with a named diagnosis attached — outside this range means a sizing bug, halt — leaves nothing to argue about, and it costs 83 lines and one commit, before you know which way the answer will go.

And the caveat that travels with it: a bar is only as wide as the question it asks. This one caught the defect it was shaped for and was blind to another the same night. A person read a printed number and stopped the run — which is a weaker-sounding claim than an automated halt, and the only one the record supports.

§10 · Want to run a test like this yourself?

Everything we make is free — the reports, the ledgers, all of it. The only thing that costs us real money is the market data.

Get every report + the 7 Questions one-pager (free)