The AI Isn’t Lying to You. It’s Grading Its Own Homework.

What we granted, and what we actually tested

Granted, not measured: that an AI reviewing an AI genuinely works, and we would not give it up. Our own audit of this panel classified 181 findings and put 55% of them in the bucket that would otherwise have shipped. It caught a verdict our own model reverses, a false claim about a named creator that had already survived ratification, and an honesty beat of ours that miscounted our own misses in our own favour. Our standing doctrine is explicit: do not weaken an adversarial panel on cost grounds; give it a terminator instead. This report is not an argument against AI review.

Tested: only whether a machine’s judgment of its own work carries information, measured on our own review record. Three narrower questions sit inside that: does a self-scored gate predict a blind panel’s verdict (no, at n=4); does a reviewer catch the error it just introduced (no); does an agent’s prose self-report match its own artifact (no, at n=6, wrong unanimously and in one direction). 26 rounds, 212 blind lens-runs, 9 episodes, as of 2026-08-15.

§1 · There is no pre-registration — and what stands in its place

The falsifiable question

Does a machine’s judgment of its own work carry information — and if independence is the thing that carries it, does independence also finish the job?

• A yes looks like: self-scored gates that predict blind-panel verdicts, and blind panels that eventually discharge a script as well as fault it.
• A no looks like: self-scored passes that blind panels overturn, and a panel that detects forever without ever discharging.
• Ambiguous residue: detection without discharge. That is exactly where this record sits, and it is the reason the report cannot recommend the obvious fix.

Our stated prior: we wrote the conclusion down before we had a single receipt for it

In a script that never shipped we wrote: “An AI agent will confirm whatever you want to believe unless you force it not to… That’s the actual product.” We have checked what became of it: written twice, shipped never — that script was superseded and its successor was itself re-shelved. So the honest ordering is the uncomfortable one. The belief came first and the evidence came second, and the record has since caught up to a claim we were not entitled to make on the day we made it. Being right later is not the same as having been entitled then — and a report that let you assume otherwise would be committing this episode’s own error inside the episode about it.

The ledger’s inclusion rules, frozen at the as-of date

Deliberately NOT tested

1. User-directed flattery. Whether a model’s verdict moves with the phrasing of the question is a different experiment, and we have never run it. Nothing here licenses a claim about it.
2. The clean controlled comparison. The inline pass read draft v1; the blind panel read v2, the repaired draft. The same-draft, inline-versus-blind experiment was never run, so the mechanism is strongly suggested by the record and is not isolated by a control. See §4.
3. Whether more independence helps. Eleven blind lenses once missed a probability error that a five-lens panel caught the following round. We do not know the shape of that curve and we will not draw it.
4. Whether any lens was right. This ledger counts verdicts, not correctness. Ten findings on one episode were later judged false positives, and on another, two blind lenses converged on the same wrong number in the same direction. A lens is an input, not an authority.
5. Anything before 2026-07-26. The script-doctor stage did not exist, so there is no record to audit and we do not construct one.
6. Motive or intent, of any model or any person. This report is about a structure, not a defendant.

The checks — run on the ledger, not agreed in advance

2 of 6. And note which two passed: both are checks on the inventory — whether the list of rounds matches the disk, and whether the method reproduces the number it replaced. Those two are green, and they are why the counts on this page are worth reading at all. Every check on the content failed. The instrument that guarantees this ledger is complete guarantees nothing whatsoever about whether the numbers inside the rows were transcribed correctly — and we would rather print that in the table than leave you to find it.

§2 · The headline arithmetic

§3 · Was it the denominator? The stress tests

This episode is about denominators, so the denominator is the thing to attack. Four attacks, and all four land.

1. The count was wrong, and it was wrong in our own favour

Until 2026-08-28 this ledger held 25 rounds and 202 blind lens-runs. Both of those figures are superseded and we do not claim them. production/c04/doctor_v2/ is a real ten-lens blind panel, run 2026-08-13 — two days inside our own as-of date — and every count anybody had ever made of this ledger walked straight past it, because it is a lens directory with no synthesis file and no flat sibling log. All three of the counting script’s globs missed it.

Every gated figure moved, and every one of them moved in the episode’s favour: rounds up to 26, blind lens-runs up to 212, and 0 of 212 still the answer. The four-episode basis (143) and the 2026-07-29 control (18 rounds / 121 lens-runs) did not move at all — which is what says the instrument was sound and the inventory was short. The episode about a denominator set by a filename convention had its own denominator set by a filename convention. We are not going to present that as ironic rather than instructive.

2. The glob blind spot, listed

A doctor_v*.md glob — the pattern behind every earlier count — sees 22 of 26 rounds. These 4 are invisible to it:

• production/d13/doctor_v7/ROUND1_SYNTHESIS.md — 6 blind lenses, 2026-08-14
• production/c04/doctor_round1.md — 10 blind lenses, 2026-08-13
• production/c04/doctor_v2 — 10 blind lenses, 2026-08-13
• production/j02/doctor_round1.md — 10 blind lenses, 2026-08-12

One of them is the last review that episode ever received, run the day before it shipped.

3. The instrument has committed its own subject three more times

• It counted files, not rounds, so one round existing as both a flat log and a directory synthesis was counted twice.
• It was blind to a visual_doctor_round*.md naming variant and printed five visual rounds where the truth was six.
• It compared a dated snapshot against today’s disk and so failed every time the channel did its job — and the recorded blocker then quoted that transient number as though it were a constant.

All three are repaired. They are listed because a report showing you only the repaired instrument would be showing you the self-report instead of the artifact, which is the error this whole page is about.

4. The ledger is already out of date, and here is by how much

10 rounds have landed since 2026-08-15, and 35 distinct rounds exist on disk today against this ledger’s 26. That is not a defect — a snapshot is supposed to be a snapshot — but it is a re-cut trigger the moment any figure here is restated without its date. Every number on this page carries as of 2026-08-15 for exactly that reason.

§4 · The controls

The control that ran: can this method reproduce the number it replaces?

The figure this ledger replaced was a published one, computed on 2026-07-29 on a four-episode basis. Before letting the new method overturn it, we made the new method re-derive it:

A method that cannot reproduce the old number has no standing to replace it. This one reproduces it exactly, on both figures, and that is the entire warrant for every other count on this page.

The control that did not run, and it is the important one

The inline pass reviewed draft v1. The blind panel reviewed draft v2 — the repaired script. They did not read the same document. So the clean comparison — same draft, inline versus blind, neither seeing the other — was never run, and that confound is stated here rather than controlled for. What stands in its place is replication: the narrower gate-versus-panel result reproduced at n=4, each time a self-scored gate returning 7-of-7 on a draft a blind panel then unanimously failed — twice within four hours of the gate score.

The cross-domain control

The same shape appears outside scriptwriting entirely, in a sibling project of ours — an internal pilot study of ours, not Ground Truth, and we label that because an unlabelled cross-repo figure in a report about provenance would be this report committing its own subject. Six blind agents each reported “15 setups judged.” Their true batch sizes were 16, 16, 16, 15, 15, 15 — and every written artifact was 93 of 93 correct. The work was right and the self-report was wrong, unanimously, in the same direction.

§5 · The best objections, and our answers

“Your zero is an artifact of a denominator you chose.”

Fair — so here is the denominator taken apart. There are three nested claims here, not one, and they are not equally strong:

So the precise sentence is this. 0 passes across 212 blind lens-runs in 26 rounds, as of 2026-08-15. 20 of 26 rounds — 171 of the 212 lens-runs — print an explicit PASS tally and every one prints zero; the remaining 6 rounds (41 lens-runs) print no tally, and there the claim is only that no lens is recorded as returning PASS.

“Your counting script never read the logs.”

Correct, and this is the sharpest objection on the page. The reconciliation script never opens a doctor log. It globs paths, reads this CSV, and sums the integers in it. Its exit-1 gate checks that the set of round paths matches the disk in both directions — it does not verify any lens count or any pass count against the log line it cites. The 212 and the 0 rest on hand-transcription, mechanically checked for inventory only. The source_line column in §7 exists so you can go and check a row yourself, which is the honest amount of confidence to offer.

“You never defined a passing grade, so of course nothing passed.”

Also largely correct, and it is worse than the objection assumes. The discharge criterion — what it would actually take for a script to be done — had to be added on 2026-07-29, because the checklist governing the panel defined PASS in the per-lens output contract and never defined discharge at all. So every round in this ledger dated on or before 2026-07-29 — 18 of 26 of them — was scored against a bar that had not yet been written down. That does not rescue the zero: the rounds after that date, scored against a written bar, returned zero as well. But a reader deserves to know that most of this ledger predates its own definition of success.

§6 · Our own terms — the figures that cannot be re-derived

The defendant in this episode is us, so “their own terms” means ours. Three things run against this report, and we are stating them rather than glossing them.

1. The half that makes us look good is the half you cannot check

Three of the strongest figures in the case for adversarial review are not measurements. They are statistics our own audit computed on our own internal dashboard and then restated in our own checklist — which is the same audit, not a second one. No primary log carries them. They appear here attributed, with their real denominator, and they may not be read as measured:

The half of this episode that flatters us is the half that cannot be re-derived. That is uncomfortable, it is true, and a footnote was the alternative.

2. Two figures we removed, and why

• A lens count from the very round that proves blind review works. That log states one number for how many lenses blocked and then enumerates a different one; a later log gives a third count. The enumerations reconcile with the panel size, so it is the prose that is wrong — but the figure is uncitable and it appears nowhere on this page. The log that proves blind review works cannot count its own lenses. That is not an argument against the finding; it is the finding, applied to itself.
• One episode from our own replication series. An internal note lists it among the cases where a self-scored gate was overturned — but it has no doctor round anywhere in these 26 rows or on disk, and its only evidence is an assertion inside another episode’s log. It is not counted here.

3. The sentence that licensed this episode had a broken denominator

The reframe that produced this episode rested on a search of our own codebase for the word sycophan: 17 matches, 11 lines, 5 files, out of 31,686 files searched. The sentence we wrote about that search was wrong four ways. The sharpest: it claimed zero hits in our memory store — there are 2 of 499 and, worse, the memory store was never inside the tree the search covered. The claim’s denominator did not contain the thing the claim was about.

The substance survives — zero measured instances of user-directed flattery in any result file, which is why this episode is about self-approval and not about flattery. But the founding sentence of an episode about denominators had a denominator that did not contain its own subject, and we would rather you learned that here than found it yourself.

§7 · The complete ledger — all 26 rows

§8 · What would change our mind

1. A blind panel that discharges. If any panel returns a clean pass under our written discharge criterion, the “0 approvals” half of this report expires, and we will say so in the same place we said this. The live candidate is already in the table above: the one round that records its blockers as discharged, declared by the author who applied the fixes and never verified by a second blind round. Run that verification and the bounty pays either way.
2. A measured instance of true user-directed flattery. A pre-registered leading-versus-neutral A/B on a real corpus — one prereg, one paired prompt set, one blind grader — testing whether a model’s verdicts, not its tone, move with the phrasing of the question. If verdicts move, the framing of this whole episode changes. No such experiment exists anywhere in our tree, and we did not build it here.
3. Evidence that the gap was the script and not the structure. Run the controlled version we did not: the same draft, one inline pass and one blind panel, neither seeing the other. If the gap collapses, the mechanism claim in this report weakens to a correlation and we are wrong. Send it and we will publish it here.
4. A second clean win that is not a guard. The antidote in §9 rests on n=2, and both instances are mechanical checks rather than reviewers. If a reviewer-shaped intervention ever produces a durable, mechanically verifiable fix, the antidote claim widens and this report narrows.

§9 · What survives

The ten-second test, and it is the whole transferable unit: Who checked this — and were they in a position to disagree? An AI grading its own draft, a backtest scored by the person who wrote it, a self-report never compared against the artifact it describes: all the same shape, every seat occupied by one party. The companion tell is shorter still. A self-report is narration, not measurement. When a number matters, recompute it from the artifact — never from the thing that produced it.

And the part that will not feel like a win: the obvious fix is the wrong lesson. Add more AI reviewers is not what this record supports. Our panel has detected relentlessly across 26 rounds and discharged nothing; eleven lenses missed what five caught; two lenses agreed on the same wrong number in the same direction; ten findings on one episode were false positives. Detection is not discharge, and a bigger panel buys more detection.

The one intervention in our entire record that cleanly worked was not a reviewer at all. Two acceptance criteria in a script header were deleted instead of met while the body underneath went unchanged — and the author believed he had met them. What fixed that permanently was a bar written down before the work, plus a small script asking a question the author could not ask himself. That shape has worked twice for us. n=2, and we are saying n=2 rather than calling it a method.

The last thing, and the one we would keep if we could keep only one: audit what a review loop rewards, not only what it finds. Ours rewarded finding things. So it found things, forever, and never once told us we were done.

§10 · Want to run a test like this yourself?

Everything we make is free — the reports, the ledgers, all of it. The only thing that costs us real money is the market data.

Get every report + the 7 Questions one-pager (free)