Where Part 8 Stopped
Part 7 built a guard that could grade my work, and Part 8 gave that guard a spine... it caught more than voice, it traveled across runtimes, and only a human held the key that could change it. But it couldn't propose anything on its own, every change still waited for me to decide on it, so the guard could say "this is wrong" and it couldn't say "here's a better version, and here's the proof it's better." For eight parts I improved the harness by hand, and Part 9 is the first time it improves itself.
I waited on the self-writing part because a loop that writes its own changes is only as safe as whatever decides which changes to keep. Get that decider wrong and "autonomous" just means autonomously getting worse in a way that looks like progress, so before I let anything propose a change, I had to build something that could say no to it.
Two Gates and a Sealed Set
How do you know a change the loop wrote is actually better? The loop itself is small, I pick a target it's allowed to touch, a cheap model proposes a change, and the change has to clear two gates before anything happens. It can't make any existing capability worse, and it has to improve at least one thing we declared we cared about, by a margin we can actually measure. Whatever survives gets checked one more time against a set of cases it has never seen and can't influence, and if it clears that set the change is kept. If it fails, the change is thrown away and the baseline stands.
A coding agent already proposes changes all day, and the two gates are the same preserve-and-extend rule the voice work used from the start. The sealed set is what turns a suggestion box into something I can leave running, as long as it never gets to teach the proposer how to pass it.
I built a toy version with three candidate changes so you can watch the gates work. Two of them look good on the tests the proposer can see and only one survives the sealed held-out, so pick one, press Run the loop, and see where it stops.
When I run all three, A clears everything. B is correct and improves nothing, so gate 2 reverts it and the held-out never runs. C sails through both visible gates because it was built to, and only the sealed set catches it, which works only because the proposer never got to learn what's in that set. The three candidates are made up for the demo, but the pieces of the loop exist and work, and there's a hand-built version of the whole shape, a compressor whose own verifier rejects it and falls back to the original if it can't prove the compression was faithful.
Why the Held-Out Can't Feed Back
Candidate C is a small version of an old problem. Any metric a system is optimized against stops being a good metric, so point a loop at a score and it'll find the cheapest way to move the number, which is rarely the same as doing the work well. The defense I trust is a set of cases the optimizer can't see or tune against, scored last, where a failure is final... no re-optimizing until the held-out finally says yes, because that's just teaching it the answers slowly.
NVIDIA's SoL-Pi work builds harness improvements this way and names the sealed, no-feedback held-out as the thing that stops the loop from overfitting itself. Salesforce's own research, SFR-AutoR&D, does the strict version for code, where the improved function has to produce byte-identical output to the original across a big matrix of cases before any speedup counts. Their published results are theirs, on their harnesses and their benchmarks, and I haven't borrowed their percentages, every number on this page is one I measured on my own targets.
So a competitor's lab and my own company's researchers both treat the check as the hard part, and after building this I stopped treating it as a detail. Writing a change turned out to be the easy half of this loop, and the hard engineering is in the check that refuses to be fooled by one. That check is where the loop's autonomy comes from, because the proposer can't see it or argue with it, and a smarter proposer wouldn't have bought me any of it.
The loop has also run for real on a governed agent I built for Part 10, where the held-out check caught a genuine gap, I fixed it, re-ran, and it went green. That's one gated improvement, and I'm not claiming compounding. A flywheel where each improvement makes the next one easier is still a hope, and SoL-Pi calls it a future direction too.
What the Loop Is Allowed to Touch
A sealed set can tell the loop a change made things worse, and it can't decide what the loop should be allowed to change in the first place, which is where custody from Part 8 picks up a second job. The loop runs free on things that don't matter much if it gets one wrong... a log compressor, a formatting pass, throwaway targets. The moment a proposed change touches the safety layer itself, meaning the checks, the trust root, and the rules about who can approve what, it isn't applied. It goes to the human at the custody line, the same line the builder couldn't cross in Part 8.
So when I call the loop fully autonomous, I mean it improves the work on those low-stakes targets while a person still owns what "improved" is allowed to mean, and it never gets to rewrite its own guardrails while nobody's looking.
The Loop Picks the Model Too
The thing the loop proposes doesn't have to be an instruction, it can be a cheaper model. Most tasks don't need a frontier brain... mapping some fields, reformatting, a routine bit of code. So the loop can propose running a route on a smaller, cheaper model, and that proposal goes through the same sealed held-out eval as any other change. It keeps the cheaper model only if it clears a bar it can't see, and it falls back to the bigger one if it doesn't.
There's a second reason to reach for a different model, and it has nothing to do with cost. When the thing being checked is my own work, the worst judge is another copy of me, because it shares my blind spots, so the check that decides what's kept runs on a different lab's model than the one that proposed the change. I measured this earlier in the series, on a small controlled set of four documents, where a same-model judge graded its own house style about twelve points too kind and the different-vendor judge didn't.
Both of these were wired and tested, and neither ran on its own. A cheaper model stays a candidate until an eval promotes it, and nothing had been promoted yet when I wrote this, so what I could do next was point the cross-vendor half at actual work and watch what it did.
A School's Enrollment Schema
The task I picked was designing the enrollment CRM schema for a small private school that wants to grow. It's the kind of work I author myself, which made it a fair test of the thing I wanted to know... does a second opinion from a different lab actually change the answer? You can click through the six steps it went through.
The two labs disagreed in three places, and the one that mattered was whether to store each family's religion, employer, and income to target them. One lab treated that data as a growth lever and the other refused it as a privacy-and-fairness landmine, and that governance catch is one a copy of me would likely have shipped right past.
Then a Rate Limiter
A reconciled design is still somebody's opinion, and with code I could end on a test instead, so I ran the loop again on a small build, a token-bucket rate limiter, which looks trivial and is full of floating-point traps. Opus wrote the spec and a starter set of tests and handed the build to a Sonnet subagent, and Sonnet's first version was good, it passed all five starter tests.
Then the code went out through OpenCode to two other labs for an adversarial architect review, OpenAI's model and xAI's Grok. Grok timed out twice, so the agent running the loop sent the second audit to Google's Gemini and kept going, without anyone touching a config file, because the only rule the harness cared about was that the reviewer couldn't come from the lab that wrote the code. The timeouts were at least partly my fault. My script gave up on Grok after three minutes, then again after five, and a day later, on a different document, I found Grok, a thinking model, spending its whole output budget thinking at the limits I'd given it.
Seven findings came back, and the easy move is to hand a list like that straight to the builder. Opus judged it instead, and every finding had to become a failing test against Sonnet's code before it earned a fix.
| What the auditors found | Who | Verdict |
|---|---|---|
| A refill rate of zero times an infinite time gap turns the token count into NaN | both labs | real · failing test |
| A NaN clock reading slips straight past the time guard | Gemini | real · failing test |
| Very small refills round away and are lost for good (two findings, one root cause) | OpenAI | real · failing test |
| A capacity too large to count down one token at a time | OpenAI | real · the fix rejects it |
| Add a tolerance so three refills of 0.3 add up to 0.9 | Gemini | dismissed · it would hand out tokens that don't exist |
The dismissed row is the easy one to get wrong, because a tolerance that lets three refills of 0.3 add up to 0.9 sounds like good floating-point manners right up until you notice it hands out tokens that don't exist. OpenAI's model also questioned how the spec described the clock, and there the code was right and the wording was loose, so the code stayed. The revised build plan came back as four new failing tests plus a fix list that was just as clear about what not to touch. Sonnet fixed those four, left the dismissed one alone, and both suites went green, five of five and four of four, and I re-ran them myself before I believed the subagent.
I was still the one driving it, one session and one build, and a rate limiter is the friendly kind of target where a test settles every argument... so this is a long way from the loop running on its own.
Reviewing the Part 10 Page
Prose doesn't have a test like that, so when the same loop had OpenAI's and Google's models review the Part 10 page and its build spec, every finding got checked first against the source where it would be true. Eight of the 24 merged findings were wrong, like reviewers treating 2026 dates as the future, and the check still caught an error I'd shipped, where the page said the employee agent runs as its own user when it runs as the signed-in user, with that person's permissions and sharing. Grok's spec review only came back after I fixed that budget and the wait. At 8K and 16K tokens it had returned nothing and at 32K it took about two and a half minutes, so I gave it a bigger budget and a longer wait, and made an empty answer fail loudly.
A review like that one is where I trust the loop least, because not every check is as hard to game as a test. Code with a differential test is the good case, since the original function is a perfect oracle, and a judge scoring an agent's answer is the weak case, because a clever loop can learn to please the judge. Where the oracle is soft, the sealed held-out is doing all the work, and I trust the loop exactly as far as I trust that set.
When the Two Outside Labs Split
That review was the soft-oracle case. The next one was a rule about prose, and it turned out to have a hard oracle after all, a big enough pile of writing by real people. One check in my voice gate rejected a draft outright if it hit any of seven pinned phrasings. Four models from three labs, Anthropic's Opus and Sonnet, Google's Gemini and xAI's Grok, had written 124 fresh drafts the gate had never seen, on generic business topics with no project context, and on 2026-10-01 a verdict Claude wrote from them, after two adversarial reviews of the measurements, said to keep two of the seven as automatic rejections and turn the other five into flags.
The verdict went out to OpenAI's model and Google's Gemini. Gemini agreed, and OpenAI's model disagreed with eight errors and 10 missing tests, so I tested each of the eight, mostly by running the candidates over the 124 drafts and 39,163 documents people wrote before 2022, public writing plus a public archive of business email, scored on my laptop. The seven phrasings rejected 59 of those human documents, and the two Claude wanted to keep still rejected 10. One narrow phrase OpenAI's model had floated rejected none of them and caught 3 of the 124 drafts. Of the eight errors OpenAI's model listed, four held up, two were partly right and two were wrong, and because it floated that phrase before anyone had opened the email archive, the archive was a clean hold-out for it. So the gate pins that one phrase, and the other seven became fix-before-shipping flags instead of automatic rejections.
Three of 124 is a small catch, and it's in-sample, from 2 of 16 topics, so I don't know yet how that phrase does on drafts it wasn't picked against. On the same 32 Opus drafts, Google's model flagged 5 and Grok flagged 16, a bigger disagreement between two outside labs than the twelve-point self-bias that sent me to another lab in the first place, though the two were measured differently. Swapping OpenAI's model in for Gemini on Anthropic-written drafts raised catches from about 28% to 52%, with none of 400 human documents rejected, and that number is in-sample too.
The subagents around those calls were supposed to be tiered, Sonnet for collecting and counting and Opus for the verdicts, and when I counted the transcripts by workflow phase, Sonnet had run 32 of the 69 collecting and building subagents on 2026-10-01, and 45 of the 315 workflow subagents that answered across 2026-09-29 to 2026-10-01. Of the 68 Sonnet overrides I'd written, those 45 ran as written and the other 23 died on an API error before answering, and I re-ran them on Opus. Everywhere I hadn't written one the run inherited Opus, so the tiering only happened where I wrote it down.
The Hook Nobody Heard
And the seal itself is kept by discipline. Nothing physically stops a careless operator from letting held-out results leak back into the proposer, the isolation is enforced by how the loop is wired, and a wiring mistake turns autonomy back into overfitting without anything telling you it happened.
I've shipped a silent failure like that myself, in the hooks of the public starter kit I share with other SEs. They printed a “result” field Claude Code doesn't read, so the guardrail hook detected a force-push to main and let it run anyway, and every warning nudge got dropped the same way. The hooks had looked fine as code, and only a headless run in a scratch repo showed it. Re-running setup after that fix upgrades any hook you haven't edited.
That's one small data point for why Part 10 puts what the harness proved into the platform's own primitives. If the part that lasts is the checks and the custody and the loop, and none of it depends on which model you rent this quarter, where should it live when a check I bolt onto my own scaffold can fail and nobody hears it? Part 10 puts it in the platform itself, and it's the last one, so go read Part 10 →