Autonomy is earned, not configured
I have a build pipeline, autoloop. I point it at two codebases: the app it’s building (Patina), and itself. Same exact loop: triage an issue, decompose it, implement, run tests, review, and open a PR. If PR has issues, the pipeline attempts to fix the PR. The only thing that changes is which repo it’s working on.
So I did an experiment. I let autoloop pipeline build both for a couple of months and watched the numbers. If graduated autonomy means anything, it’s a system that earns trust from correctness metric instead of being handed over. The two targets shouldn’t earn the same amount.
The setup
Nothing dramatic. Each repo gets an autoloop.toml and a set of GitHub issues. The config names what the pipeline is not allowed to touch:
# autoloop.toml: paths the pipeline can't modify
protected_paths = ["autoloop/", "autoloop.toml"] # Patina
protected_paths = ["autoloop.toml"] # autoloop, building itself
The list is short. When autoloop builds itself, the rest of the boundary lives outside the code it can edit: the agent runs under permission rules that deny pushing, merging, resetting and deleting, and nothing reaches main without my merge. That matters more when the pipeline is building itself. When it builds Patina, the worst case is a bad feature. When it builds autoloop, a bad PR getting merged could break the thing that’s supposed to keep it safe. Same tool, very different blast radius.
The metrics
Every implementation gets logged: did it pass on the first attempt, how many retries, did I have to fix it by hand after merging. That last one turned out to be harder to measure than it sounds; more on that below. Both dashboards are public: Patina and autoloop.
“First attempt” is a specific bar. When a PR comes back broken (merge conflict, failing test, lint error), the pipeline doesn’t just stop and wait for me. It auto-fixes: rebase on main, resolve the conflicts, run the formatter, have the model repair the failing test, re-check. But only for a capped number of rounds. First-attempt success means it cleared everything without needing a repair loop to run. The retries are the pipeline fixing its own work, which is another kind of autonomy, and one I deliberately set a limit boundary. Self-repair that can loop forever isn’t a feature. It’s a way to burn tokens on a change that was never going to be successfully implemented.
Building Patina: 35 implementation runs, 86% first-attempt success. Most issues resolved in a single pass (30 of 35 one-shot; the rest took a retry or two).
Building itself: 104 implementation runs across 70 issues, 60% first-attempt success.
Notice that gap. The pipeline is noticeably better at building the product than at building its own infrastructure. Self-modification is where it’s least reliable, and that’s exactly where the boundary is tightest: the agent can’t touch its own config, can’t push or merge, and every change waits for me. The guardrails aren’t just decoration. They sit precisely where the measured failure rate is worst.
I’ve left out the human-edit rate. The way it was measured counted any later change I made to the same files, months later included, as a fix to the pipeline’s work. That’s too loose to quote, and it’s being fixed.
Why is this graduated autonomy?
Most people treat autonomy as a switch: either on or off. That’s rather incorrect. How much we should trust the agent depends on what it’s building, and it has to earn that from what it actually ships to production in that repo.
In practice: nothing auto-merges yet. A couple of small modules do sit at 100%, but on 1 to 6 runs, below any realistic volume I’d trust handing autonomy to the agent. A rung isn’t earned by one lucky pass. To auto-merge, a target has to clear three bars at once: over 90% first-attempt success, under 5% of PRs needing a hand-edit, and at least 10 clean merges behind it. The pipeline stays gated until all three clear, per target, on its own recorded history.
module first-attempt* runs auto-merge?
src/patina/ 93% 14 no
src/patina/adapters/ 83% 6 no (too few to call)
src/autoloop/ 85% 62 no
* per PR opened. Runs that failed outright never open a PR,
which is why these sit above the repo-wide rates.
When I first published this, the adapters row said 36% and I called it the hardest code to build. That number was wrong: a bug counted PRs with no recorded run as failures. Fixed, it’s 83% over 6 runs, which is too few to call either way. That’s the whole point: the system doesn’t get to feel confident, and neither do I. It gets a number, and the number has to be right and big enough to mean something.
The number decides, not me
I don’t set the trust level. The build history does. Did the tests pass the first time. Did I have to fix it by hand. Has it done enough of these to mean anything. None of that is my opinion of how it’s doing. It’s just what happened.
Same idea as the last post. The observer watches and writes down what actually happened; this is what you do with those records: you let them decide something. The model doesn’t get a vote on whether I trust it. The count already answered.
There’s a stricter version in Patina’s code: a 0-to-6 ladder that promotes and demotes itself. It’s going to be on a post. This one is the plain running version. One pipeline that earns more trust building my app than building itself, and knows the difference because it counted.
What this doesn’t prove
2 months, 2 repos, one operator reviewing every merge. The rates are calibrated to my code and my taste, not anyone else’s. The dashboards’ headline numbers are lifetime totals; the trend rows underneath show each period on its own, and those are the ones to watch. The numbers shift as the system improves or regresses. A flattering cumulative average would hide exactly the trends that matter.
But the pattern holds: point the same builder at two problems and it earns a different amount of trust on each. Autonomy that doesn’t grow nor shrink by what you’re actually good at isn’t earned. It’s just switched on.
target first-attempt runs auto-merge
patina 86% 35 gated
autoloop 60% 104 gated (protected paths active)
Correction (2026-10-26): the first version of this post quoted autoloop’s EVAL.md while it had two bugs. Triage runs were counted as implementations (#400), and the per-module table counted PRs with no recorded run as failures (#432). The figures above are recomputed from the cleaned run history as of the original publish date: Patina 65 → 35 runs and 78% → 86%; autoloop 78 → 104 runs and 59% → 60%; adapters 36% → 83% on 6 runs. I also corrected the config example and the closing block, which showed a command that doesn’t exist, and removed the human-edit rates (#450). The EVAL.md dashboards linked below now include these fixes and newer runs, so their numbers differ from this post.
Code: github.com/Sanctum-Origo-Systems/patina · autoloop · Eval: Patina EVAL.md · Autoloop EVAL.md · Previous: The observer files the bug · The $110/month pipeline