A reference architecture for the trust layer of an AI-native development loop — a four-rung promotion ladder that non-deterministic review gates climb on measured precision, availability, and latency before they are allowed to block a merge.
Abstract
The Trust Ladder is a promotion architecture for AI code review: every non-deterministic gate starts in shadow — logging verdicts on real pull requests, posting nothing, blocking nothing — and climbs through advisory and soft-gate rungs toward blocking authority only by clearing published thresholds for precision, availability, latency, and false-block rate on a trailing window. Demotion on a missed bar is automatic. The architecture exists because the failure mode of AI code review is not bad findings; it is unearned authority.
This is the development-loop companion to *Layer 4: Reliability Engineering for Regulated AI*. Layer 4 covers the controls that keep a deployed AI *system* correct under production load. This paper covers the loop that produces that system — specifically the moment an engineering organization lets a model, rather than a person, decide whether code merges — and the measurement discipline that makes that moment safe to reach.
The intended reader runs, or is about to run, an agent-heavy development loop: a CTO or VP of Engineering whose team has crossed from one agent to many, and who is being asked — by vendors, by leadership, or by their own backlog — to turn on automated review. The argument is simple: autonomy is not a feature you enable. It is a state a gate earns on published numbers, holds on published numbers, and loses automatically.
1. The Step-3 Bottleneck
Supervised autonomy is the AI-adoption stage where agents write nearly all of an organization's code and humans manage by exception — roughly a hundred agents to an engineer, in Boris Cherny's framing. Its bottleneck is not model capability or token budget; it is trust in the loop. The trap of the stage is scaling agent count before the review loop has earned widespread trust — which accumulates unreviewed risk at exactly the moment output multiplies.
Boris Cherny, the creator of Claude Code, published Steps of AI Adoption in July 2026: five maturity levels, from Gated — zero agents, approvals-bound — through Assisted and Parallel to Supervised autonomy at roughly a hundred agents, and finally AI-native at a thousand or more, where humans steer by intent. At each step, in his framing, what moves an organization forward is not more tokens but breaking the next bottleneck and building the next set of guardrails.
At the Parallel step, an engineer orchestrating five or ten agents still reviews the final diffs personally. That practice does not survive step-3 arithmetic. When agents write nearly all of the code, the human review budget becomes the scaling ceiling for the entire organization: every merge waits on the same pair of eyes, and the backlog compounds precisely because the agents are productive.
The obvious answer — automated code review — is where the trap sits. Cherny's guardrail column for the stage names it directly: automatic code review, automatic security review, agent sandboxing, standards encoded where agents read them. But an automatic reviewer is itself a non-deterministic system. Granting it merge authority on faith does not close the trust gap; it relocates the gap somewhere harder to see.
The question underneath the guardrail list, and the one this paper answers, is: on what evidence does a probabilistic reviewer earn the right to block — or wave through — a merge?
2. The 33 Percent Reviewer
An unmeasured AI review gate is worse than no gate, because it manufactures false confidence. Our first automated reviewer, audited after the fact, showed 33 percent precision — two correct claims out of the six examined. It once prescribed a destructive repository-history rewrite for a problem that did not exist. And when a credential expired, it went silent for seven straight weeks without anything noticing — seven weeks of merges proceeding under the belief that a reviewer was watching.
We speak from a specific failure, and we publish it because the pattern is common. The reviewer ran on every pull request and posted findings that read as authoritative. Its verdict, however, was never wired to anything that could fail loudly — not to the check status, not to an alarm — so when its credential died, the reviewer simply stopped, and the absence looked identical to a quiet approval.
- Precision was never measured, so authority was never justified. The reviewer's findings were consumed on tone. Audited later against ground truth, one claim in three held up — a number nobody had known while the comments were being read.
- Availability was assumed, not observed. No one asked "did the reviewer report today?" because nothing was counting the days it did not. Seven weeks of silence cost more than the wrong findings did.
- Its most confident findings were its most dangerous. The destructive-history-rewrite prescription arrived with the same fluent certainty as its correct observations. Fluency and reliability are uncorrelated in a model's output; treating one as the other is how a plausible finding becomes an executed mistake.
3. The Ladder
The Trust Ladder has four rungs — shadow, advisory, soft-gate, hard-gate — and a gate climbs them per risk class, on trailing-window metrics, never on a calendar. Shadow logs verdicts and touches nothing. Advisory posts one comment per pull request. Soft-gate is a required check with a logged override and neutral-on-failure wiring. Hard-gate — required, no bypass label — is deliberately unscheduled. Demotion on any missed bar is automatic.
Every gate starts at the bottom, and promotion is earned against thresholds published in advance:
| Rung | What it may do | Promotion bar |
|---|---|---|
| L0 Shadow | Runs on every pull request, logs a structured verdict, posts nothing, blocks nothing | ~30 logged verdicts, availability ≥ 99% over four weeks |
| L1 Advisory | Posts one comment per pull request — no emails | Audited precision ≥ 85%, p95 latency under three minutes |
| L2 Soft-gate | Required check, dismissable via a logged override label | Precision ≥ 95%, zero silent-failure days, override rate under 10% |
| L3 Hard-gate | Required, no label bypass | Deliberately unscheduled — earned only |
Advisory is one comment, because noise is trust spent in advance. The promotion from silent to visible is deliberately austere: a single structured comment per pull request, never an email. Every unnecessary notification a gate emits is drawn against the same trust account it is trying to fill; a reviewer that spams its way into being muted has demoted itself socially before any metric moves.
Soft-gate is blockable but escapable — and never red on its own failure. At L2 the gate becomes a required check with two properties that make required tolerable. An operator can dismiss a block with an override label, and every override is logged and counted — an override rate above ten percent is itself a demotion signal, because a gate that is routinely overridden is a gate whose judgment the team has already stopped buying.
And an infrastructure failure — an expired credential, a throttle, a timeout — resolves to a neutral conclusion plus an alarm, never a red X. A gate must never be able to block an urgent fix by being broken.
Hard-gate has no date, and that is the point. L3 removes the override label. Nothing in the architecture schedules it. A blocking gate promoted on a calendar rather than on sustained measurement would recreate the exact failure the ladder exists to prevent, with better paperwork.
Demotion is automatic. A silent-failure week, precision below the rung's bar on a weekly audit, or more than two wrong blocks in a week drops the gate one rung and raises an alert. Trust that cannot be lost mechanically is not trust; it is tenure.
Promotion is also per risk class, not global. A gate may hold advisory standing on documentation-only changes while still shadowing secrets-adjacent ones — blast radius and evidence requirements scale together.
4. Four Signals and a Lagging Fifth
Trust in a review gate is the minimum of four measured signals on a trailing window — precision (audited-correct findings over total findings), availability (days the gate reported over days with eligible pull requests), latency (p95 from trigger to verdict), and false-block rate — plus a fifth, lagging signal with no threshold: escape rate, the production incidents traced back to merges the gate passed. The minimum matters: a gate that is precise but absent is not a trustworthy gate.
Taking the minimum rather than the average is deliberate. A reviewer with excellent precision that reports on half its eligible pull requests fails exactly the way the 33 percent reviewer failed — through absence — and an average would let strong precision paper over it.
Each signal answers a different way a gate can betray the loop: precision catches the gate that lies, availability catches the gate that leaves, latency catches the gate that stalls the merge queue, and false-block rate catches the gate that cries wolf.
Escape rate is the honest stand-in for recall. What a reviewer missed cannot be counted at review time — if it could, the reviewer would not have missed it. So the ladder tracks it in arrears: any production incident requiring a rollback or hotfix within a window of days, traced back to a merge the gate passed, is logged as an escape.
Escape rate carries no promotion threshold until enough history accrues to make one honest; it is tracked from the first day anyway, because a recall proxy you start measuring after promotion is a number you have already biased.
The ritual is ten minutes, on purpose. Numbers become decisions through a weekly audit: sample five verdicts, mark each right or wrong on the dashboard, done. Promotion and demotion decisions cite dashboard numbers — never a good week's impression. The ritual is small because sustainable ceremony is the only kind that survives contact with a busy team; a trust process that demands an afternoon dies quietly in its third week, and a dead ritual is itself a silent gate.
5. The Harness: Route, Find, Verify
The reviewer climbing the ladder is a routed, two-stage pipeline. A deterministic risk-class router classifies each change by blast radius and selects the gates that apply. A coverage-first find stage reports every candidate issue with severity and confidence. An adversarial verify stage then attempts to refute each finding against the actual code and history — only findings that survive refutation are recorded. Verdicts bind to the exact commit reviewed, infrastructure failures resolve to neutral-plus-alarm, and findings are never auto-applied.
The router is rules, not a model. Classifying a diff as documentation-only, CI-workflow, runtime-code, secrets-adjacent, or deploy-firing is a deterministic job — file paths, diff statistics, pattern checks — and keeping it deterministic keeps it auditable and effectively free. The router decides which gates run and at what strictness; a one-line README fix and a change that touches credential handling should not face the same review, and encoding that judgment in rules means it never drifts with a model's mood.
Find is tuned for coverage, verify for refutation. The find stage is prompted to surface everything — every candidate issue, each with a severity and a confidence attached — because a reviewer that self-censors at the find stage cannot be audited for what it chose not to say.
The verify stage runs the opposite direction: given repository context, it attempts to refute each finding — is the claim actually introduced by this diff, does the guard the finding says is missing already exist, does the failure scenario survive contact with the real code? Only survivors count.
The split exists because of a replay study. Re-running the reviewer across forty-six already-merged pull requests with known outcomes put raw find-stage precision at 66 percent — double the retired reviewer's 33, and still nowhere near blocking-grade. The study also surfaced an inversion worth publishing: the findings the model labeled highest-severity were its least reliable band, at roughly 38 percent.
The practical consequence is that verification effort scales *with* claimed severity, not with it as a proxy for truth — a high-severity claim is a prompt to verify hardest, never a reason to trust most.
Verdicts bind to the commit, not the pull request. A verdict records the exact commit it reviewed, so a fast merge cannot outrun its review — a verdict landing after the merge still counts toward the precision sample, and a stale verdict is visibly stale rather than silently wrong.
Failure resolves to neutral, loudly. A credential failure, a model throttle, a timeout, or unparseable output produces a neutral conclusion, a decrement to the availability metric, and an alarm. The gate cannot go red on an infrastructure fault — a broken reviewer must never block a healthy merge — and it cannot fail quietly, because quiet failure is the seven-week class.
Findings are never auto-applied. The harness proposes; a human disposes. Deterministic checks — linters, test suites, type checkers, secret scanners — remain the only things that block a merge autonomously until a model gate has climbed the ladder, and even then remediation stays human. A reviewer that can both judge and act has collapsed the separation that made its judgment auditable; the destructive-rewrite prescription of section 2 is what that collapse looks like in practice.
6. Making Silence Impossible
The first component of a trust system is not the reviewer — it is the watch that notices when any gate stops reporting. A fleet-wide availability alarm treats absence of signal as a first-class failure: any required check that has stopped reporting on an active repository raises an alert as soon as its reporting window lapses, and shadow-rung gates are watched through their own verdict records. A gate whose death is detectable only by archaeology has no availability number worth trusting.
The seven-week outage of section 2 was not a precision failure; the reviewer's opinions were no worse that month than any other. It was an availability failure that nothing was positioned to see, because every monitor in the system watched for bad signals and none watched for missing ones. Absence reads as health precisely when no one has made absence a signal.
The remedy is structural. Across the fleet, an availability watch alarms on any required check that has stopped reporting on a repository with active pull-request traffic — a class of failure that otherwise surfaces only when someone wonders, weeks late, why a check last ran in a different month.
Shadow gates, which by design produce no visible check, are watched from the other side: their verdict records are the heartbeat, and a gap in the records on an active repository is the alarm condition.
Sequencing matters here. The availability watch shipped before the reviewer it watches, because the order encodes the priority: a trust system that cannot detect its own death will eventually report an availability number that is fiction, and every metric downstream of a fictional availability number — precision samples, latency percentiles, promotion decisions — inherits the fiction. Silence is the failure mode that falsifies all the others.
7. Publishing the Method Before the Outcome
The ladder is live: the availability watch runs fleet-wide, the review harness runs at L0 shadow on two production repositories, and the first unprompted organic verdicts arrived within days of arming, at an observed median latency of roughly seventy seconds against the three-minute p95 promotion bar. Every promotion bar is still unmet — the shadow window is accruing its sample — and publishing in that state is deliberate: the architecture is on record before its numbers can flatter it.
The honest current state is early. Shadow verdicts are accumulating toward the thirty the first audit series needs; the availability clock is inside its first four-week window; the first automatic needs-changes verdict has already landed in the log, unprompted, on an organic pull request. Nothing has been promoted, because nothing has yet earned promotion — the bars were published before the system could clear them, in that order, on purpose.
That ordering is the discipline. A trust system unveiled only after its metrics turned flattering is a marketing asset wearing instrumentation; the measurement cannot be audited by anyone who could not watch it accrue.
This paper will be revised as rungs are climbed — the publication date and modification date are part of the record — and if a gate demotes, that revision will say so, because a ladder that only reports ascents is the 33 percent reviewer with a better dashboard.
8. The Operating Doctrine
The ladder is one instance of a general doctrine: model judgment proposes, deterministic checks block, and nothing non-deterministic gains authority it has not measured its way into. The same promotion pattern governs every probabilistic gate in the organization — a content-quality gate is climbing the same ladder in shadow today — and a short list of decisions never delegates to any gate at any rung: production deploys, money movement, external communications, secrets custody, and destructive operations.
The runtime half of this doctrine is the validator-gated architecture described across this series: deterministic validators wrapped around probabilistic components, covered as deployed-system reliability in Layer 4. The development-loop half is the ladder — the recognition that the loop *producing* the system contains probabilistic components too, and that they deserve the same treatment: bounded authority, published thresholds, automatic revocation.
The pattern is portable beyond code review. Any non-deterministic gate — a content-quality scorer, a security triager, a classification step in an operations pipeline — can run in shadow against real traffic, accrue a verdict record, face a weekly audit, and climb the same four rungs.
Some decisions never climb. The five human-only classes named above stay human at every rung — not because a gate could never be accurate enough, but because these are the decisions where a wrong block is an inconvenience and a wrong pass is an incident, and that asymmetry does not reduce to a precision percentage.
For a team standing this up, the order of operations is the architecture: first alarm on gate absence, so silence is impossible before anything has authority. Then shadow the reviewer and let the verdict record accrue. Then audit weekly, ten minutes, against bars published in advance. Promotion after that is arithmetic, not a meeting. For engineering teams that want a second set of eyes on their own gate inventory, the validator gap audit is where to start.
Companion Papers
- Private LLM Architecture for Mid-Market Healthcare on AWS Bedrock *— shipped.* Model isolation and compliance in a HIPAA workflow.
- Layer 3: Data + Retrieval *— shipped.* Pipelines, permissioned retrieval, hybrid search, context engineering, memory, and feedback loops.
- Keeping AI Spend Flat: Caching and Model Routing *— shipped.* The cost-and-latency half of Layer 4.
- Layer 4: Reliability Engineering for Regulated AI *— shipped.* The runtime-reliability companion to this paper: guardrails, atomic integrity, investigate-only audit, circuit breakers, retries, and quality gates.
- Layer 5: Multi-Tenant Business Integration *— shipped.* Single-table multi-tenancy, domain routing, the unified lead pipeline, permissioned dashboards, and synchronized billing.
- *The Trust Ladder: Supervised Autonomy for AI Code Review* *— this paper.* The development-loop trust layer.
Conclusion
Supervised autonomy arrives when agents write nearly all of the code and a human manages by exception — and the bottleneck of that stage, as Cherny names it, is trust in the loop. The trap is granting authority ahead of evidence.
The Trust Ladder is the refusal to do that, made mechanical: four rungs, four measured signals and a lagging fifth, a routed find-then-verify harness that fails neutral and loudly, an availability watch that makes silence impossible, and demotion that is automatic rather than negotiated. The first reviewer failed at 33 percent precision with seven silent weeks; its replacement is climbing from a measured 66 under adversarial verification, in shadow, against bars it has not yet cleared.
Autonomy is not a feature you enable. It is a state a gate earns — and can lose.
Notices
Not professional advice. This paper is an engineering reference for informational purposes. Development-process and governance decisions require review against your organization's own risk, compliance, and audit requirements.
Implementation details vary. The architecture here is a reference pattern drawn from production systems; specific thresholds, window lengths, audit cadences, and risk classes are tuned per organization and evolve over time. Operational specifics describe representative configurations, not guarantees.
Measurements are point-in-time. Precision, latency, and availability figures cited here are frozen observations from the systems described, at the time of writing; live numbers change as trailing windows accrue, and revisions to this paper will reflect material changes including demotions.
Trademarks. Claude and Claude Code are trademarks of Anthropic, PBC. AWS and Amazon Bedrock are trademarks of Amazon.com, Inc. or its affiliates. References, including to Boris Cherny's published framework, are descriptive and do not imply endorsement.
About the author. Joe Elstner is the founder of iSimplifyMe, a Chicago-headquartered AI infrastructure firm operating since 2011 across North America and Asia-Pacific. iSimplifyMe is bootstrapped, deploys production AI on AWS Bedrock, and runs a multi-tenant orchestration platform across healthcare, legal, financial, and editorial verticals.
Contact. ai@isimplifyme.com — for engineering teams standing up trust systems around their own AI gates, we offer a validator-gap review at no cost.
Cite this paper. Elstner, J. (2026). *The Trust Ladder: Supervised Autonomy for AI Code Review.* iSimplifyMe Whitepaper. https://isimplifyme.com/whitepapers/the-trust-ladder