Why does an AI code reviewer start in shadow?

Shadow is the only rung where every property of the gate can be measured without the gate influencing anything. The reviewer runs its full pipeline on real pull requests — risk-class router, find stage, verify stage, verdict record — while posting nothing and blocking nothing. Four weeks of shadow operation establishes availability and latency and accumulates the verdict sample a precision audit needs. A reviewer that cannot run reliably for a month with no authority has no business holding any.

What happened to the first automated reviewer?

An after-the-fact audit measured 33 percent precision — two correct claims of the six examined — and it once prescribed a destructive repository-history rewrite for a problem that did not exist. The deeper failure was availability: a dead credential silenced it for seven straight weeks and nothing alerted, because its verdict was never wired to anything that could fail loudly. It was retired, and the ladder was designed so neither failure class can recur quietly.

What does it mean that the gate fails neutral?

An infrastructure failure — an expired credential, a model throttle, a timeout — produces a neutral check conclusion, a decrement to the gate's availability metric, and an alarm. It never produces a red X, so a dead credential cannot block an urgent fix, and it never produces silence, so a dead gate cannot go unnoticed for weeks. Neutral-plus-alarm is the property that keeps the availability number honest.

How are findings verified before they count?

A second, adversarial stage attempts to refute every finding against the actual code and history before it is recorded; only survivors count toward precision. A replay against forty-six merged pull requests with known outcomes showed why: raw find-stage precision measured 66 percent, and the findings labeled highest-severity were the least reliable band at roughly 38 percent. High claimed severity therefore earns the closest scrutiny, precisely because it measured as the least reliable band.

When does the hard gate turn on?

It has no scheduled date, and that is deliberate. A hard gate is earned by sustained numbers — 95 percent or better audited precision held for weeks, perfect availability, and a near-zero false-block rate at the soft-gate rung first. Nothing in the roadmap schedules it, because a blocking gate promoted on a calendar rather than on measurements would repeat the exact failure the ladder was built to prevent.

The Trust Ladder — How AI Code Review Earns Autonomy

Shadow → advisory → soft-gate → hard-gate: AI code review earns merge authority on measured precision.

Abstract

The Trust Ladder is iSimplifyMe's promotion system for AI code review: a four-rung ladder — shadow, advisory, soft-gate, hard-gate — that a non-deterministic reviewer climbs on published metrics before it is allowed to block a merge. It exists because the failure mode of AI code review is not bad findings; it is unearned authority. This dossier documents the ladder, the harness that climbs it, and the numbers that decide promotion.

Why a ladder?

Boris Cherny, the creator of Claude Code, published Steps of AI Adoption in July 2026 — five maturity levels running from Gated, with zero agents, to AI-native, with a thousand or more agents. Step 3, supervised autonomy, is the level where agents write nearly all of the code and a human manages by exception. His stated bottleneck for that step is the reason this system exists: trust in the loop.

The trap, in his words, is scaling agent count before the loop has earned widespread trust.

iSimplifyMe already runs the furniture of step 3 — scheduled agents, overnight batches, and deterministic CI gates across a fleet of more than forty repositories behind roughly a hundred live properties — with agents doing most of the writing. The missing piece was a measured answer to a single question: when the reviewer is a model, how do you know when to stop reading every diff?

What the first attempt taught

Our first automated reviewer earned the opposite of trust. Audited after the fact, it measured 33 percent precision — two correct claims out of the six examined, a sample whose very smallness is part of the story. Worse than the wrong findings was the silence: a dead credential killed the reviewer outright for seven straight weeks and nothing noticed, because its verdict was never wired to anything that could fail loudly.

At one point it prescribed a destructive repository-history rewrite for a problem that did not exist.

That failure produced the two rules everything else here follows. First, a non-deterministic gate must be measured before it is trusted — precision is a number, not a feeling. Second, silence must be impossible; a gate that can die quietly is worse than no gate at all.

The four rungs

Every AI gate starts at the bottom. Promotion is earned per gate, per risk class, against published thresholds on a trailing window:

Rung	What it may do	Bar to enter this rung
L0 Shadow	Runs on every pull request, logs a structured verdict, posts nothing, blocks nothing	None — every gate starts here; the first audit needs ~30 logged verdicts and availability ≥ 99% over four weeks
L1 Advisory	Posts one comment per pull request — no emails	Audited precision ≥ 85%, p95 latency under three minutes
L2 Soft-gate	Required check, dismissable via a logged override label	Precision ≥ 95%, zero silent-failure days; an override rate under 10% must then be held at L2
L3 Hard-gate	Required, no label bypass	Deliberately unscheduled — earned only

Demotion is automatic. A silent-failure week, precision below the rung's bar on a weekly audit, or more than two wrong blocks in a week drops the gate one rung and raises an alert.

What gets measured

Trust in an AI code-review gate is the minimum of its four signals. Precision is audited-correct findings over total findings; availability is days the gate reported over days with eligible pull requests; latency is the p95 from trigger to verdict; the fourth is the false-block rate. A fifth, lagging signal — escape rate, production incidents traced back to a merge the gate passed — is tracked as the recall proxy.

The ritual that turns those numbers into decisions is deliberately small: a weekly ten-minute audit samples five verdicts and marks each one right or wrong on a dashboard. Promotion decisions cite dashboard numbers. Nothing promotes on a good week's impression.

The harness

A risk-class router decides first. It classifies every change by blast radius — a documentation-only diff and a secrets-adjacent diff do not face the same review — and routes each one to the gates that apply. The reviewer itself is a two-stage pipeline: a coverage-first find stage reports every candidate issue with severity and confidence, then an adversarial verify stage attempts to refute each finding against the actual code and history before anything is recorded.

Only findings that survive refutation count.

The verify stage exists because of a replay study. Re-running the reviewer against forty-six already-merged pull requests with known outcomes put raw find-stage precision at 66 percent — and surfaced an inversion worth publishing: the findings the model labeled highest-severity were its least reliable band, at roughly 38 percent. Claimed severity is a prompt to verify, never a reason to trust.

Three more properties are load-bearing. The harness fails neutral: a credential failure, a throttle, or a timeout produces a neutral conclusion, an availability decrement, and an alarm — it cannot go red on an infrastructure fault, and it cannot die quietly. What's more, verdicts bind to the exact commit they reviewed, so a fast merge cannot outrun its review. And findings are never auto-applied: the harness proposes, a human disposes, and only deterministic checks block merges on their own.

Status

Every layer described above is in place today; what remains is two bars:

Rung, as of 2026-09-15: L0 shadow. Audited precision 51 percent against the 85 percent advisory bar; p95 latency 298 seconds against 180. No promotion.
The availability watch — an alarm on any required check that has stopped reporting on an active repository — is live across the full fleet. Silent gate death now raises an alert instead of waiting to be discovered.
The review harness runs at L0 shadow on two production repositories, writing a verdict record for every pull request. As of 2026-09-15: ninety verdict rows over eight weeks — on eighty-six pull requests, a re-review writing its own row — a verdict row for every eligible pull request, median latency 116 seconds, p95 298 seconds — outside the 180-second advisory bar.
Since 2026-09-15, harness v0.3.0 on both shadow repositories: the verify stage reads the source of every symbol a finding names before it rules, and a pull request over 1,000 changed lines receives a neutral verdict that the verdict record counts as an availability miss, never as coverage — seven of the ninety pull requests on record, counted per pull request, about 92 percent against the 99 percent availability bar — a miss the record can see, traded for a verdict that ran past the latency bar every time one was tried. The availability watch has a daily leg on the weekly audit ritual itself, alarming once the ledger's newest label is more than ten days old. An unlabeled week, past its three days of grace, is a silent-failure week.
A trust dashboard backs the weekly audit: ladder position, per-gate metrics against the promotion bars, the full verdict log, and the escape-rate record. Promotion decisions cite it.
Promotion to advisory waits on two numbers: 85 percent audited precision and a p95 under three minutes. The verdict count and the clean weeks cleared before the cap; section 7 of the paper records the cap's trade. The second weekly audit (2026-09-15), a census of every stage-2-verified finding on every needs-changes verdict since the first, measured 22 of 43 correct — 51 percent — so the gate stays at shadow. That audit closed a seven-week lapse in the weekly ritual; the verdict record never lapsed, the labeling did, and the white paper's dated revision records it as the silent-gate failure it is. The hard gate has no scheduled date, and that is the design.
The white-paper edition — the full reference architecture, with the rung rationale and the measurement doctrine — is published.

The ladder is not internal housekeeping. It is the promotion discipline underneath the validator-gated architecture iSimplifyMe deploys for clients — model judgment proposes, deterministic checks block, and nothing non-deterministic gains authority it has not measured its way into. The content pipeline's own quality gate is climbing the same ladder in shadow today. For teams that want their own AI surfaces held to that bar, the validator gap audit is where to start.

Trust Ladder

What is the Trust Ladder?