Why does an AI code reviewer start in shadow?

Shadow is the only rung where every property of the gate can be measured without the gate influencing anything. The reviewer runs its full pipeline on real pull requests — risk-class router, find stage, verify stage, verdict record — while posting nothing and blocking nothing. Four weeks of shadow operation establishes availability and latency and accumulates the verdict sample a precision audit needs. A reviewer that cannot run reliably for a month with no authority has no business holding any.

What happened to the first automated reviewer?

An after-the-fact audit measured 33 percent precision — two correct claims of the six examined — and it once prescribed a destructive repository-history rewrite for a problem that did not exist. The deeper failure was availability: a dead credential silenced it for seven straight weeks and nothing alerted, because its verdict was never wired to anything that could fail loudly. It was retired, and the ladder was designed so neither failure class can recur quietly.

What does it mean that the gate fails neutral?

An infrastructure failure — an expired credential, a model throttle, a timeout — produces a neutral check conclusion, a decrement to the gate's availability metric, and an alarm. It never produces a red X, so a dead credential cannot block an urgent fix, and it never produces silence, so a dead gate cannot go unnoticed for weeks. Neutral-plus-alarm is the property that keeps the availability number honest.

How are findings verified before they count?

A second, adversarial stage attempts to refute every finding against the actual code and history before it is recorded; only survivors count toward precision. A replay against forty-six merged pull requests with known outcomes showed why: raw find-stage precision measured 66 percent, and the findings labeled highest-severity were the least reliable band at roughly 38 percent. High claimed severity therefore earns the closest scrutiny, precisely because it measured as the least reliable band.

When does the hard gate turn on?

It has no scheduled date, and that is deliberate. A hard gate is earned by sustained numbers — 95 percent or better audited precision held for weeks, perfect availability, and a near-zero false-block rate at the soft-gate rung first. Nothing in the roadmap schedules it, because a blocking gate promoted on a calendar rather than on measurements would repeat the exact failure the ladder was built to prevent.

The Trust Ladder — How AI Code Review Earns Autonomy

Shadow → advisory → soft-gate → hard-gate: AI code review earns merge authority on measured precision.

Abstract

The Trust Ladder is iSimplifyMe's promotion system for AI code review: a four-rung ladder — shadow, advisory, soft-gate, hard-gate — that a non-deterministic reviewer climbs on published metrics before it is allowed to block a merge. It exists because the failure mode of AI code review is not bad findings; it is unearned authority. This dossier documents the ladder, the harness that climbs it, and the numbers that decide promotion.

Why a ladder?

Boris Cherny, the creator of Claude Code, published Steps of AI Adoption in July 2026 — five maturity levels running from Gated, with zero agents, to AI-native, with a thousand or more agents. Step 3, supervised autonomy, is the level where agents write nearly all of the code and a human manages by exception. His stated bottleneck for that step is the reason this system exists: trust in the loop.

The trap, in his words, is scaling agent count before the loop has earned widespread trust.

iSimplifyMe already runs the furniture of step 3 — scheduled agents, overnight batches, and deterministic CI gates across a fleet of more than forty repositories behind roughly a hundred live properties — with agents doing most of the writing. The missing piece was a measured answer to a single question: when the reviewer is a model, how do you know when to stop reading every diff?

What the first attempt taught

Our first automated reviewer earned the opposite of trust. Audited after the fact, it measured 33 percent precision — two correct claims out of the six examined, a sample whose very smallness is part of the story. Worse than the wrong findings was the silence: a dead credential killed the reviewer outright for seven straight weeks and nothing noticed, because its verdict was never wired to anything that could fail loudly.

At one point it prescribed a destructive repository-history rewrite for a problem that did not exist.

That failure produced the two rules everything else here follows. First, a non-deterministic gate must be measured before it is trusted — precision is a number, not a feeling. Second, silence must be impossible; a gate that can die quietly is worse than no gate at all.

The four rungs

Every AI gate starts at the bottom. Promotion is earned per gate, per risk class, against published thresholds on a trailing window:

Rung	What it may do	Promotion bar
L0 Shadow	Runs on every pull request, logs a structured verdict, posts nothing, blocks nothing	~30 logged verdicts, availability ≥ 99% over four weeks
L1 Advisory	Posts one comment per pull request — no emails	Audited precision ≥ 85%, p95 latency under three minutes
L2 Soft-gate	Required check, dismissable via a logged override label	Precision ≥ 95%, zero silent-failure days, override rate under 10%
L3 Hard-gate	Required, no label bypass	Deliberately unscheduled — earned only

Demotion is automatic. A silent-failure week, precision below the rung's bar on a weekly audit, or more than two wrong blocks in a week drops the gate one rung and raises an alert.

What gets measured

Trust in an AI code-review gate is the minimum of its four signals. Precision is audited-correct findings over total findings; availability is days the gate reported over days with eligible pull requests; latency is the p95 from trigger to verdict; the fourth is the false-block rate. A fifth, lagging signal — escape rate, production incidents traced back to a merge the gate passed — is tracked as the recall proxy.

The ritual that turns those numbers into decisions is deliberately small: a weekly ten-minute audit samples five verdicts and marks each one right or wrong on a dashboard. Promotion decisions cite dashboard numbers. Nothing promotes on a good week's impression.

The harness

A risk-class router decides first. It classifies every change by blast radius — a documentation-only diff and a secrets-adjacent diff do not face the same review — and routes each one to the gates that apply. The reviewer itself is a two-stage pipeline: a coverage-first find stage reports every candidate issue with severity and confidence, then an adversarial verify stage attempts to refute each finding against the actual code and history before anything is recorded.

Only findings that survive refutation count.

The verify stage exists because of a replay study. Re-running the reviewer against forty-six already-merged pull requests with known outcomes put raw find-stage precision at 66 percent — and surfaced an inversion worth publishing: the findings the model labeled highest-severity were its least reliable band, at roughly 38 percent. Claimed severity is a prompt to verify, never a reason to trust.

Three more properties are load-bearing. The harness fails neutral: a credential failure, a throttle, or a timeout produces a neutral conclusion, an availability decrement, and an alarm — it cannot go red on an infrastructure fault, and it cannot die quietly. What's more, verdicts bind to the exact commit they reviewed, so a fast merge cannot outrun its review. And findings are never auto-applied: the harness proposes, a human disposes, and only deterministic checks block merges on their own.

Status

Every layer described above is in place today; what remains is accumulation:

The availability watch — an alarm on any required check that has stopped reporting on an active repository — is live across the full fleet. Silent gate death now raises an alert instead of waiting to be discovered.
The review harness runs at L0 shadow on two production repositories, writing a verdict record for every pull request. The first unprompted organic verdicts landed within days of arming, at an observed median latency of roughly seventy seconds — well inside the three-minute p95 bar.
A trust dashboard backs the weekly audit: ladder position, per-gate metrics against the promotion bars, the full verdict log, and the escape-rate record. Promotion decisions cite it.
Promotion to advisory waits on the numbers — thirty verdicts, four clean weeks, 85 percent audited precision. The hard gate has no scheduled date, and that is the design.
The white-paper edition — the full reference architecture, with the rung rationale and the measurement doctrine — is published.

The ladder is not internal housekeeping. It is the promotion discipline underneath the validator-gated architecture iSimplifyMe deploys for clients — model judgment proposes, deterministic checks block, and nothing non-deterministic gains authority it has not measured its way into. The content pipeline's own quality gate is climbing the same ladder in shadow today. For teams that want their own AI surfaces held to that bar, the validator gap audit is where to start.

Trust Ladder

What is the Trust Ladder?