How do you know an agent handles a refund dispute, a prior-authorization packet, or a P2 ticket the same way your team handles it today? Most infrastructure leaders answer that with a pilot, and a pilot answers a narrower question — whether the agent can do the work at all, on a curated set of cases, with someone watching.
The phase that follows is where production confidence gets built. In a shadow-mode parallel run, the agent processes the same live traffic as the existing team, every divergence between the two is logged and scored, and cutover waits until the variance is understood, not merely until it looks small.
What Is A Shadow-Mode Parallel Run?
The term comes from deployment practice, where shadow traffic lets a new service version see real requests without serving responses. Applied to agents, the mechanics are similar, but the comparison is harder — you are diffing one judgment against another, and the reference answer came from a person who may have had context the ticket never recorded.
A shadow-mode parallel run sends live production inputs to both the human team and the agent. Only the human decision reaches the system of record. The agent's decision is captured, diffed, and scored.
That said, the goal stays simple. You want a dataset of real cases, each with a human decision and an agent decision side by side, large and varied enough that the cutover decision rests on evidence rather than a demo.
Where Shadow Mode Sits Between Pilot And Decommission
Shadow mode is the second of five phases in the path we use to move a workflow from a manual process to an agent-operated one. The pilot-to-production transition ends when the agent has shown it can complete the task. The manual process decommission begins only after the agent has shown it completes the task the way the business needs it done.
Here's how the phases compare on what the agent touches and what each phase has to prove:
| Phase | Agent input | Agent writes to | Exit evidence |
|---|---|---|---|
| Pilot | Curated cases | Sandbox | Task is feasible at target accuracy on known cases |
| Shadow | Full live traffic | Isolated shadow sink | Divergence rate and taxonomy understood per category |
| Assisted | Live traffic | Draft queue, human approves | Approval rate stable, edits trending toward cosmetic |
| Segment cutover | Selected categories | System of record | No critical divergence in live segments over a full cycle |
| Decommission | All traffic | System of record | Human path retired, rollback path documented |
Notice that shadow is the only phase where the agent sees everything and changes nothing. This is why it produces the cleanest evidence — and also why teams are tempted to skip it, since it costs money and produces no visible output.
How To Fork Live Inputs Without Letting The Agent Touch Production
The first engineering problem is isolation. The agent needs the same inputs the human sees, at roughly the same moment, with no path to change the system of record.
Here's the setup we use on AWS, which translates cleanly to other clouds:
- Fork at the event bus. An EventBridge rule or SNS topic fans each inbound case out to the existing human queue and to an SQS queue the shadow agent consumes. The human path stays untouched, so the shadow agent can fail, throttle, or back up without affecting service.
- Deny writes at the credential layer. The shadow agent runs under an IAM role with explicit deny statements on every write action, and its Salesforce or ServiceNow access goes through a read-only integration user. In IAM, an explicit deny overrides any allow, so a misconfigured tool cannot grant itself access.
- Intercept mutating tool calls. Any tool in the registry that would create, update, or send is wrapped so the call is recorded as a proposed action with its full arguments. Those proposed actions become the agent's answer for comparison.
- Key everything to the case ID. The human's ticket or case ID becomes both the idempotency key and the join key for the diff. Without that shared key, the comparison degrades into fuzzy matching, and fuzzy matching hides exactly the divergences you need to see.
- Snapshot the read state. Record what the agent read and when, because a human who works the ticket four hours later may see an updated balance or a new attachment. A stale-state read looks like a disagreement until you check the timestamps.
Keep in mind that the shadow path should be cheap to turn off. If disabling it requires a deploy, it will eventually be left running long past its usefulness.
Why Divergence Logging Is Harder Than It Looks
Logging that the agent and the human disagreed is easy. Logging what they disagreed about, in a form a reviewer can act on, takes design work before the first case runs.
The core decision is which fields count as the answer. For a claims workflow, that might be disposition, reason code, amount, and routing destination; for a support queue, it might be priority, category, assignee, and escalation.
Diff structured decision fields — disposition, reason code, amount, routing — keyed to the human's case ID. Comparing prose hides real disagreements and flags harmless ones.
Two replies to a customer can differ in every sentence and still be equivalent. Two reason codes that differ by a single character, on the other hand, can send a case down entirely different paths.
Every divergence record should carry the inputs, the agent's full trace, the human's decision, both timestamps, and the pinned model and prompt versions. This is the same evidence an agent audit trail preserves after cutover, so building it now turns the shadow log into the template for production logging.
What's more, the divergence stream belongs in the same dashboards as the rest of your agent observability stack. A divergence rate that spikes every Tuesday afternoon is telling you something about input mix or upstream schema drift, and you will only see it if it is plotted next to volume and P95 latency.
How Do You Score A Divergence?
A raw divergence rate tells you very little on its own. A 9% rate made up of formatting differences is a better result than a 2% rate that includes three wrongful denials.
Every divergence gets a category from a fixed taxonomy, and the categories we use include but are not limited to:
- Agent error. The agent reached a decision the policy does not support. These are scored by severity, and a single critical error in a regulated workflow can outweigh hundreds of minor ones.
- Human error. The agent was right and the existing process was wrong. These deserve careful counting, because they become the strongest evidence in the eventual agent business case.
- Policy ambiguity. Both answers are defensible under the written policy. These go to the process owner, since the agent cannot be graded against a rule nobody wrote down.
- Context gap. The human used information the agent never saw — a phone call, a Slack thread, a note from the previous shift. These are integration problems, fixed by widening the agent's inputs or narrowing its scope.
- Timing artifact. The two decisions were made against different states of the record. Once the snapshot timestamps confirm it, these are excluded from accuracy scoring.
Who does the scoring matters as much as the taxonomy. An adjudicator who can tell which answer came from the agent is not a neutral judge, and the findings in position bias and reviewer agreement as preconditions for any win rate apply directly: strip the source labels, randomize the order, and confirm your reviewers agree with each other before trusting what they say about the agent.
That same exercise produces the benchmark that matters most. Have two senior operators independently work an identical sample of cases, and their disagreement rate becomes the bar the agent is measured against.
Score the agent against human-to-human disagreement on the same cases. If two senior operators already disagree on 6% of cases, an agent at 5% with no critical errors sits inside existing variance.
Additionally, run the agent against itself. Identical requests can return different outputs on models with no sampling controls, so replaying a few hundred shadow cases a second time separates real policy disagreements from run-to-run noise.
How Many Cases Are Enough?
Enough to say something meaningful about the categories that carry risk. A shadow run that logs 20,000 cases, only 40 of them high-value exceptions, has told you a great deal about the easy path and very little about the one that matters.
Accordingly, stratify the sample by case category and set a floor per category before the run starts. The rule of three gives a useful planning number: if you observe zero critical errors in n cases, the 95% upper confidence bound on the true critical-error rate is approximately 3/n.
Set a case floor per category. With zero critical errors in n cases, the 95% upper bound on the true rate is roughly 3/n, so 3,000 clean cases bounds a category near 0.1%.
In practice, 300 clean cases in a category bound the critical-error rate near 1%, and 3,000 clean cases bound it near 0.1%. If your tolerance for a wrongful denial is one in a thousand, a category with 400 cases of shadow history has not yet earned cutover, regardless of how strong the topline number looks.
Calendar time matters too. A shadow run should cover at least one full business cycle — a month-end close, a billing run, open enrollment, a seasonal surge — because input mix shifts at those moments and an agent's weakest categories often surface only then.
What A Shadow Run Actually Costs
Shadow mode is overhead for its entire duration, and the number should be named before anyone approves it. The agent processes full production volume, the humans still do all of the work, and someone has to adjudicate the diffs.
For example, a workflow handling 400 cases a day over a 45-day window produces 18,000 shadow cases. If adjudicators blind-review a stratified 10% sample at roughly three minutes each, that is 1,800 reviews and about 90 hours of senior-operator time, on top of the inference bill for 18,000 full agent runs.
This is why we sample lightly on high-volume, low-risk categories and review exhaustively on low-volume, high-risk ones. Caching shared context and routing the classification step to a smaller model — the approach behind keeping AI spend flat while token usage grows — keeps the inference line in check, provided the shadow configuration matches what will actually ship.
Write the cutover criteria before the first case runs. Per-category case floors, the maximum tolerated critical-error bound, the human-to-human benchmark, and the minimum calendar window should all be signed off by the process owner in advance.
Criteria written after the data arrives tend to bend toward whatever the data happens to show.
What Divergences Reveal About The Human Process
One of the more valuable outputs of a shadow run has little to do with the agent. Clusters of context-gap and policy-ambiguity divergences map the parts of the existing process that live in people's heads instead of in the runbook.
For instance, a team may handle one vendor's invoices differently because of a dispute two years ago that nobody documented. The agent, reading the written policy, will diverge on every one of those invoices — and the divergence log becomes the first place that exception was ever written down.
Treat those findings as process debt to resolve before cutover, either by documenting the rule and adding it to the agent's context or by routing that category to a human permanently. Left unresolved, the gap returns after cutover as runbook drift, where the documented process and the operated process quietly separate.
How Cutover Actually Happens
Cutover is a sequence of narrower steps rather than a single switch. Once a category clears its written criteria, it moves from shadow into an assisted phase, where the agent drafts the decision and a human approves, edits, or rejects it inside the existing queue.
That assisted phase is a form of supervised autonomy that widens one rung at a time, and its approval and edit rates are the next layer of evidence. When approvals hold steady and edits trend toward cosmetic, the category earns direct write access while the remaining categories stay in shadow or assisted mode.
Cut over one category at a time. A category goes live only after it clears its written criteria, holds stable approval rates in an assisted phase, and has a staffed human rollback path.
Several conditions should hold at the moment a segment goes live:
- Pinned versions. The model version, prompt, and tool registry that earned the evidence are the ones that ship. Any change resets the evidence for the affected path, which is why cutover belongs inside a disciplined agent release management process.
- A routed exception path. Cases the agent declines or flags as low-confidence land in a staffed agent exception queue with an owner and an SLA. Without one, a cutover quietly converts uncertainty into errors.
- A warm rollback. The human process stays staffed and trained for the segment for a defined period after cutover. Reverting a category should be a routing change measured in minutes.
- A tripwire. An error-rate threshold, checked against a continuing sample of human-reviewed cases, triggers automatic rollback to assisted mode. That tripwire belongs in the same incident tooling as the rest of your reliability engineering for regulated AI.
Only after every category has run live through a full cycle without a rollback does the conversation turn to retiring the manual path. By then, the agent adoption metrics you track in production should be continuous with the divergence data you gathered in shadow.
Signs A Shadow Run Has Stalled
In our experience, shadow runs tend to fail by drifting rather than by breaking. Some warning signs include:
- Nobody has reviewed the diffs in two weeks. A divergence log without adjudication is just storage cost. Assign an owner and a weekly review cadence before the run begins.
- The topline rate is the only number reported. If leadership sees “94% agreement” with no category breakdown, the critical categories are hiding inside the average.
- The model or prompt changed mid-run. Evidence gathered on one configuration does not transfer to another. The clock restarts for every path the change touched.
- There is no end date. A shadow run with no scheduled decision point becomes permanent overhead. Set a date to cut over, extend with a stated reason, or shut the workflow down.
Overall, the discipline is the same as any production change: define success up front, gather evidence on the exact configuration you intend to ship, and keep a way back. Shadow mode simply makes that discipline visible to the auditors, the process owners, and the operators who will live with the result.
Scoping Your First Parallel Run
If you're moving a workflow out of pilot and need to prove the agent matches your team before anything touches the system of record, the team at iSimplifyMe builds and operates production agent systems across CRM, ticketing, and data warehouse environments every week. We'll design the fork and isolation layer, define the divergence taxonomy and per-category cutover criteria with your process owners, and leave you with a staged cutover plan that has a rollback at every step.
For the broader operating model around this phase, start with our overview of AI agent operations.