How many of the tickets your agent closed last week has a person actually read? If that agent writes to Salesforce, ServiceNow, or a claims ledger, the honest answer is usually a handful, pulled by whoever had a spare hour.
Most teams can quote their eval pass rate and their P95 latency to the decimal. That said, very few can put a number on how much of the agent's finished work was correct when it landed in the system of record, with a confidence interval attached.
Pre-production evaluation tells you whether the agent was good enough to ship, and live observability for production agents tells you whether it is running. Output sampling covers a third question, whether the work it already closed was right, and it is the step many programs skip.
What Is Agent Output Sampling?
Agent output sampling is a recurring human review of completed agent work, planned with statistics. A reviewer checks the final artifact against a written rubric after the agent has closed it. That artifact might be the ticket resolution, the CRM update, the drafted reply, or the coded claim.
Agent output sampling is a recurring human review of a random, risk-stratified slice of closed agent work. It estimates the true defect rate with an interval and catches drift before it spreads through your records.
Keep in mind that the population here is closed work: the final records the agent produced. A trace shows the agent called the right tool, but only the closed record shows whether the customer, patient, or ledger ended up in the right state.
Why Eval Suites And Dashboards Leave A Gap
Each existing control watches a different surface, and none of them puts human judgment on finished work at production volume. The table below shows what each layer can and cannot see.
| Control | When It Runs | What It Measures | Blind Spot |
|---|---|---|---|
| Pre-production evaluation | Before release, on a fixed dataset | Pass rate against known cases | Inputs the dataset never contained |
| Live observability | Continuously, on every request | Latency, errors, tool calls, token cost | Outputs that are well-formed but wrong |
| Output sampling | Daily or weekly, on closed work | Estimated defect rate with an interval | Defects rarer than the sample can resolve |
| Incident response | After someone reports a failure | Root cause of a known problem | Everything nobody noticed |
The agent evaluation suite you built before launch was frozen on the day you put it together. Production traffic drifts away from that dataset within weeks as new SKUs, new policy language, and a reorganized knowledge base arrive. The suite keeps passing because it never sees those inputs.
Observability has the opposite problem. It sees every request, yet a confident, well-formatted, wrong refund decision produces the same span shape as a correct one.
Observability records whether the agent ran, how fast, and which tools it called. It cannot tell a correct refund from a confidently wrong one, because both produce the same trace shape in your telemetry.
How Do You Size The Sample?
Sample size follows from two choices you make before anyone reviews anything: how precisely you need to know the defect rate, and how confident you need to be in that estimate. Write both down, because they become the contract between the agent's owner and whoever approves its autonomy.
To estimate a proportion, use the standard formula n = z² × p(1 − p) / e². Here z is 1.96 at 95% confidence, p is the expected defect rate, and e is the margin of error. With no prior data, p = 0.5 is the safe default, and it gives the familiar figure of about 384 items for a ±5-point margin.
When the weekly population is small, apply the finite population correction, n / (1 + (n − 1) / N). For example, an agent closing 2,000 items a week needs about 322 reviewed items for the same ±5-point margin, and one closing 400 a week needs about 197.
After a few weeks of data, replace p = 0.5 with the rate you actually observed. An agent running near a 4% defect rate needs only about 59 items for ±5 points. A margin that wide tells you nothing at that rate, though, so tighten it to ±2 points and the requirement rises to about 369.
Start from the margin of error you need, not a share of volume. At 95% confidence, a ±5-point margin needs about 384 items. Once the observed defect rate is low, tighten the margin to ±2 points and resize.
Sizing For Rare, Severe Defects
Some failure modes are rare enough that a precision target misses them entirely. For these, size for detection instead. The sample needed to see at least one defect with 95% probability is ln(0.05) / ln(1 − r), where r is the true defect rate.
For a defect that occurs in 1% of items, that works out to about 299 reviewed items. At 0.5% it climbs to about 598. This sizing applies within each stratum, which is why a flat random sample fails on the work that matters most.
Stratify By Risk Instead Of Volume
A flat random sample spends most of its review budget on the agent's easiest, highest-volume work. Suppose 85% of closed items are password resets and 2% are refunds over $500. A 300-item flat draw then gives you about six refunds, far too few to say anything about the stratum that can actually cost money.
Stratification splits closed work into risk tiers and sizes each tier separately. When you report an overall rate, you weight each tier's results back by its share of total volume. Common keys for stratifying include but are not limited to:
- Write target. Items that changed a system of record rank above items that only drafted text for a person to send. Examples are a Salesforce opportunity stage, a ServiceNow incident state, or a ledger entry.
- Financial or clinical exposure. Refund amount, credit issued, PHI touched, or any action covered by a BAA belongs in its own tier with its own tolerance.
- Agent confidence. Some items land near a threshold, such as the model's own score, a Bedrock Guardrails flag, or a retrieval miss. These carry more defects per item and deserve a higher sampling rate.
- Novelty. Drift shows up first in intents, customer segments, or document types that appeared after the last eval refresh.
- Change exposure. Work closed in the 72 hours after a model-version change, prompt edit, or tool-schema update gets its own temporary stratum.
Each of these keys has to exist on the closed record or in the audit trail when the work closes. If you cannot query a key in Snowflake or Postgres afterward, you cannot stratify on it, so add it to the agent's output schema before you design the plan.
Yes. Review high-exposure strata at much higher rates than routine work. Then weight each stratum back by its share of volume when you report an overall defect rate, so the extra sampling does not inflate the headline number.
A Worked Weekly Plan
Take an agent that closes 5,000 support items a week: 4,200 informational replies, 650 account changes, and 150 refunds and credits. Each tier gets its own sampling target, sized to the exposure it carries.
Informational replies get 150 items, enough to catch a 2% defect with 95% probability. Account changes get about 200, enough to catch a 1.5% defect at the same confidence. Refunds and credits are reviewed in full.
That adds up to 500 reviewed items a week. At four minutes per item, it costs about 33 reviewer-hours.
Draw The Sample Without Bias
How you pull the sample matters as much as how many items you pull. The draw should be random within each stratum, reproducible, and invisible to the agent and to the people who tune it.
In practice, you can hash each record ID with a weekly salt and select items that fall below a threshold. One way is to take a SHA-256 of the ticket ID plus the ISO week, modulo 10,000. Anyone can rerun the same draw for an audit, and the rotating salt keeps people from learning which records will be reviewed.
Avoid sampling from whatever queue reviewers happen to open, from escalations, or from records a customer reopened. Those are useful signals on their own. However, they are skewed toward visible failures, so they overstate the defect rate in some strata and hide it in others.
How Do You Keep Reviewers Consistent?
A sample is only as accurate as the people grading it. If two reviewers disagree on 20% of items, that noise is larger than any confidence interval you computed.
First, write a rubric with a small, fixed set of defect codes and a severity scale tied to business impact. Codes such as wrong action, wrong data written, policy violation, unsupported claim, and tone or format cover most cases. Then have a second reviewer blindly grade about 10% of each week's sample, and compute Cohen's kappa for each stratum.
A kappa above about 0.7 is workable. Below about 0.6, fix the rubric and retrain reviewers before you trust the numbers. The same rule applies to LLM-as-judge pre-screens: as our research on position bias and agreement as preconditions for a win rate shows, you have to measure agreement before you report a score.
Not on its own. An LLM judge can pre-screen and rank items for review. Its agreement with trained human reviewers has to be measured for each stratum before any of its verdicts count toward the defect rate.
Turning Findings Into Fixes Before Drift Lands
Reviewing is the cheap half of the work. The program pays off only when a finding changes the agent before the next thousand records repeat the same defect.
Route every confirmed defect to one of four owners, based on where the fix belongs:
- Prompt or policy. The agent followed its instructions, and the instructions were wrong or out of date. This is the pattern we describe in runbook drift in production agents.
- Retrieval. The agent answered from a missing or outdated document in the knowledge base. The fix belongs to whoever owns that content.
- Tool or schema. A downstream API changed shape, a field mapping broke, or an idempotency key collided.
- Model behavior. The agent had the right instructions and context and still reasoned badly. This usually points to routing or model-version pinning.
Every confirmed defect should also become a new regression case in the eval suite. That is how sampling feeds pre-production evaluation: the suite grows from real production failures instead of from what the team guessed at launch.
Severity sets how fast you respond. A low-severity format defect waits for the weekly fix cycle. A single confirmed case of wrong data written to a patient record or a ledger opens a ticket in your agent incident response process the same day.
Fix The Records As Well As The Agent
Remember that each defect found in the sample implies more in the work nobody reviewed. If the sample shows a 3% wrong-write rate in a stratum of 650 items, about 20 records in the system of record are probably wrong, and you have reviewed only a few of them.
For each confirmed defect type, you therefore need a sweep query that finds every affected record and a compensating transaction that corrects them. Build that reversibility into the agent's write path from the start; it is far cheaper than rebuilding state from CloudTrail later.
Set Thresholds That Trigger Action
A defect rate with no threshold attached gives nobody a reason to act. Each stratum needs a tolerance and a defined response when the estimate crosses it.
Compare the upper bound of the confidence interval against tolerance, not the point estimate. A Wilson score interval stays accurate at the low rates and small samples typical of high-risk strata. For a quick check, use the rule of three: zero defects in n items puts the 95% upper bound near 3/n.
- Upper bound below tolerance for three periods in a row. Keep the current rate. If the stratum is stable and the agent's configuration has not changed, reduce it by a set step.
- Upper bound crosses tolerance. Double the sampling rate for that stratum and open a root-cause ticket.
- Point estimate crosses tolerance. Send the stratum through human approval or a deterministic path, using the routes in your agent fallback design. Keep it there until a fix ships and a new sample passes.
These rules make the agent earn its autonomy, and lose it, on evidence. The same principle, where review effort drops only after measured reliability justifies it, underpins our work on supervised autonomy for AI code review.
When the point estimate of its defect rate crosses the stratum's tolerance. Send that work through human approval or a deterministic fallback until a fix ships and a new sample passes the threshold.
What Does A Sampling Program Cost?
Reviewer time is the main cost, and it is easy to estimate. At an average of four minutes per item, a 500-item weekly sample takes about 33 reviewer-hours, a little under one full-time reviewer.
Complex items take longer. Clinical documentation or multi-step claims review can run 10 to 15 minutes per item, so a 300-item sample in those strata costs 50 to 75 hours a week. Budget it as a standing line instead of borrowing hours from the support floor.
Compare that to not sampling at all. If the agent closes 5,000 items a week with an unsampled 2% wrong-write rate, and each bad record costs $40 to find and fix later, that is $4,000 a week in cleanup. That figure comes before any customer churn, audit finding, or regulator letter.
Fold this line into your AI agent cost governance model. That way the review budget grows with volume and is not the first thing cut when token spend climbs.
Where Sampling Fits In The Operating Model
Sampling sits between release and incident, and it works best when one team owns it end to end. That should be the team that owns the agent in production, with reviewers drawn from the business function the agent serves.
It also changes how you run releases. When you test a new model version with parallel runs against the current agent, the shared rubric and defect codes let you compare the two on closed work instead of a frozen dataset.
For the wider structure, see our guide to the agent operating model and the rest of the AI agent operations cluster. Together they cover who approves autonomy changes, how findings reach the backlog, and how review roles are staffed.
Frequently Asked Questions
The questions below cover edge cases that come up when a team sets up its first sampling program.
How large should a weekly QA sample be?
For a ±5-point margin at 95% confidence, about 384 items. After finite population correction, an agent closing 2,000 items a week needs about 322. Split that total across risk strata instead of drawing it flat.
What if reviewers find zero defects in a stratum?
Zero defects gives you an upper bound only. By the rule of three, zero failures in 300 reviewed items puts the 95% upper bound near 1%. The accurate claim is that the rate is below about 1%.
How do you know reviewers agree with each other?
Have a second reviewer blindly grade about 10% of the sample, and compute Cohen's kappa for each stratum. Below about 0.6, fix the rubric and retrain reviewers before you trust any defect rate the sample produces.
When can the sampling rate come down?
Only after several periods in a row where the upper confidence bound stays below the stratum's tolerance. Raise it right away after any model, prompt, tool, or retrieval change.
Who should own the sampling program?
The team that owns the agent in production, with reviewers drawn from the domain the agent serves. Findings go into the same backlog as incidents so fixes get scheduled and shipped.
Build The Plan Before The Drift
Any agent that writes to a system of record piles up small errors faster than a dashboard will show them. A sampling plan turns those errors into a known number with an interval, an owner, and a defined response.
If you run an agent that already closes work in Salesforce, ServiceNow, Zendesk, or a clinical record and nobody can state its defect rate, the team at iSimplifyMe builds and operates production agent systems across CRM, ticketing, and data warehouse environments every week.
Reach out for a working session: we'll stratify your closed work, size a sample your reviewers can keep up with, and leave you with a rubric, a sampling query, and threshold rules you can run the following Monday.