How many AI deployments is your team running in production right now, and who exactly gets the call when one of them starts drifting at 2 a.m.? If the honest answer to the second half is "whoever notices first," the fleet has outgrown the way you are watching it.
Most platform teams start with one agent, one dashboard, and one engineer who knows its quirks. By the time the count reaches eight or twelve — a Bedrock-backed support triage agent, a Salesforce enrichment workflow, a ServiceNow ticket router, a RAG assistant over Confluence — that model of oversight stops scaling, and the temptation is to hand the watching to another agent that also holds the keys to fix things.
That said, the more defensible pattern is a monitoring layer that diagnoses and proposes but never remediates on its own. We call it investigate-only monitoring, and the rest of this post covers how it is scoped, how its findings are triaged, and how each one lands in front of a named human who signs off before anything changes.
What Is Investigate-Only Monitoring?
Investigate-only monitoring is a supervisory process that reads across every deployment in a fleet, correlates signals, and writes up what it finds. Its output is a finding with a proposed fix attached — a diff, a config change, a rollback target — routed to a person who holds the authority to apply it.
Investigate-only monitoring is a fleet-level layer that reads logs, traces, and costs across AI deployments, diagnoses problems, and proposes fixes. It holds no write permissions, so a named human approves every change.
The defining constraint is structural rather than procedural. The monitor holds credentials to look and to file tickets, and nothing else, so the safeguard is that it cannot act — which is far stronger than a policy saying it should not.
Why Fleet Oversight Is A Different Problem Than Single-Agent Observability
Single-agent agent observability answers whether one workflow is healthy: its traces, its tool-call error rates, its P95 latency, its token spend per run. Fleet oversight answers a harder question, which is whether the pattern you are seeing in one deployment is also happening in four others and shares a cause.
For instance, a Bedrock model-version update, a rotated KMS key, or a schema change in a shared Snowflake table rarely breaks only one agent. A per-agent dashboard will show four separate yellow lights, while a fleet monitor can group them under one root cause and file one finding instead of four pages.
Per-action approval gates address a third problem entirely. A gate sits inline and asks a human to approve a specific action an agent is about to take, whereas an investigate-only monitor sits outside the request path and asks a human to approve a change to the system itself.
Observability watches one agent's health. Approval gates hold one action for sign-off. Investigate-only monitoring correlates signals across many deployments and proposes system-level fixes for a human to approve.
The three patterns complement one another, and a mature fleet runs all of them. Here is how they compare on the dimensions that matter to an operator:
| Dimension | Agent observability | Approval gates | Investigate-only monitoring |
|---|---|---|---|
| Scope | One deployment | One action | The whole fleet |
| Position | Alongside the agent | Inline in the request path | Outside the request path |
| Output | Metrics, traces, alerts | Approve or deny one action | A finding plus a proposed change |
| Who decides | On-call engineer | Operator at the gate | Named approver per deployment |
| Write access | None | Executes after approval | None, enforced in IAM |
| Latency tolerance | Real time | Seconds to minutes | Minutes to hours |
What The Monitor Is Allowed To Do
Keep in mind that "investigate-only" is enforced in IAM, not in the system prompt. The monitor's permissions fall into three groups, and anything outside them should hit an explicit deny:
- Read. CloudWatch Logs Insights queries, CloudTrail LookupEvents, X-Ray or OpenTelemetry traces, Cost Explorer, and read-only copies of each deployment's evaluation results. It sees prompts and outputs only where the deployment's data retention policy already permits operator access.
- Reason. InvokeModel on a pinned model version for its own diagnosis — Claude on Bedrock in our deployments — with a hard budget cap on its own spend. Its tool registry contains query tools and nothing that mutates state.
- Write findings. PutItem on a findings table in DynamoDB, SendMessage to a routing queue in SQS, and ticket creation in ServiceNow or Jira. It cannot update a Lambda configuration, move an alias, roll back a model version, or edit a prompt template.
Moreover, we attach a service control policy that denies mutating actions on agent resources to the monitor's role, regardless of what its identity policy says. That way a well-meaning edit to the monitor's permissions six months from now cannot quietly hand it remediation rights — the same principle behind sound agent identity and access design generally.
Enforce it in IAM. Give the monitor read access to logs, traces, and costs, permission to write findings and tickets, and an explicit SCP deny on every mutating action against agent resources.
How Findings Are Triaged
A fleet monitor that files everything it notices becomes noise within a week. Therefore triage happens in three passes before any human sees a finding.
- Correlate. The monitor groups anomalies that share a time window and a dependency — the same model ID, the same upstream API, the same IAM role, the same retrieval index. Twelve elevated error rates that began within four minutes of a model-version change become one candidate finding.
- Classify. Each candidate gets a class (cost, quality, reliability, security, or compliance) and a severity based on blast radius and reversibility. A prompt regression in an internal summarizer and a PHI exposure in a patient-facing NexV deployment do not share a queue.
- Deduplicate against open work. If a candidate matches a finding that is already open, the monitor appends evidence to it rather than filing a new one. In our experience, this pass removes more noise than the other two combined.
All of this adds up to a smaller, sharper queue. As a result, the approver sees one finding with twelve affected deployments attached rather than twelve tickets they have to connect themselves.
Severity then determines who is notified and how fast they must acknowledge. The defaults below are illustrative starting points that most teams tune within the first month:
| Severity | Example | Routes to | Acknowledge within |
|---|---|---|---|
| Sev 1 | PHI or credential exposure, runaway spend | Deployment owner plus security, paged | 15 minutes |
| Sev 2 | Error or quality regression on a customer-facing workflow | Deployment owner, paged in business hours | 2 hours |
| Sev 3 | Cost drift, latency creep, declining eval scores | Owner's queue | 1 business day |
| Sev 4 | Unused tools, stale prompts, a model on its retirement path | Weekly review | Next review |
Routing Every Finding To A Named Human
The routing rule we hold to is simple: every deployment in the fleet has a named owner and a named backup, and a finding goes to a person, never only to a channel. A Slack channel is a fine place to broadcast a finding, but a channel cannot approve a change and nobody in it feels obligated to.
In practice, this means maintaining an ownership register alongside the deployment inventory. Each entry carries the following fields:
- Primary approver. The person accountable for the deployment's behavior — usually the engineering lead who shipped it or the business owner who requested it.
- Backup approver. The person who receives the finding if the primary does not acknowledge it within the severity window.
- Class overrides. Cost findings above a set threshold also route to the FinOps owner, and compliance findings route to the compliance lead, regardless of who owns the deployment.
- Change window. When an approved fix is allowed to run, inherited from the team's change window policy.
Note that the register goes stale the moment someone changes teams. We validate it on a schedule — the monitor itself files a Sev 4 finding when an owner's account is deactivated in the identity provider or an owner has not acknowledged anything in ninety days.
Route each finding to a named primary approver for the affected deployment, with a named backup and an acknowledgment timer. A channel can broadcast a finding, but only a named person can sign off.
What A Finding Should Contain
A finding is only as useful as the approver's ability to say yes or no to it in a few minutes. Every finding the monitor writes should include:
- Observed state. The evidence, with links to the exact Logs Insights query, trace IDs, and CloudTrail events, plus a hash of the configuration the monitor observed.
- Diagnosis and confidence. What the monitor believes happened, how confident it is, and which alternative explanations it ruled out.
- Proposed change. A concrete diff — the alias to roll back, the prompt version to restore, the concurrency limit to set — written so an executor can apply it without interpretation.
- Rollback path. How to undo the proposed change if it makes things worse.
- Expiry. A time after which the proposal is considered stale and must be regenerated.
The configuration hash and the expiry together prevent a subtle failure: an approver signing off on Tuesday morning on a proposal written against Monday night's state. When the executor picks up an approved change, it compares the live configuration hash to the one in the finding, and if they differ, it refuses and sends the finding back for re-investigation.
Evidence links, a diagnosis with confidence, a concrete proposed diff, a rollback path, an expiry, and a hash of the observed configuration so execution is refused if state has drifted.
Who Executes The Approved Change?
Approval and execution should run under different identities. Once the named approver signs off, a separate executor role — scoped to the specific resource and action in the proposal, and issued as a short-lived session — applies the change inside the deployment's change window and writes the result back to the finding.
This separation is what keeps the audit trail legible. CloudTrail then shows three distinct principals — the monitor that proposed, the human who approved, and the executor that applied — which is the chain an auditor will ask you to reconstruct and the same chain your incident response runbooks rely on after the fact.
A useful test before go-live: pick any change applied to the fleet last month and try to name, from CloudTrail and the findings table alone, who proposed it, who approved it, and what state it was approved against. If any of the three takes more than a few minutes to answer, the separation is not finished yet.
Where Investigate-Only Monitoring Breaks Down
No supervision pattern is free, and this one has predictable failure modes. The ones we design around include:
- Approver fatigue. If Sev 3 findings page someone, they will start approving without reading. Tune the severity map before adding deployments, not after.
- Monitor drift. The monitor runs on a model with its own retirement date, so pin its version and replay a fixed set of past incidents against it whenever that version changes.
- Single-approver bottlenecks. One platform lead who owns nine deployments becomes the queue. Spread ownership, or the backup quietly becomes the de facto primary.
- Overreaching proposals. A proposal that touches four services is harder to approve than four proposals that each touch one. Constrain the monitor to single-resource changes wherever possible.
- Monitoring spend. A monitor that runs a frontier model over every log line can cost more than the deployments it watches. Use cheap deterministic filters first and reserve model reasoning for correlated candidates, following the same logic as sound AI agent cost governance.
Of course, the monitor is one more production system. It needs the same reliability engineering for regulated AI — pinned versions, replay tests, dead-letter handling, postmortems — as the deployments it watches.
How Do You Know The Monitor Is Working?
Measure the monitor the way you would measure a new on-call engineer: on precision, speed, and whether its proposals hold up. The metrics we track include:
- Approval rate without edits. A persistently low rate means the diagnoses are weak. A rate near 100% sustained over months may mean approvers have stopped reading.
- Time to acknowledge and time to approve. Tracked per severity and per approver, which surfaces bottlenecks in the ownership register before they turn into outages.
- Incidents caught by humans first. Every incident that a person or a customer found before the monitor did becomes a new case in its replay suite.
- Post-change outcome. Whether the metric that triggered the finding actually recovered after the approved change ran.
Track the approval-without-edits rate, time to acknowledge and approve per severity, incidents humans caught first, and whether the triggering metric recovered after the approved change ran.
When To Let A Finding Class Remediate Itself
Over time, some finding classes earn a record that justifies more autonomy. If the monitor has proposed the same concurrency-limit adjustment forty times, and every one was approved unchanged and recovered the metric, that class is a reasonable candidate for automated remediation with after-the-fact review.
Promotion should be deliberate, per class, and reversible, following the evidence thresholds of supervised autonomy earned one rung at a time. Everything else — anything touching PHI, credentials, model versions, or customer-facing prompts — stays investigate-only, and even the automated rung still writes a finding so the named owner sees what happened.
Approved proposals also double as raw material for runbook capture. After all, a finding that was diagnosed, approved, and verified forty times is effectively a runbook the team has already reviewed.
Bring Fleet Oversight Under Human Sign-Off
Watching several AI deployments at once is a coordination problem as much as a tooling one, and investigate-only monitoring keeps judgment where your audit trail says it lives. It belongs alongside agent orchestration and the rest of the AI agent operations discipline, designed in from the start rather than bolted onto a dashboard later.
If you're running more than a handful of production agents and want a second set of eyes on how they are supervised, the team at iSimplifyMe builds and operates agent fleets across CRM, ticketing, and data warehouse environments every week. Reach out for a working session — we will inventory your deployments, draft the ownership register and severity map, and leave you with an IAM-enforced monitoring design you can deploy.