Can an agent read a regulatory standard and tell you what your application needs in order to meet it? It can do most of the work: parse the control language, collect evidence from your live systems, and draft findings. That said, the architecture only holds up in an audit if every gap finding ends with a human signature. This post walks through the workflow, the failure modes, and a rollout sequence we would actually run in production.
What would it take to show an auditor, control by control, that the systems you already run meet a standard you were handed last quarter? If you own a production platform in a regulated industry, you have probably answered that question with a spreadsheet, several weeks of console screenshots, and a long tail of Slack threads asking who owns which IAM role.
A visitor to our concierge desk recently asked a sharper version of that question: can an agent ingest a standard and assess what their application needs to meet it? The answer is yes, provided you treat the agent as an evidence collector and a drafter of findings, and keep the verdict with a person who signs it.
A compliance gap agent parses a regulatory standard into testable requirements, collects evidence from the systems you run, and drafts gap findings. A named human reviewer accepts or rejects every finding before it counts.
What A Compliance Gap Agent Actually Does
The workflow is a pipeline with five stages, and each stage has a different trust level. Keep in mind that the agent is strongest at the mechanical stages and weakest at the judgment stage, which is exactly why the judgment stage belongs to a person.
The stages break down as follows:
- Parse. The agent reads the standard and splits it into discrete requirement records, each tied to its exact source clause. Nothing downstream is allowed to cite a requirement that cannot be traced back to specific language.
- Map. Each requirement is mapped to the systems in scope: the Postgres cluster holding PHI, the S3 buckets behind the document pipeline, the Okta tenant, the GitHub org. The map is a draft that an engineer confirms once, then reuses.
- Collect. The agent queries live systems through read-only credentials and stores each result as a timestamped, hashed evidence object. Configuration and log metadata are the targets, not records.
- Draft. For each requirement, the agent writes a proposed status (met, partially met, no evidence found) with its reasoning and the evidence IDs that support it.
- Route. Drafts go to the control owner for review. Only the reviewer can move a finding to accepted, and only an accepted finding appears in the gap report.
All of the above is ordinary agent engineering. Accordingly, the patterns that keep any production agent honest — scoped tools, audit trails, evaluation suites — apply here with higher stakes, since the output may end up in front of an assessor.
Why Control Language Is Harder To Parse Than It Looks
Standards are written for humans who share a professional context, not for machines. The HIPAA Security Rule, for instance, labels its implementation specifications as either required or addressable, and addressable does not mean optional — it means you implement the specification or document why an equivalent alternative is reasonable.
PCI DSS v4.0 adds a different wrinkle. Its twelve principal requirements can be met through the defined approach or, for mature organizations, through a customized approach where you design a control that meets the stated objective, which means the testable requirement depends on which path you chose.
ISO/IEC 27001:2022 reorganized Annex A into 93 controls across four themes, and many of them describe an outcome rather than a mechanism. In addition, words like appropriate, periodic, and timely show up constantly, and none of them can be tested until someone in your organization decides what they mean.
Split a standard at the clause or implementation-specification boundary, not at a fixed token count. Each requirement record keeps its source citation, so every later finding traces back to exact language.
This is why the parsing stage looks more like structured document ingestion with an agent than like summarization. The same lesson from retrieval chunking applies: chunk on the document's own structure, because a requirement split across two chunks becomes two half-requirements that neither test can satisfy.
A usable requirement record carries a handful of fields:
- Source citation. The clause ID and verbatim text, for example 45 CFR 164.312(b).
- Testable assertion. A single sentence that can be true or false about your environment.
- Evidence type. Configuration state, log sample, policy document, or interview — the last of which the agent cannot collect.
- Interpretation flag. Set whenever the assertion depends on a term like appropriate, so a human defines the threshold once and the definition is versioned.
- Owner. The person who signs findings against this requirement.
Here's how that translation looks for a few controls that infrastructure teams meet constantly:
| Source control | Testable assertion | Evidence source |
|---|---|---|
| HIPAA 164.312(b) — Audit controls | Every read and write against PHI tables is logged with actor identity and timestamp | pgaudit configuration on Postgres; CloudTrail data events on PHI buckets |
| HIPAA 164.312(a)(2)(iv) — Encryption and decryption (addressable) | Every PHI data store is encrypted at rest with a customer-managed KMS key, or a documented alternative exists | RDS StorageEncrypted flag; S3 default encryption; KMS key policies |
| PCI DSS v4.0 8.4.2 — MFA into the CDE | All non-console access into the cardholder data environment requires MFA | Okta or Entra sign-on policies; IAM condition keys on CDE roles |
| ISO 27001:2022 A.8.15 — Logging | Security logs are retained for the period the organization defined and protected from tampering | CloudWatch log group retention; S3 Object Lock on log archive |
| SOC 2 CC6.1 — Logical access | No IAM principal holds wildcard permissions on production data stores | IAM Access Analyzer findings; policy simulation output |
Note that every row ends in something an API can return. If a requirement cannot be pushed to that point, it is a governance control, and the agent should mark it for interview rather than invent evidence.
Collecting Evidence Without Widening Your Blast Radius
The evidence stage is where a gap agent can quietly become a security finding of its own. An agent with broad read access across production is, after all, exactly the kind of over-privileged principal the assessment is supposed to catch.
Give the agent read-only, per-system roles scoped to the configuration and log APIs each requirement needs. It should never hold write access to a system it assesses, and every call should log under its own identity.
In practice, that means a dedicated IAM role per account with a policy built from the evidence map, not from SecurityAudit plus a wish list. The patterns in agent identity and access apply directly: a distinct principal, short-lived credentials, and no shared service accounts with the humans being assessed.
Most of the evidence you need already exists in services you run. AWS Config, Security Hub, IAM Access Analyzer, CloudTrail, and your Terraform state cover a large share of the technical safeguards, while the GitHub and Okta APIs cover change management and access review.
Every evidence object should record the query, the system, the credential identity, the timestamp, and a hash of the raw response. This is the same discipline described in agent audit trails, and it is what lets a reviewer, or an assessor, reproduce what the agent saw.
Why Configuration Evidence Beats Policy Documents
Your encryption policy says every PHI store is encrypted with a customer-managed key. Your RDS API may say that one read replica, created during an incident eighteen months ago, uses the AWS-managed default key instead.
That diff between documented intent and deployed state is where gap agents earn their keep. Humans reviewing screenshots tend to check the primary and move on, while an agent enumerating every resource in scope does not get bored at resource forty — the same drift problem covered in agent runbook drift, pointed at your compliance posture instead of your runbooks.
How Do You Keep A Human Sign-Off On Every Gap Finding?
Make sign-off a state transition the agent is structurally unable to perform. A finding moves from draft to reviewed to accepted or rejected, and the accepted state requires an authenticated action from the named owner, enforced in the workflow service rather than in the prompt.
Model each finding as a state machine where only a named reviewer can set accepted or rejected. Enforce it in the workflow service, not the prompt, so no model output can mark a gap as final.
The reviewer should see four things side by side: the source clause, the testable assertion, the evidence objects, and the agent's reasoning. If any of the four is missing, the finding is not reviewable and should bounce back automatically.
Rule of thumb: the agent is allowed to say no evidence found. Only a named person is allowed to say gap.
Rejections are the most valuable data the system produces. Each rejection reason, such as a compensating control the agent missed or an inherited control it did not recognize, goes into the evaluation set so the next run makes that mistake less often.
Over time, you may want to consider letting high-agreement control families move faster through review. That is the supervised autonomy model where an agent earns scope through its measured track record, and even at the top of that ladder, the signature on a finding stays human.
Where Gap Agents Fail In Practice
The failure modes are predictable, which is good news. Most of them come from the agent confusing what it could not see with what is not there.
The ones we watch for include:
- Absence of evidence read as non-compliance. A missing log sample may mean the agent's role lacks permission, not that logging is off. Every no-evidence result should state which call returned empty and why.
- Compensating controls missed. An addressable HIPAA specification met by a documented alternative looks like a gap to an agent checking only the default mechanism.
- Inherited controls flagged. Physical safeguards for a Bedrock or RDS workload are covered by AWS under the shared responsibility model, and the agent needs the provider's attestation in its evidence map rather than a failing check.
- Stale snapshots. Evidence collected on Monday does not describe Friday's environment after a Terraform apply. Every object needs an expiry tied to your audit window.
- Interpretation drift across model versions. A model upgrade can shift how the agent reads periodic or appropriate. Pin the model version, and treat a change as a release that reruns the full evaluation suite, the same way model deprecation migrations should be handled.
Controls your cloud provider operates, such as data center physical security, should map to the provider's attestation report. Flagging them as gaps wastes reviewer time and signals the mapping is wrong.
Each of these failures shows up first as a rejection in review. This is because the human checkpoint doubles as your detection layer, which is a strong argument for never removing it.
How Do You Know Your Gap Agent Is Working?
Measure it the way you would measure any production agent, with metrics tied to the decision it supports. The useful ones are reviewer acceptance rate per control family, false-gap rate, evidence freshness at report time, and rerun stability.
Rerun stability deserves attention on its own. If you run the same assessment twice on identical inputs and get different findings, the variance is coming from inference rather than your environment, and an assessor will reasonably ask which run to believe.
The practical mitigation is to make the agent's findings depend on deterministic checks wherever possible. For example, the encryption assertion should resolve from a boolean returned by the RDS API, with the model writing the explanation rather than deciding the outcome.
Cost is rarely the constraint here. A full run across roughly a hundred requirements, at tens of thousands of input tokens per requirement once evidence is attached, lands in the low millions of tokens — single-digit to low double-digit dollars of model spend at Sonnet-class Bedrock pricing, which is trivial next to the reviewer hours it saves.
That said, evidence payloads grow quickly when someone attaches full CloudTrail exports. Summarize at collection time, cache the static portions of the prompt, and track spend per run using the guardrails in AI agent cost governance.
A Practical Rollout Sequence
Start narrow and run in shadow mode before anyone relies on the output. Here's the sequence we would use for a first deployment:
- Pick one control family. Technical access controls or logging are good candidates, because the evidence is almost entirely API-reachable.
- Build the evidence map by hand once. An engineer confirms which systems are in scope for each requirement, and that map becomes a versioned artifact.
- Run in shadow mode. The agent assesses the same family your team is already assessing manually, and nobody sees its output until the manual pass is done.
- Compare and record disagreements. Every place the agent and the humans differ becomes a labeled case in the agent evaluation suite.
- Turn on review routing. Once acceptance rates hold steady across two or three runs, route drafts to control owners for real sign-off.
- Expand one family at a time. Governance-heavy families come last, and some will stay interview-driven permanently.
Run the agent in shadow mode on one control family while your team assesses it manually. Promote it to live review routing only after acceptance rates hold steady across two or three runs.
Overall, the sequence trades speed for a defensible record. Remember that the goal is a gap report your assessor accepts on the first read, not the fastest possible first draft.
Scoping A Gap Assessment Against Your Own Stack
Gap assessment is one of the clearer fits for agents in regulated infrastructure, because the work is mostly evidence collection and the judgment can be cleanly separated out. The architecture decisions that matter are the same ones covered across our AI agent operations series: scoped identity, durable audit trails, measured evaluation, and a human signature where the decision carries liability.
If you're mapping HIPAA, PCI DSS, SOC 2, or ISO 27001 against an AWS estate and want a second set of eyes before you build, the team at iSimplifyMe designs and operates production agent systems across regulated data environments every week. Reach out for a working session — we'll map your first control family to its evidence sources, name the failure modes you're about to hit, and leave you with a review workflow your control owners can sign against.