By mid-2026, on a lot of content-heavy sites, the automated requests hitting your origin from AI crawlers reportedly rival — and on some properties exceed — the requests coming from actual humans. GPTBot, ClaudeBot, and PerplexityBot now show up in your access logs next to Googlebot, except almost nobody decided, on purpose, whether they belong there.
That decision lives at the infrastructure layer — the CDN, the WAF, the edge rules, and the robots directives — and on most properties it lives nowhere in particular. Marketing wants the citations, security wants the bots gone, and the person holding the Cloudflare or Fastly console was never in the room.
Who Actually Owns AI Crawler Access?
Ask three teams who governs AI-crawler access and you will get three different answers, which is another way of saying no one governs it. Marketing assumes robots.txt covers it, security assumes bot management covers it, and the platform team assumes marketing has an opinion on record.
AI crawler access control is the combination of edge rules, bot-management policies, and robots directives that decide whether crawlers like GPTBot and ClaudeBot can read your content. It rarely has one owner.
The gap matters because the two failure modes are opposite, and both are expensive. Block everything and you vanish from the AI answers your buyers now use; allow everything and you hand your entire content library to model training, with no attribution and a real bandwidth bill attached.
This is an infrastructure decision wearing a marketing costume. Whoever owns the CDN and the WAF is already making it — passively, through default config — every day the site is up.
What The Three Control Points Actually Do
There are exactly three places you can enforce a crawler policy, and they operate at different layers with very different levels of teeth. Knowing which does what is the difference between a policy that holds and a robots.txt file everyone politely ignores.
- robots.txt directives. This is the published request layer, where you name a user-agent token and Allow or Disallow paths. It is honored only by crawlers that choose to honor it, which makes it a statement of intent rather than a control.
- Edge bot management. This is the enforcement layer at your CDN — Cloudflare, Fastly, Akamai — where requests are verified by IP range and signature and can be blocked, challenged, or rate-limited before they ever reach origin. This is the only layer that can actually stop a crawler that ignores your robots file.
- Origin and rate-limit rules. This is the backstop, where you cap request volume per source and shed load from anything crawling aggressively. It protects your bandwidth and your P95 latency even when identity is ambiguous.
robots.txt is only advisory. Well-behaved crawlers like GPTBot honor a Disallow, but the directive is just a request — spoofed or non-compliant bots ignore it, so it cannot block anything by itself.
Keep in mind that these three layers are meant to stack rather than substitute for one another. The robots file states the policy, the edge enforces it, and the rate limiter catches whatever slips the identity check.
The Crawlers Actually Hitting Your Origin
Before you can write a policy, you need to know what is knocking. The names below are the user-agent tokens you will see in logs and the ones you target in both robots.txt and your edge rules.
| Crawler | Operator | Primary job | robots.txt token | Blocking it costs you |
|---|---|---|---|---|
| GPTBot | OpenAI | Training data collection | GPTBot | Nothing — no citations depend on it |
| OAI-SearchBot | OpenAI | ChatGPT search citations | OAI-SearchBot | Your presence in ChatGPT search answers |
| ChatGPT-User | OpenAI | User-triggered fetch in ChatGPT | ChatGPT-User | On-demand answers when users share your URL |
| ClaudeBot | Anthropic | Broad crawl | ClaudeBot | Content availability to Claude |
| PerplexityBot | Perplexity | Index for Perplexity answers | PerplexityBot | Citations in Perplexity results |
| Google-Extended | Gemini / Vertex training | Google-Extended | Nothing in Google Search — only Gemini training | |
| CCBot | Common Crawl | Open dataset used for training | CCBot | Inclusion in a dataset many models train on |
| Bytespider | ByteDance | Training data collection | Bytespider | Little for most Western sites |
Note that this list moves. New agents appear quarterly, and vendors split one crawler into several as they separate training from retrieval — which is exactly the distinction most policies get wrong.
Why A robots.txt-Only Policy Fails
The most common posture in the wild is a robots.txt entry and nothing behind it. That works right up until a crawler decides not to comply, and several already have a public track record of crawling content that asked to be left alone.
Perplexity, for instance, has been reported by Cloudflare to fetch pages through undeclared user-agents after being disallowed, which means the robots directive did nothing. Once a crawler is willing to ignore the file, only edge enforcement — identity verification plus a block or challenge — actually removes it.
Real enforcement lives at the edge. Cloudflare, Fastly, or Akamai bot management verifies crawler identity by IP and signature and can block, challenge, or rate-limit a bot, whereas robots.txt can only ask politely.
This is where ownership stops being academic. If the only control is a text file the marketing team edits, then infrastructure has no policy — it has a suggestion, and the suggestion is being declined.
A Third Option Between Allow And Block
Until recently the choice was binary: let a crawler in or shut it out. That is changing, because the infrastructure vendors have started to build a middle path where access becomes a transaction rather than a giveaway.
Cloudflare, for instance, rolled out a pay-per-crawl model in 2025 that lets a site charge AI operators for access at the edge, and it began defaulting new zones to block AI training crawlers unless the owner opts in. For infrastructure teams, that reframes the crawler question from allow-or-deny to allow, deny, or meter — and metering is only possible if you already control access at the edge rather than in a text file.
Beyond allow-or-block, Cloudflare's pay-per-crawl lets sites charge AI operators for access at the edge. Metering only works if you enforce crawler identity at the CDN, since robots.txt cannot gate access behind payment.
The Retrieval-Versus-Training Distinction Nobody Configures
The mistake that quietly costs the most is treating "OpenAI" or "Anthropic" as a single thing to allow or block. Each vendor runs separate crawlers for separate jobs, and the job determines whether blocking helps you or hurts you.
GPTBot crawls to build training data, and blocking it keeps your content out of future model weights. OAI-SearchBot and ChatGPT-User, however, fetch pages at answer time to cite them in ChatGPT — so blocking those is what actually erases you from the answers your buyers read.
Blocking GPTBot stops training crawls but not citations. ChatGPT's answers are fetched by OAI-SearchBot and ChatGPT-User, so blocking those agents is what removes you from the AI answers your buyers see.
The same split runs through the other vendors. ClaudeBot handles Anthropic's broad crawl, Google-Extended governs Gemini training while Googlebot still indexes you for Search, and Perplexity separates its index crawler from its user-triggered fetch.
The practical upshot is that a blanket vendor block is almost never what you want. Most teams want to keep training crawlers out and let retrieval agents in, which is a per-agent decision rather than a per-vendor one, and it maps directly onto the retrieval blind spots that keep you out of AI answers.
How To Decide What To Allow
The policy itself is short once you separate the axes. Run every crawler through three questions and the answer falls out.
- Training or retrieval? Retrieval agents earn citations and referral traffic, so most sites allow them. Training crawlers give you no attribution, which makes blocking them a defensible default unless you have a licensing deal.
- Compliant or not? If a crawler honors robots.txt, the file is enough. If it has a record of ignoring it, the rule belongs at the edge with a hard block or a managed challenge.
- What does the traffic cost? Pull the crawl-to-referral ratio from your logs. A bot that fetches thousands of pages and sends back a handful of visitors is a bandwidth line item, not a growth channel.
The team that owns the CDN and WAF owns AI crawler policy, whether they know it or not. The default config already decides who reads your content, so treat it as a governed decision rather than a default.
What's more, this is a decision worth revisiting on a schedule, because the crawler roster and each vendor's compliance record both drift. A quarterly review of your access logs against your allow-and-deny rules keeps the policy honest, and it pairs naturally with the way you already track your share of AI citations.
If you want the upstream context on why any of this affects revenue, the mechanics sit inside answer engine optimization and the broader question of how AEO differs from traditional SEO. Crawler access is the gate, and everything downstream assumes the gate is open to the right bots.
Where This Leaves The Infrastructure Team
The uncomfortable truth is that you are already enforcing a policy — the default one your CDN shipped with. The only open question is whether it reflects a decision anyone made on purpose.
Pull your bot traffic, name the agents, and write the three-line policy: training crawlers denied, retrieval agents allowed, non-compliant bots blocked at the edge. That single afternoon of work closes a governance gap most of your competitors have not noticed they have.
If you are scoping AI-crawler policy and want a second set of eyes on the edge rules before you ship them, the team at iSimplifyMe builds and operates production agent and retrieval infrastructure across CDN, WAF, and data-warehouse environments every week. Reach out for a working session — we will map your crawler traffic, name the agents you should block versus allow, and leave you with a robots-plus-edge policy you can deploy the same day.
Frequently Asked Questions
The questions infrastructure teams ask most often when they first take ownership of AI-crawler policy.
Does blocking GPTBot remove my site from ChatGPT answers?
No — GPTBot handles training crawls, while ChatGPT's answers are fetched by OAI-SearchBot and ChatGPT-User. To keep citations but opt out of training, block GPTBot and allow the retrieval agents in both robots.txt and your edge rules.
Is robots.txt enough to block AI crawlers?
Not on its own — robots.txt is advisory, so compliant bots honor it but spoofed or non-compliant crawlers ignore it entirely. Enforcement requires edge bot management at your CDN or WAF that verifies bot identity and can block or challenge requests.
Which AI crawlers should infrastructure teams know about?
The main ones are GPTBot, ChatGPT-User, and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, Google-Extended, CCBot from Common Crawl, Bytespider, and Amazonbot. Each has its own user-agent and robots.txt token.
Does Google-Extended affect my Google Search rankings?
No — Google-Extended only controls whether your content trains Gemini and Vertex AI models. Googlebot still indexes your pages for Search and AI Overviews regardless, so disallowing Google-Extended does not hurt your organic rankings.
What does allowing every AI crawler actually cost?
Two things: bandwidth from high-volume crawlers that can rival human traffic, and handing your full content library to model training with no attribution. Cloudflare data shows some crawlers fetch thousands of pages for every referral they send back.