Separate the three crawler classes
Training crawlers (GPTBot, ClaudeBot, CCBot) fetch for model training; search-index crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) build AI search indexes; user-driven fetchers (ChatGPT-User, Claude-User, Perplexity-User) open your page live at a user's request. The last two drive acquisition directly; training crawlers shape long-term presence in model corpora.
One more distinction: search and training crawlers obey robots.txt, but user-driven fetchers generally do not — OpenAI's and Perplexity's docs both say robots.txt may not apply to a user-triggered fetch. Blocking them takes firewall or IP-level action.
One easily confused name: Google-Extended is not a crawler — it has no separate user-agent string and never appears in access logs. It is a robots.txt control token stating whether content Google crawls may train future Gemini models or ground Gemini Apps. Google's docs are explicit: Google-Extended does not affect inclusion or ranking in Google Search, nor does it decide AI Overviews.
The real cost of blocking
Most blocking is accidental, not chosen: a CDN default, a security vendor's rule pack, one broad Disallow line in robots.txt. Fetch your site as a bot before debating policy — the most common finding is operators who believe they allow crawlers while blocking all of them. A subtler source: a managed service writing your robots.txt for you — Cloudflare's September 2025 Content Signals launch added a default line (search=yes, ai-train=no) to over 3.8 million domains.
Protection, in three parts
- Keep sensitive paths closed — /admin, backends and internal tools stay shut regardless of crawler policy;
- Rate-limit by real IP — aimed at flood-level abuse, not normal indexing;
- Declare Content-Signal — state ai-train / search / ai-input permissions separately, in fine grain.
Content-Signal's status, stated plainly: a robots.txt extension proposed by Cloudflare in September 2025 (three signals, each yes or no; omitting one states no preference), with the IETF's AI Preferences working group standardising a similar vocabulary — still a draft/proposal, not an adopted standard. Cloudflare's own wording: signals express preferences, not technical countermeasures.
When blocking is right
If the content itself is the paid asset — full research reports, databases — or the site has no acquisition role, blocking training crawlers is legitimate. The test is singular: is this channel a customer channel for you? If yes, do not lock your own door.
When you do block, block the right thing: blocking GPTBot affects training only, not ChatGPT search citations — that is OAI-SearchBot's job, and OpenAI's docs state the two settings are independent. The most common mistake is one broad Disallow shutting both together.
Decide per engine class, not with one switch
- Search and citation crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot): these decide whether you can appear in AI answers at all — if you do GEO, allow them. Google's AI Overviews and AI Mode rest on Googlebot's index, so blocking Googlebot exits classic search and AI Overviews together;
- User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User): a real person's question fetches your page in real time — allow them; these fetches produce the highest-quality citations;
- Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended): a copyright and data-position decision. Allowing them does not directly buy citations, but blocking everything reduces how familiar engines are with your site. Negency's position: allow search and user-triggered classes fully; allow training crawlers by default, adjustable one by one.
A rule in motion: an IETF draft of 30 October 2026 proposed a shared training-preference signal, with Google, Microsoft, OpenAI and Anthropic announcing support — Google aims for early 2027, and Anthropic has recognised the signal since 30 October 2026. Until it lands, each vendor's current documentation governs.
Three things to do after allowing them
- Audit robots.txt and the firewall/WAF rule by rule: many sites are blocked by a CDN or security rule their owners never set — we have audited sites whose robots file allowed crawlers while the WAF blocked their IP ranges;
- Verify in the logs: allowed does not mean visiting — count actual crawl frequency and coverage per engine. Do not trust the user-agent string alone (it is easy to forge); check each vendor's published IP ranges. OpenAI notes about 24 hours from a robots.txt update before its systems adjust, so do not judge a change the same day;
- Keep a readable version ready: server-rendered body text and machine-readable surfaces for core pages (Markdown versions, llms.txt) lower the engines' crawl and parse cost.
Not sure who your site currently allows? A free AI visibility audit simulates the major AI crawlers — look at the result before setting policy.