FAQ

Should websites allow AI crawlers like GPTBot and ClaudeBot?

Separate the crawlers by purpose before deciding: search-index crawlers (OAI-SearchBot, PerplexityBot) and user-triggered fetches (ChatGPT-User) decide whether you can appear in AI answers at all; training crawlers (GPTBot and peers) are a separate licensing choice — a company may allow search while declining training. For a website whose job is winning customers, explicitly allow at least the search and user agents: if they cannot fetch your pages, AI answers cannot cite you. Whether to allow training follows your own content-licensing policy. In our audit of 30 Chinese outbound companies, only 3 blocked all three major AI crawlers — and those three have capped their own visibility ceiling. Allowing is not exposure: keep admin paths closed, rate-limit by real IP against abuse, and declare training/search/input permissions separately via Content-Signal. Negency's own robots policy — allow everything, close only admin paths, rate-limit for protection — is public and reusable.

Figure 01 · Sample robots.txt: allow search, block training

# Example policy: allow search and citation crawlers, block training ones
# Adjust to your own copyright and acquisition position; Negency's
# default is training allowed, adjustable per crawler

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /admin/
Disallow: /internal/
  1. One group per crawler classRules apply only to the named crawler; OAI-SearchBot and GPTBot are independent of each other
  2. Google-Extended is a control token, not a crawlerNo separate user-agent string, never appears in logs; states training and grounding use only, and does not affect Search inclusion
  3. Content-Signal is a preference, not a blockDraft/proposal stage, not an adopted standard; omitting a signal states no preference
  4. User-triggered fetchers generally ignore robots.txtControl ChatGPT-User and friends at the firewall/IP layer if needed; this sample sets no group for them
  5. Sensitive paths stay closed regardless/admin/ and internal tools are closed to all crawlers
Crawler names and behaviour follow each vendor's official documentation (OpenAI, Anthropic, Perplexity, Google, Common Crawl — see Sources). After changing robots.txt, verify actual crawls in the logs; OpenAI notes its systems take about 24 hours to adjust. Content-Signal is a Cloudflare extension proposed in September 2025; IETF standardisation is in progress and it is not yet a formal standard.

Separate the three crawler classes

Training crawlers (GPTBot, ClaudeBot, CCBot) fetch for model training; search-index crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) build AI search indexes; user-driven fetchers (ChatGPT-User, Claude-User, Perplexity-User) open your page live at a user's request. The last two drive acquisition directly; training crawlers shape long-term presence in model corpora.

One more distinction: search and training crawlers obey robots.txt, but user-driven fetchers generally do not — OpenAI's and Perplexity's docs both say robots.txt may not apply to a user-triggered fetch. Blocking them takes firewall or IP-level action.

One easily confused name: Google-Extended is not a crawler — it has no separate user-agent string and never appears in access logs. It is a robots.txt control token stating whether content Google crawls may train future Gemini models or ground Gemini Apps. Google's docs are explicit: Google-Extended does not affect inclusion or ranking in Google Search, nor does it decide AI Overviews.

The real cost of blocking

Most blocking is accidental, not chosen: a CDN default, a security vendor's rule pack, one broad Disallow line in robots.txt. Fetch your site as a bot before debating policy — the most common finding is operators who believe they allow crawlers while blocking all of them. A subtler source: a managed service writing your robots.txt for you — Cloudflare's September 2025 Content Signals launch added a default line (search=yes, ai-train=no) to over 3.8 million domains.

Protection, in three parts

  • Keep sensitive paths closed — /admin, backends and internal tools stay shut regardless of crawler policy;
  • Rate-limit by real IP — aimed at flood-level abuse, not normal indexing;
  • Declare Content-Signal — state ai-train / search / ai-input permissions separately, in fine grain.

Content-Signal's status, stated plainly: a robots.txt extension proposed by Cloudflare in September 2025 (three signals, each yes or no; omitting one states no preference), with the IETF's AI Preferences working group standardising a similar vocabulary — still a draft/proposal, not an adopted standard. Cloudflare's own wording: signals express preferences, not technical countermeasures.

When blocking is right

If the content itself is the paid asset — full research reports, databases — or the site has no acquisition role, blocking training crawlers is legitimate. The test is singular: is this channel a customer channel for you? If yes, do not lock your own door.

When you do block, block the right thing: blocking GPTBot affects training only, not ChatGPT search citations — that is OAI-SearchBot's job, and OpenAI's docs state the two settings are independent. The most common mistake is one broad Disallow shutting both together.

Decide per engine class, not with one switch

  • Search and citation crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot): these decide whether you can appear in AI answers at all — if you do GEO, allow them. Google's AI Overviews and AI Mode rest on Googlebot's index, so blocking Googlebot exits classic search and AI Overviews together;
  • User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User): a real person's question fetches your page in real time — allow them; these fetches produce the highest-quality citations;
  • Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended): a copyright and data-position decision. Allowing them does not directly buy citations, but blocking everything reduces how familiar engines are with your site. Negency's position: allow search and user-triggered classes fully; allow training crawlers by default, adjustable one by one.

A rule in motion: an IETF draft of 30 October 2026 proposed a shared training-preference signal, with Google, Microsoft, OpenAI and Anthropic announcing support — Google aims for early 2027, and Anthropic has recognised the signal since 30 October 2026. Until it lands, each vendor's current documentation governs.

Three things to do after allowing them

  • Audit robots.txt and the firewall/WAF rule by rule: many sites are blocked by a CDN or security rule their owners never set — we have audited sites whose robots file allowed crawlers while the WAF blocked their IP ranges;
  • Verify in the logs: allowed does not mean visiting — count actual crawl frequency and coverage per engine. Do not trust the user-agent string alone (it is easy to forge); check each vendor's published IP ranges. OpenAI notes about 24 hours from a robots.txt update before its systems adjust, so do not judge a change the same day;
  • Keep a readable version ready: server-rendered body text and machine-readable surfaces for core pages (Markdown versions, llms.txt) lower the engines' crawl and parse cost.

Not sure who your site currently allows? A free AI visibility audit simulates the major AI crawlers — look at the result before setting policy.

Related questions

How is GEO different from SEO?

SEO optimises a page's rank in a list of links, aiming for clicks. GEO (Generative Engine Optimization) optimises how discoverable, understandable, trustworthy and citable a company is inside AI-generated answers — the goal is to be in the answer itself. They share technical foundations (crawlability, structured data, content quality), but GEO additionally requires governed entity facts, a machine-readable surface (llms.txt, Markdown mirrors) and a third-party mention ecosystem. A site that ranks well is not automatically visible to AI: engines cite the companies whose facts they can fetch, parse and verify. The concrete differences sit in four dimensions — goal, object, content structure and evaluation — compared item by item in the body below.

How can Chinese companies going global build AI search visibility?

For Chinese companies going global, the visibility gap is rarely content volume — it is the machine-readable surface. Our audit of 30 Chinese outbound companies found machine-readability the weakest category (69.2/100 average): 17 had no JSON-LD on the homepage, 15 had a missing or thin sitemap, 13 had a missing or duplicated H1. The path has four steps: audit bot reachability and machine-readability first; govern entity facts so Chinese and English statements agree; publish question-shaped content built on real overseas buyer questions; then earn third-party mentions (media, communities, partners) and verify with periodic resampling against a fixed question set.

What should a corporate website change for the AI search era?

Five changes. (1) Bot reachability: robots.txt explicitly allows AI crawlers and live fetches are not stopped by firewalls or challenge pages. (2) A machine-readable surface: an llms.txt fact file (optional aid), a complete sitemap, Markdown versions of core pages (optional). (3) Entity structure: JSON-LD naming entity types such as Organization, clean title/H1 discipline, hreflang pairing on bilingual sites. (4) Question-shaped content: pages organised around buyer questions, short answers first, FAQ schema attached. (5) Measurability: analytics that separate human and bot traffic and attribute AI-referred inquiries. Negency's own site was rebuilt to exactly this standard, and the public audit tool scores against it.

What is llms.txt, and should every company website have one?

llms.txt is a plain-text brief at the site root written for AI systems: who the company is, what it does, where the core pages live, and the canonical facts and definitions. It does not by itself raise citation rates — in Ahrefs' May 2026 study of 137,210 domains, 97% of published llms.txt files received zero requests that month, so its direct effect is near zero — but it is the cheapest fact-governance move available: one file anchors a single version of the truth that crawlers can rely on. Verdict: worth adding, useless alone; it works as part of entity-fact governance, structured data and question-shaped content. Negency's llms.txt is a full fact bible: company definition, offering boundaries, sourced cases and citation rules.

What should an enterprise prepare before deploying AI agents?

Four preparations, and we advise against starting without any of them. (1) Scenario selection: pilot on a workflow that is frequent, comparatively rule-clear and verifiable — not the most glamorous one. (2) Knowledge and data foundations: govern the knowledge base first (deduplication, versioning, permissions); a dirty knowledge base only scales errors. (3) Governance design: permission boundaries, human approval points and audit logs decided before launch, not after. (4) A measurement baseline: record the manual process's time and error rates before the pilot, or value can never be proven afterwards. Negency's method is a small controlled pilot, verified before scaling, with governance as a design input rather than a patch.