FAQ

What should a corporate website change for the AI search era?

Five changes. (1) Bot reachability: robots.txt explicitly allows AI crawlers and live fetches are not stopped by firewalls or challenge pages. (2) A machine-readable surface: an llms.txt fact file (optional aid), a complete sitemap, Markdown versions of core pages (optional). (3) Entity structure: JSON-LD naming entity types such as Organization, clean title/H1 discipline, hreflang pairing on bilingual sites. (4) Question-shaped content: pages organised around buyer questions, short answers first, FAQ schema attached. (5) Measurability: analytics that separate human and bot traffic and attribute AI-referred inquiries. Negency's own site was rebuilt to exactly this standard, and the public audit tool scores against it.

Figure 01 · Four rebuild steps: order and acceptance test

  1. 01

    Step 1: Clear the bot gate

    Live-test the homepage and core pages as three or more AI crawlers; check robots.txt and WAF rules line by line

    Acceptance: All return 200 with extractable body text

    Reachability mean 91.1 across 30 sites, 21 at full marks
  2. 02

    Step 2: Add the machine-readable surface

    Clean up the sitemap (no dead links, current version); llms.txt and Markdown versions are optional aids

    Acceptance: Body text extractable from raw HTML

    15 of 30 had sitemap problems; machine-readable mean 69.2 (median 68.5), weakest of five
  3. 03

    Step 3: Structure and content

    Add Organization/WebSite JSON-LD; one H1 per page; short answer first, then expansion

    Acceptance: Markup matches visible text item by item

    17 of 30 had no JSON-LD; 13 of 30 had missing or non-unique H1; 10 of 30 scored below 70 on both machine readability and entity structure
  4. 04

    Step 4: Measure

    Report bot and human traffic separately; tag AI-referred inquiries distinctly

    Acceptance: Before-and-after re-tests with the same instrument

    No corresponding measurement; this step is a method requirement
Data: Negency Flagship Report 2026 (Issue 1), measured across 30 Chinese exporter websites in a single-day sweep on 30 October 2026. The report measures technical visibility only, not mention rates.

The order of work: four steps, not interchangeable

Website rebuilds have a fixed order: clear the bot gate first, then add the machine-readable surface, then fix structure and content, and finally measure. In the wrong order, budget goes to the wrong fixes — a site bots cannot enter will never be cited. Each step comes with an acceptance test: pass it before moving on.

Step 1: Clear the bot gate first

Many sites are not unquotable — bots simply cannot get in: a Disallow in robots, a CDN default blocking AI crawlers, or a 200 that is really a challenge page. Fetch your homepage as GPTBot, ClaudeBot and OAI-SearchBot — a live test beats reading configuration.

A live test checks three things: a 200 status, real body text in the response (not a challenge page or an empty shell), and the same result under each identity. Configuration alone misses the most common failure: robots allows the crawler while the CDN or WAF blocks its IPs.

Acceptance test: the homepage and core product pages return 200 with extractable body text under three or more AI crawler identities, with robots.txt and WAF rules checked line by line. Across 30 Chinese exporter websites (Negency Flagship Report 2026, Issue 1, sampled 30 October 2026), crawl reachability averaged 91.1/100 and 21 sites scored full marks — most pass this gate, but failing it makes everything else moot. (Sampling note: the sample is the global homepages of 30 Chinese exporter websites, measured in a single-day sweep on 30 October 2026 with the Negency audit engine — about 30 signals in five groups. The report measures technical visibility only — whether AI can reach and read the site — not brand mention rates in AI answers.)

Step 2: Add the machine-readable surface

The sitemap guarantees full discovery; clear structure and factual wording let machines read accurately. llms.txt (the company's facts as one document) and Markdown versions of core pages are optional aids: they lower the cost of machine reading, are not a requirement for AI search, and guarantee no citation. Google's docs state that appearing in AI Overviews and AI Mode requires no new machine-readable files or AI text files — llms.txt is an auxiliary piece, not a pass.

This layer is easy to underestimate: 15 of the 30 sampled sites had sitemap problems, and the machine-readable dimension averaged only 69.2 (median 68.5), the weakest of the five. Dead links, a version untouched for years, a file never read — all common. Fix the sitemap first; llms.txt comes after.

Acceptance test: the sitemap exists, is current, has no dead links and has been submitted; core-page body text is extractable from raw HTML; if llms.txt is live, its facts match the website's and update with them.

Step 3: Structure and content

Add Organization/WebSite entity types to JSON-LD; keep one H1 per page; organise content around questions with the quotable answer first. AI cites passages, not pages.

The numbers make the case: 17 of 30 sampled sites had no JSON-LD at all, 13 had missing or non-unique H1 headings, and 10 scored below 70 on both machine readability and entity structure. Each fix is a small job: Organization markup usually takes half a day for a first version, H1 uniqueness is one site-wide sweep, a short answer is two or three sentences. The hard part is reconciling every page — markup contradicting the visible text is worse than no markup.

Acceptance test: every core page carries Organization markup consistent with its visible text; every page has exactly one H1 stating its topic; core concept pages lead with a quotable short answer.

Step 4: Finally, measure

Separate bot traffic from human traffic in your reporting: which bot classes visit, which pages they fetch, how often they succeed. Tag AI-referred inquiries distinctly, or the rebuild's effect can never be proven.

Measure on two levels: crawl level (per-engine visit frequency, coverage and success rate in the logs) and business level (AI-referred visits and inquiries carrying a source tag). Take a baseline before the rebuild and re-test with the same instrument afterwards.

Acceptance test: bot and human traffic are reported separately; AI-referred visits and inquiries carry their own tags; before-and-after re-tests use the same instrument.

A concrete example: Negency's own website

Take Negency's own website: its AI visibility audit on 11 November 2026 scored 97/100 overall, with "performance and basics" at 80 and the homepage responding in 3.08 seconds. Even a 97-point site has a weakest dimension — rebuilding is a loop: find it, fix it, re-test, find the next one. Run the audit for a baseline first, then work the four steps in order.

How-to detail for each piece lives in the related questions on this site — crawler policy, the four JSON-LD types, writing llms.txt; the first step of any rebuild is a free AI visibility audit for your own baseline.

Related questions

How is GEO different from SEO?

SEO optimises a page's rank in a list of links, aiming for clicks. GEO (Generative Engine Optimization) optimises how discoverable, understandable, trustworthy and citable a company is inside AI-generated answers — the goal is to be in the answer itself. They share technical foundations (crawlability, structured data, content quality), but GEO additionally requires governed entity facts, a machine-readable surface (llms.txt, Markdown mirrors) and a third-party mention ecosystem. A site that ranks well is not automatically visible to AI: engines cite the companies whose facts they can fetch, parse and verify. The concrete differences sit in four dimensions — goal, object, content structure and evaluation — compared item by item in the body below.

How can Chinese companies going global build AI search visibility?

For Chinese companies going global, the visibility gap is rarely content volume — it is the machine-readable surface. Our audit of 30 Chinese outbound companies found machine-readability the weakest category (69.2/100 average): 17 had no JSON-LD on the homepage, 15 had a missing or thin sitemap, 13 had a missing or duplicated H1. The path has four steps: audit bot reachability and machine-readability first; govern entity facts so Chinese and English statements agree; publish question-shaped content built on real overseas buyer questions; then earn third-party mentions (media, communities, partners) and verify with periodic resampling against a fixed question set.

What is llms.txt, and should every company website have one?

llms.txt is a plain-text brief at the site root written for AI systems: who the company is, what it does, where the core pages live, and the canonical facts and definitions. It does not by itself raise citation rates — in Ahrefs' May 2026 study of 137,210 domains, 97% of published llms.txt files received zero requests that month, so its direct effect is near zero — but it is the cheapest fact-governance move available: one file anchors a single version of the truth that crawlers can rely on. Verdict: worth adding, useless alone; it works as part of entity-fact governance, structured data and question-shaped content. Negency's llms.txt is a full fact bible: company definition, offering boundaries, sourced cases and citation rules.

What should an enterprise prepare before deploying AI agents?

Four preparations, and we advise against starting without any of them. (1) Scenario selection: pilot on a workflow that is frequent, comparatively rule-clear and verifiable — not the most glamorous one. (2) Knowledge and data foundations: govern the knowledge base first (deduplication, versioning, permissions); a dirty knowledge base only scales errors. (3) Governance design: permission boundaries, human approval points and audit logs decided before launch, not after. (4) A measurement baseline: record the manual process's time and error rates before the pilot, or value can never be proven afterwards. Negency's method is a small controlled pilot, verified before scaling, with governance as a design input rather than a patch.

Why do enterprise AI agents need governance and human approval points?

Because an agent differs from ordinary software in one decisive way: it chooses its own next actions, and its errors amplify at machine speed. Governance answers four questions — which data it may touch (permissions), which actions require a human (approval points), what it did (audit trail), and how it stops (kill switch). Approval points are not distrust of the technology; they keep accountability on the human side: external sends, money, deletions and commitments pass a human, everything else flows automatically. The EU AI Act likewise requires human oversight for higher-risk systems — governing early is the cheapest compliance path.