The order of work: four steps, not interchangeable
Website rebuilds have a fixed order: clear the bot gate first, then add the machine-readable surface, then fix structure and content, and finally measure. In the wrong order, budget goes to the wrong fixes — a site bots cannot enter will never be cited. Each step comes with an acceptance test: pass it before moving on.
Step 1: Clear the bot gate first
Many sites are not unquotable — bots simply cannot get in: a Disallow in robots, a CDN default blocking AI crawlers, or a 200 that is really a challenge page. Fetch your homepage as GPTBot, ClaudeBot and OAI-SearchBot — a live test beats reading configuration.
A live test checks three things: a 200 status, real body text in the response (not a challenge page or an empty shell), and the same result under each identity. Configuration alone misses the most common failure: robots allows the crawler while the CDN or WAF blocks its IPs.
Acceptance test: the homepage and core product pages return 200 with extractable body text under three or more AI crawler identities, with robots.txt and WAF rules checked line by line. Across 30 Chinese exporter websites (Negency Flagship Report 2026, Issue 1, sampled 30 October 2026), crawl reachability averaged 91.1/100 and 21 sites scored full marks — most pass this gate, but failing it makes everything else moot. (Sampling note: the sample is the global homepages of 30 Chinese exporter websites, measured in a single-day sweep on 30 October 2026 with the Negency audit engine — about 30 signals in five groups. The report measures technical visibility only — whether AI can reach and read the site — not brand mention rates in AI answers.)
Step 2: Add the machine-readable surface
The sitemap guarantees full discovery; clear structure and factual wording let machines read accurately. llms.txt (the company's facts as one document) and Markdown versions of core pages are optional aids: they lower the cost of machine reading, are not a requirement for AI search, and guarantee no citation. Google's docs state that appearing in AI Overviews and AI Mode requires no new machine-readable files or AI text files — llms.txt is an auxiliary piece, not a pass.
This layer is easy to underestimate: 15 of the 30 sampled sites had sitemap problems, and the machine-readable dimension averaged only 69.2 (median 68.5), the weakest of the five. Dead links, a version untouched for years, a file never read — all common. Fix the sitemap first; llms.txt comes after.
Acceptance test: the sitemap exists, is current, has no dead links and has been submitted; core-page body text is extractable from raw HTML; if llms.txt is live, its facts match the website's and update with them.
Step 3: Structure and content
Add Organization/WebSite entity types to JSON-LD; keep one H1 per page; organise content around questions with the quotable answer first. AI cites passages, not pages.
The numbers make the case: 17 of 30 sampled sites had no JSON-LD at all, 13 had missing or non-unique H1 headings, and 10 scored below 70 on both machine readability and entity structure. Each fix is a small job: Organization markup usually takes half a day for a first version, H1 uniqueness is one site-wide sweep, a short answer is two or three sentences. The hard part is reconciling every page — markup contradicting the visible text is worse than no markup.
Acceptance test: every core page carries Organization markup consistent with its visible text; every page has exactly one H1 stating its topic; core concept pages lead with a quotable short answer.
Step 4: Finally, measure
Separate bot traffic from human traffic in your reporting: which bot classes visit, which pages they fetch, how often they succeed. Tag AI-referred inquiries distinctly, or the rebuild's effect can never be proven.
Measure on two levels: crawl level (per-engine visit frequency, coverage and success rate in the logs) and business level (AI-referred visits and inquiries carrying a source tag). Take a baseline before the rebuild and re-test with the same instrument afterwards.
Acceptance test: bot and human traffic are reported separately; AI-referred visits and inquiries carry their own tags; before-and-after re-tests use the same instrument.
A concrete example: Negency's own website
Take Negency's own website: its AI visibility audit on 11 November 2026 scored 97/100 overall, with "performance and basics" at 80 and the homepage responding in 3.08 seconds. Even a 97-point site has a weakest dimension — rebuilding is a loop: find it, fix it, re-test, find the next one. Run the audit for a baseline first, then work the four steps in order.
How-to detail for each piece lives in the related questions on this site — crawler policy, the four JSON-LD types, writing llms.txt; the first step of any rebuild is a free AI visibility audit for your own baseline.