Scenario selection: easy first
A pilot scenario should be frequent (happens daily), bounded (right and wrong are judgable) and contained (errors can be rolled back). Support Q&A, document processing and internal knowledge retrieval usually qualify; deciding for management does not — that is the destination, not the starting point.
All three tests must pass: frequent but unbounded work ("judge how interested this prospect is") can't be verified at scale; bounded but rare work saves less than the build cost; frequent and bounded but irreversible work (automatic dunning letters) turns one misjudgement into an external incident. See the safest first use cases.
Knowledge foundations: govern before connecting
An agent's answer quality is capped by its knowledge base. Before connecting: merge duplicates, retire stale versions, classify sensitive content, give every document an owner. Skip this and the agent will confidently quote a three-year-old policy.
Stale versions are the trap: a policy revised three times still has all three versions on the shared drive, and the agent may pull the old one by similarity and answer with it, confidently. Keep one current version per document; retire the rest from retrieval scope. If the agent also reads the public website, check its machine-readable foundations first: in Negency's flagship measurement of the global homepages of 30 Chinese exporter companies (a single-day sample on 30 October 2026, measuring technical website visibility only — not brand mention rates in AI answers), machine readability averaged only 69.2 (median 68.5) and 17 of 30 sites had no JSON-LD at all. Source: Flagship Report 2026, Issue 1.
Governance: rules before launch
Who may instruct the agent to do what, which actions require human approval, how long logs are kept — written into the pilot design. The approval list must name concrete actions ("send a customer email", "change a price list"), not vague phrases like "human review when necessary". In the EU, GDPR and AI Act classification come on top: classify first, deploy second — the risk grade decides which obligations apply (see why agents need governance and approval points).
Measurement: no baseline, no result
Record the manual baseline when the pilot is chartered: handling time, first-contact resolution, error rate. Compare on the same definitions after launch; scale only on evidence, stop cleanly otherwise — fair to the business and to the agent alike.
"Same definitions" is the operative phrase: measure after launch exactly as you measured before. Excluding cases handed back to humans flatters the error rate overnight — the number fools the report, not the business. Write pass and stop lines into the brief, with re-measurement dates.
The pre-launch readiness checklist, item by item
Nine checks in four groups: what to do, what breaks if skipped, rough effort.
- Scenario: the pilot is named, owned, and sized — and passes all three tests: frequent, bounded, reversible. Skip it and nobody is accountable for acceptance. Effort: usually days.
- Scenario: a negative list states what the agent must never do. Without one, the agent may cross a line nobody drew — such as sending a commitment to a customer. Effort: usually days.
- Knowledge: pilot documents deduplicated, stale versions retired, every document owned. Skip it and the agent quotes outdated policy at volume. Effort: usually weeks.
- Knowledge: sensitive content classified; agent read scope matches employee permissions. Skip it and the agent may disclose salary bands or price floors to the wrong people. Effort: usually weeks.
- Governance: approval list final, approvers named. An approval point that only says "review when necessary" is no approval point. Effort: usually days.
- Governance: log fields and retention set; any incident replayable end to end. Without logs you cannot tell which step failed or who approved it. Effort: usually weeks.
- Governance: for EU operations, AI classification and the GDPR assessment come first. Finding out after launch means rebuilding compliance at far higher cost. Effort: usually weeks.
- Measurement: human baseline recorded — handling time, first-contact resolution, error rate. Without a baseline, pilot results are unjudgeable. Effort: usually 1–2 weeks of sampling.
- Measurement: pass and stop lines in the brief, with re-measurement dates scheduled. Without a stop line, a failing pilot keeps running on sunk cost. Effort: usually days.
Red flags: do not start yet if you see these
- The knowledge base holds three versions of the same policy and nobody can say which one is current;
- Approval points say only "human review when necessary", with no action list and no named owner;
- Nobody can state the current process's handling time or error rate;
- The pilot scenario is "advise management on decisions" — a task with no objective right or wrong.
All nine green, no red flags: start the small, controlled pilot. Anything missing gets fixed first, in checklist order.