Before choosing a vendor, write one sentence clearly: which step of which business process will this agent own, and how is success measured? Without that sentence, any selection criteria turn into a demo contest — and demos are the least reliable signal.

Questions to ask before procurement

  • Has the scenario defined a verifiable business outcome — handling time, resolution rate, error rate — rather than an untestable phrase like “improved efficiency”?
  • Does the vendor distinguish workflows from agents? A process whose steps are known in advance should be a workflow; only steps that need autonomous judgement deserve an agent. Building everything as an agent is a sign of over-engineering.
  • How are writes controlled? Which data can the agent change, which systems can it call, does every step pass a permission check and human approval, and is there a complete audit log?
  • How are models and data separated? Is enterprise data used for training, does it leave the country, and which third parties sit on the call chain?
  • How is the system evaluated? Is there an evaluation set built from real business cases with a baseline, or only spot-checks of a few conversations?
  • Who operates the system after delivery? Are model upgrades, knowledge updates, and incident handling included, and with what response times?
  • Are you locked into one model or platform? Is the cost of switching models or vendors written into the contract?

How to judge technically

Judge a vendor on three lines, not on the demo:

  • Scenario verifiability. Ask the vendor to state the scenario in testable form: inputs, outputs, success criteria, and failure fallbacks. A scenario that cannot be written down cannot be verified in a pilot either.
  • Architecture auditability. Ask for one architecture diagram showing reads and writes in separate lanes: retrieval and answering on one side, writes and execution on the other, with the latter gated by permissions and approval. The four functions of the NIST AI Risk Management Framework — Govern, Map, Measure, Manage — work as a checklist: ask what the vendor does for each.
  • Deliverable checkability. Require written pilot acceptance criteria and an evaluation set; delivery counts only when the criteria are met. Anthropic's guide to building effective agents makes the point directly: start with the simplest solution and add complexity only when needed — a vendor that leads with multi-agent orchestration without showing why simpler options fall short deserves scrutiny.

On integration, ask one concrete question: does the vendor use an open tool-integration protocol (such as the Model Context Protocol), or does every system need bespoke development? Open protocols cut long-term maintenance cost and reduce vendor lock-in.

Where this applies

This selection method fits enterprises with a defined business process, a need to connect an agent to real business systems, and writes that must be controlled — customer service, internal support, operations approvals, and knowledge retrieval are the most common starting points. It does not fit a buyer who only needs a question-answering chatbot, has no internal systems to connect, or expects a turnkey black box — that purchase is a product, not a systems integration service.

Risks

  • Demo-to-production gap. Demos run on clean data and prepared questions; production faces dirty data, abnormal inputs, and concurrent load. Before signing, ask the vendor to run a pilot on your real data sample.
  • Permission creep. An agent with over-broad system permissions can corrupt data at scale from a single bad judgement. Permissions must follow least privilege, and writes must be approvable, reversible, and auditable.
  • Prompt injection and data leakage. External content (emails, web pages, uploaded files) can carry instructions that push an agent beyond its authority; the OWASP GenAI Security Project lists prompt injection among its top risk categories — ask how the vendor defends against it and tests the defence.
  • Compliance risk. In the EU market, an AI system may fall within the EU AI Act's risk tiers and obligations; scenarios involving personal data must also satisfy GDPR. A vendor should be able to say which tier your scenario belongs to and which obligations follow, not leave compliance to the client.
  • Runaway cost. Usage-priced model costs grow linearly with volume; a cheap pilot does not mean cheap production. Ask for a cost model based on your real call volumes, not demo-scale estimates.

Deliverables checklist

Confirm these deliverables are in the contract before signing:

  • Scenario specification: business process, success criteria, failure fallbacks
  • Architecture design: seven layers or an equivalent, with reads/writes in separate lanes
  • Permission and approval matrix: operator, approver, and reversal method for every write
  • Evaluation set and baseline: real business samples, pass thresholds, evaluation method
  • Pilot acceptance criteria: written, measurable, and gating any scale-up
  • Operations and upgrade plan: model upgrades, knowledge updates, incident response times
  • Audit log plan: what is recorded, retention period, and how it is queried

Sources

Next step: if you are evaluating agent vendors, start with Negency enterprise AI agent development, consulting & systems integration — eleven capabilities and a seven-step controlled launch, stated plainly; or take the AI scenario screening to confirm which service line fits your scenario.