Every vendor selling AI agents has access to the same handful of frontier models. That means the model is the least differentiated thing about any of them, and the demo, which mostly shows off the model, is the least useful part of the evaluation. What you are actually buying is everything wrapped around the model: how the agent is provisioned for your work, who supervises it, what happens when it is wrong, and what the paperwork says when you want out.
We sell managed agents, so this list is self-interested. It is also the list we would use on a competitor. Five questions about the agent and its management, five about the paperwork, and for each one, what a good answer sounds like and the answer that should end the meeting.
The agent and its management
- What does the agent actually know about my business, and how did it get there?
A good answer names an architecture you can inspect: a written role charter, a single source of truth the agent reads before it works, a stated practice for keeping facts current, and a rule for what the agent does when it does not know. The bad answer is "it's powered by [model name]" or "we fine-tuned it." The model name tells you almost nothing; the context layer is the product.
- Who reviews the agent's work before it reaches me or my customers, and what can it never do on its own?
The agent will make mistakes. The question is where the human gate sits and which actions are gated. A good answer names an accountable person, describes draft-mode as the default (the agent drafts, a human approves and sends), and states a hard rule: nothing irreversible, no send, post, filing, or payment, without a human. Ask for a real example of the gate catching something. "It's fully autonomous," pitched as a feature, is your cue to leave.
- Name three ways your agent fails, and the tripwire for each.
Every vendor claims accuracy. The tell is whether they can list their own failure modes unprompted. A vendor who has actually run agents can name stale facts, invented status, quiet model degradation, an overnight job that silently didn't run, and can point to the alarm that catches each one. A benchmark score or a guarantee is not an answer; an operator who publishes their misses is showing you the supervision working.
- What happens when the model behind the agent changes, or is unavailable for a week?
Frontier models ship every few months. Capabilities, prices, and even availability move under the vendor's feet, and premium model tiers can carry different data-retention terms. A good answer includes a per-task model policy, tested fallbacks that get logged rather than passed off as normal output, and unprompted disclosure of retention terms by tier. "We always use the best model" is a cost strategy and a retention surprise, not a continuity plan.
- What exactly am I paying for that a $30-a-month chat subscription doesn't give me?
A subscription gives your team raw capability. A managed agent should give you a provisioned role: the context work from question 1, the supervision gate from question 2, the failure instrumentation from question 3, plus isolation from the vendor's other clients, an audit trail, and access you can revoke. If the vendor cannot itemize that difference crisply, buy the subscription.
The paperwork and the exit
- Where does our material live, who can see it, and which single document governs confidentiality?
The real confidentiality risk is not hacking; it is sprawl: your material drifting into a shared drive, another client's context, or a training set. A good answer names a segregated per-client workspace, lists subprocessors, states a retention term with delete-on-request, commits in writing to no training on your data, and points to one controlling document. Two overlapping NDAs that don't reference each other are as bad as none.
- When the agent gets something wrong, who owns the error and what is my remedy?
This tests whether anyone drafted for the failure case. A good answer: a named human accountable for the output, a liability term tied to real fees and disclosed before you pay, and a stated path where a person, not the agent, reviews the error. A liability cap that exists is oddly reassuring; it means a lawyer imagined the bad day. "Our AI is very accurate" means nobody did.
- What can the agent do on its own, and what always requires a human? Show me that in writing.
Verbal assurances about a human in the loop do not survive staff turnover or a busy quarter, and authority creeps: the agent that could only draft last month starts sending this month. The good answer is a written decide-propose-escalate matrix attached to the engagement, in which every irreversible act is human-gated with no exceptions. A vendor who can't produce the matrix doesn't have one.
- What are the exit terms: cancellation, refunds, and our data at offboarding?
An engagement you cannot leave cleanly is a liability you already signed. Look for pricing that is public before anyone contacts you, month-to-month terms with cancellation by email, no auto-renewal traps, the refund policy visible at checkout rather than after payment, and a stated schedule for returning and deleting your material when you go. If the vendor cannot state those terms plainly, walk away.
- Where the work touches a licensed profession, where exactly is your line, and has a licensed professional reviewed it?
Unauthorized practice exposure in law, tax, or accounting does not rest with the vendor alone; the buyer relying on unlicensed advice bears part of it. Vendors near this line either know precisely where it sits or have not thought about it. There is no middle. A good answer is a written scope that stops at prepare, organize, and flag, with every judgment call escalated to a named human or your own advisors, and evidence that counsel reviewed the boundary rather than the vendor merely asserting it.
What to do with the answers
You do not need ten perfect answers. You need a vendor who has clearly heard these questions before, answers most of them in writing, and tells you plainly which ones they are still building toward. We can answer most of this list cleanly today; a couple of them, including the model-continuity instrumentation in question 4, are still being finished, and we will say so when you ask. That is the standard we think you should hold anyone to, including us.
None of this is legal advice, but it IS procurement advice. Run the engagement terms past your own counsel.