Build, buy or get help
Questions to ask in an AI vendor pitch

The short answer
Ask the vendor to answer your own questions, from your own documents, before you sign. Accuracy figures, certificates and the word "agent" can't be checked any other way.
Then ask three more things: what happens to your data, what it will cost when everyone uses it, and how you'd leave.
Think of hiring a contractor to refit your offices
The portfolio is full of beautiful buildings. You'd still ask for references, check their license, get a written quote that includes the extras, and maybe give them a small job first.
Judge the work on your building, not the portfolio.
The claims are often bigger than the product
In 2024, Stanford researchers tested legal research tools whose makers promised "hallucination-free" citations or said they "avoid" made-up answers. On more than 200 legal questions, they still gave wrong answers at least one time in six.
Tools marketed as free of made-up answers still got one in six or more wrong
Regulators have started acting on overclaiming. In 2024 the US Securities and Exchange Commission fined two investment firms $400,000 between them for AI they didn't have (SEC). The Federal Trade Commission has brought a string of cases since, including a 2026 settlement with three marketing firms that sold an "AI" service said to listen to conversations to target ads. It was really reselling email lists bought from data brokers (FTC, May 2026).
"Agent" is the current label to be careful with. In 2025 the research firm Gartner estimated that only about 130 of the thousands of vendors selling AI agents offered the real thing; the rest had relabeled chatbots and older automation (Reuters, June 2025).
The questions, and the answers that should worry you
- Will you answer 50 of our questions, from our documents, before we sign?Worry if the answer is a canned demo, or a trial on their sample documents.
- How often is it wrong, measured how, and on whose documents?Worry at a single accuracy figure with no test behind it, or a promise of no mistakes at all.
- Does every answer show the passage it came from?Worry if you can't click through to the source to check it.
- How does it stop people seeing answers from documents they may not open?Worry at "we'll set that up later" or "everyone sees the same thing".
- Do you, or the AI company behind you, learn from or keep our data?Worry if they can't show you where the contract says so. Where your data goes has the detail.
- What will it cost a year when everyone has it, and what can change at renewal?Worry at prices per person for everyone from day one, or usage charges you can't estimate.
- If we leave, what do we take with us, and in what form?Worry if your cleaned-up documents, test questions and settings only exist inside their product.
What a certificate proves, and what it doesn't
SOC 2 Type II
An auditor checked that the vendor's security controls worked over six to twelve months. Ask whether the AI features were included.
Not whether answers are rightISO/IEC 42001
An auditor checked the vendor runs a proper process for managing AI risks. Voluntary, and new since 2023.
Not whether answers are right"Aligned with the NIST AI framework"
A voluntary US government framework for managing AI risk. There's no certificate; vendors declare it themselves.
A claim, not an audit
Three ways to check what you're told
Watch the demo
The vendor shows the product answering questions they chose, from documents they chose.
Best for: seeing how it looks and feels, and shortlisting.
Gives up: any evidence about your documents. A demo shows the best case.
A trial on your documents
The vendor loads a slice of your real documents, and your experts mark its answers to questions they wrote.
Best for: the decision itself. It's the only test that tells you how it will do for you.
Gives up: a few weeks, some of your experts' time, and sometimes a fee.
An independent check
Your own team or an adviser runs the trial, keeps the test questions private, and compares vendors side by side.
Best for: large contracts, regulated work, and choosing between two or three finalists.
Gives up: time and cost up front, and some vendors won't take part.
What goes wrong
- Buying on the demo. It answered beautifully from their documents. Yours are older, messier and spread over five systems.
- Letting the vendor mark its own test. Write the questions yourselves, and keep some back.
- Reading a security certificate as a quality guarantee. It says the doors lock, not that the answers are right.
- A price that grows at renewal. Microsoft, for example, raised its business Microsoft 365 prices from July 2026, citing the AI it had added (Microsoft, December 2025).
- No way out. Leaving means rebuilding everything you set up inside their product. Build or buy covers lock-in.
Questions to ask your own team before the pitch
- What exactly do we want it to answer, and for whom?
- Which 50 real questions, with answers our experts agree on, will we test every vendor with?
- Which documents can we safely give a vendor for a trial?
- Who in the business will own it after launch, and will they be in the room?
- What would make us walk away?
Other names you may hear
- Proof of concept, POC, pilot
- A small trial before the full purchase. Worth most when it uses your documents.
- Made-up answer, hallucination
- A confident answer that isn't true or isn't in the documents.
- Test questions, evaluation, evals
- A set of questions with agreed answers, used to mark how often it's right.
- AI washing, agent washing
- Calling a product AI, or an agent, when it does less than the label suggests.
- SOC 2 Type II
- An independent audit that a company's security controls worked over a period.
- ISO/IEC 42001
- An international standard for how a company manages AI risks.
- NIST AI RMF
- The US National Institute of Standards and Technology's voluntary AI Risk Management Framework.
- RFP, request for proposal
- A formal written request asking vendors to bid.