EnterpriseLLM.ai

Home › Build, buy or get help

Build, buy or get help

Questions to ask in an AI vendor pitch

Updated November 2026 · About 6 minutes to read

The short answer

Ask the vendor to answer your own questions, from your own documents, before you sign. Accuracy figures, certificates and the word "agent" can't be checked any other way.

Then ask three more things: what happens to your data, what it will cost when everyone uses it, and how you'd leave.

Think of hiring a contractor to refit your offices

The portfolio is full of beautiful buildings. You'd still ask for references, check their license, get a written quote that includes the extras, and maybe give them a small job first.

Judge the work on your building, not the portfolio.

Where the picture stops working: a contractor's finished work stands still to be inspected. An AI product changes month to month, and the demo was built on documents chosen to make it shine.

The claims are often bigger than the product

In 2024, Stanford researchers tested legal research tools whose makers promised "hallucination-free" citations or said they "avoid" made-up answers. On more than 200 legal questions, they still gave wrong answers at least one time in six.

Tools marketed as free of made-up answers still got one in six or more wrong

  • Lexis+ AI17%+
  • Ask Practical Law AI17%+
  • Westlaw AI-Assisted Research34%+
Source: Magesh and others, Stanford RegLab and HAI, May 2024 (the paper). These were the 2024 versions; the products have changed since.

Regulators have started acting on overclaiming. In 2024 the US Securities and Exchange Commission fined two investment firms $400,000 between them for AI they didn't have (SEC). The Federal Trade Commission has brought a string of cases since, including a 2026 settlement with three marketing firms that sold an "AI" service said to listen to conversations to target ads. It was really reselling email lists bought from data brokers (FTC, May 2026).

"Agent" is the current label to be careful with. In 2025 the research firm Gartner estimated that only about 130 of the thousands of vendors selling AI agents offered the real thing; the rest had relabeled chatbots and older automation (Reuters, June 2025).

The questions, and the answers that should worry you

  1. Will you answer 50 of our questions, from our documents, before we sign?Worry if the answer is a canned demo, or a trial on their sample documents.
  2. How often is it wrong, measured how, and on whose documents?Worry at a single accuracy figure with no test behind it, or a promise of no mistakes at all.
  3. Does every answer show the passage it came from?Worry if you can't click through to the source to check it.
  4. How does it stop people seeing answers from documents they may not open?Worry at "we'll set that up later" or "everyone sees the same thing".
  5. Do you, or the AI company behind you, learn from or keep our data?Worry if they can't show you where the contract says so. Where your data goes has the detail.
  6. What will it cost a year when everyone has it, and what can change at renewal?Worry at prices per person for everyone from day one, or usage charges you can't estimate.
  7. If we leave, what do we take with us, and in what form?Worry if your cleaned-up documents, test questions and settings only exist inside their product.

The US government gives its own buyers similar advice: test before awarding a contract, in conditions like real use, with test data the vendor can't see, and protect against lock-in (Office of Management and Budget memo M-25-22, April 2025). The UK government's AI Playbook says much the same.

What a certificate proves, and what it doesn't

  • SOC 2 Type II

    An auditor checked that the vendor's security controls worked over six to twelve months. Ask whether the AI features were included.

    Not whether answers are right
  • ISO/IEC 42001

    An auditor checked the vendor runs a proper process for managing AI risks. Voluntary, and new since 2023.

    Not whether answers are right
  • "Aligned with the NIST AI framework"

    A voluntary US government framework for managing AI risk. There's no certificate; vendors declare it themselves.

    A claim, not an audit

Sources: NIST AI Risk Management Framework; ISO/IEC 42001; AICPA, SOC 2. Certificates are worth asking for; they just answer a different question from "will it answer ours correctly?"

Three ways to check what you're told

Watch the demo

The vendor shows the product answering questions they chose, from documents they chose.

Best for: seeing how it looks and feels, and shortlisting.

Gives up: any evidence about your documents. A demo shows the best case.

A trial on your documents

The vendor loads a slice of your real documents, and your experts mark its answers to questions they wrote.

Best for: the decision itself. It's the only test that tells you how it will do for you.

Gives up: a few weeks, some of your experts' time, and sometimes a fee.

An independent check

Your own team or an adviser runs the trial, keeps the test questions private, and compares vendors side by side.

Best for: large contracts, regulated work, and choosing between two or three finalists.

Gives up: time and cost up front, and some vendors won't take part.

What goes wrong

  • Buying on the demo. It answered beautifully from their documents. Yours are older, messier and spread over five systems.
  • Letting the vendor mark its own test. Write the questions yourselves, and keep some back.
  • Reading a security certificate as a quality guarantee. It says the doors lock, not that the answers are right.
  • A price that grows at renewal. Microsoft, for example, raised its business Microsoft 365 prices from July 2026, citing the AI it had added (Microsoft, December 2025).
  • No way out. Leaving means rebuilding everything you set up inside their product. Build or buy covers lock-in.

Questions to ask your own team before the pitch

  1. What exactly do we want it to answer, and for whom?
  2. Which 50 real questions, with answers our experts agree on, will we test every vendor with?
  3. Which documents can we safely give a vendor for a trial?
  4. Who in the business will own it after launch, and will they be in the room?
  5. What would make us walk away?

Other names you may hear

Proof of concept, POC, pilot
A small trial before the full purchase. Worth most when it uses your documents.
Made-up answer, hallucination
A confident answer that isn't true or isn't in the documents.
Test questions, evaluation, evals
A set of questions with agreed answers, used to mark how often it's right.
AI washing, agent washing
Calling a product AI, or an agent, when it does less than the label suggests.
SOC 2 Type II
An independent audit that a company's security controls worked over a period.
ISO/IEC 42001
An international standard for how a company manages AI risks.
NIST AI RMF
The US National Institute of Standards and Technology's voluntary AI Risk Management Framework.
RFP, request for proposal
A formal written request asking vendors to bid.

Photo by Pavel Danilyuk on Pexels