Can we trust the answers?

Updated November 2026 · About 5 minutes to read

The short answer

Mostly, but never completely. Even well-built systems are sometimes wrong, and they sound just as sure when they are.

So decide in advance how you'll measure it, show the source with every answer, and have a person check wherever a mistake would be costly.

Think of a bright new hire on an open-book exam

They have your binders in front of them. Most of the time they find the right page and answer well. Sometimes they read the wrong page, or skim, or fill a gap with a confident guess.

A wrong answer sounds exactly as confident as a right one.

You wouldn't stop trusting a new hire for that. You'd ask them to show where they found it, and you'd check their work on anything that matters.

Where the picture stops working: a new hire learns from each correction. The AI doesn't learn from being corrected in a conversation; the fix is better documents, better search or better instructions.

Looking things up cuts mistakes, but doesn't end them

Researchers at Stanford and Yale tested AI on specific legal questions, where every answer can be checked. These are the shares of answers that were made up or wrong.

58–88%

General chatbots, answering from memory
ChatGPT 4 and other models, asked about real court cases.
17–33%

Legal research tools that look things up first
Products from LexisNexis and Thomson Reuters, built to answer from legal sources.

Sources: Dahl and others, Large Legal Fictions (2024), and Magesh and others, Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (2024), both from Stanford and Yale researchers. Two separate studies with different questions, so read the bars as a contrast, not a precise comparison. Newer models do better, but no study has found zero.

The vendors of those legal tools had described them as avoiding or eliminating made-up answers. The study found otherwise. Your system will have its own error rate, and the only way to know it is to measure it on your own questions.

When a person must check

Match the checking to the cost of a mistake, not to how good the system seems.

Low stakes
Where's the holiday calendar? How do I book a meeting room? A wrong answer costs a minute.
Show the source
Medium stakes
What does our travel policy allow? An employee may act on it, but it can be put right.
Check a sample each week
High stakes
Anything legal, medical, financial, or promised to a customer.
A person checks every answer

If you can't afford the mistake, don't let the AI have the last word.

Three ways to know how good it is

A test set of real questions

Collect real questions with answers your experts agree on, and score the system against them before launch and after every change. Fifty is a sensible start; that number is our rule of thumb, not a standard.

Best for: everyone. It turns "it seems fine" into a number you can track.

Gives up: a few days of an expert's time up front, and it only covers the questions you thought of.

Experts review a sample of real answers

Each week, someone who knows the subject marks a handful of answers people actually got.

Best for: catching the questions nobody predicted, once it's live.

Gives up: it's ongoing work, and it finds problems after someone has seen them.

Feedback from the people using it

A thumbs-up or thumbs-down on each answer, and a way to say what was wrong.

Best for: spotting trouble early, at almost no cost.

Gives up: accuracy. People can't flag a wrong answer they believed, and most never click.

What goes wrong

Judging it by the demo. The vendor's ten questions work. Yours haven't been tried.
Answers with no source. People can't check what they can't see, so they stop trusting all of it, or none of it.
Measuring once. Documents change and the AI vendor updates its model. Last quarter's score says little about today.
Blaming the AI for missing documents. Often the right answer was never written down, or an old version won. Is our information ready?

Questions to ask your team or vendor

How often is it wrong on our own questions, and how did you measure that?
Does every answer show the passage it came from, so a person can check?
What does it do when the answer isn't in our documents: say so, or guess?
Which answers will a person check before they're used, and who?
When the documents or the AI model change, how will we know the answers are still right?

Other names you may hear

Hallucination: A made-up answer that sounds confident. NIST, the US standards agency, calls it confabulation (NIST AI 600-1).
Grounding: Answering only from the passages the system found, like being asked to show your work.
Citation, attribution: A link from the answer to the exact passage it came from, like a footnote.
Evaluation, evals, test set: Checking answers against an agreed answer key, like marking a test.
Confidence score: A number some systems give for how sure they are. A useful hint, not a guarantee.
Human in the loop: A person reviews or approves the AI's answer before it's used.