EnterpriseLLM.ai

Home › Build, buy or get help

Build, buy or get help

Look it up, retrain the AI, or hand it everything: which fits which job

Updated November 2026 · About 6 minutes to read

The short answer

For answers from your company's documents, have the AI look things up each time. It can show where every answer came from, respect who may see what, and stay current when a document changes.

Handing it everything at once suits a small, stable set of a few hundred pages. Retraining the AI changes how it writes, not what it knows, and the big AI companies are moving away from it.

Think of preparing a new hire to answer questions

You could give them the filing room and a good index, so they look up the right page each time someone asks. You could send them on a course, so they answer from memory. Or you could hand them the whole binder with every question and let them leaf through it.

Look it up, send them back to school, or hand them the binder.

Where the picture stops working: a person on a course remembers the facts. An AI model that's retrained mostly picks up how to write and what format to use, and it is poor at remembering new facts.

Retraining teaches style, not facts

The AI companies say this themselves. OpenAI's guidance says to look things up when the model "lacks contextual knowledge" or needs "proprietary information", and to retrain when the tone, the format or the reasoning is wrong (OpenAI, optimizing accuracy).

Research agrees. A Microsoft team found looking things up "consistently outperforms" retraining for adding knowledge, old or new (Ovadia and others, 2023). A Google-led study found models learn new facts slowly through retraining, and the more new facts they learn that way, the more they make things up (Gekhman and others, 2024).

And retraining is becoming hard to buy. OpenAI is winding down its retraining service: it closed to new customers in 2026, and from 6 January 2027 nobody can start a new job (OpenAI deprecations). Anthropic's own service doesn't offer it. Google still does, for its smaller Gemini models (Google Cloud).

Handing it everything works, up to a point

Today's leading models can read about a million tokens at once (a token is roughly three-quarters of a word). That's a few thousand pages. Anthropic's advice: if your material is under about 500 pages, "you can just include the entire knowledge base" with the question (Anthropic, 2024).

The catch is that more isn't always better. Anthropic's own documentation says that as the input grows, "accuracy and recall degrade" (Anthropic). Researchers have found the same in every model they tested.

When the answer sat in the middle of a long input, the AI did worse than with no documents at all

  • Right document first76%
  • Right document in the middle54%
  • Right document last63%
  • No documents at all56%
Source: Liu and others, Lost in the Middle (Stanford and others, 2023), GPT-3.5 Turbo answering from 20 documents. Newer models do better, but a 2025 test of 18 models still found accuracy fell as inputs grew (Chroma, Context Rot).

It also costs more per question, because the AI reads everything every time. At list prices, 500 pages cost about 40 cents a question, or about 2 cents once the vendor's discount for repeated text kicks in. Looking things up sends only the few passages needed, at around a cent. What it really costs has the detail.

Our arithmetic: about 200,000 tokens for 500 pages (Anthropic's own estimate), at Claude Sonnet 5.5's list price of $2 per million tokens read, or 10 cents per million for repeated text (Anthropic pricing, November 2026). OpenAI's comparable model is priced the same.

Which fits which job

Look it up each time

The system finds the few passages that answer the question and hands them to the AI. Also called RAG.

Best for: large or changing document collections, answers that must show their source, and documents only some people may see.

Gives up: simplicity. Someone has to build and maintain the search, and if it misses the right passage, the answer misses too.

Hand it everything

Send the whole document set with every question.

Best for: a small, stable set everyone may see, such as one policy manual or one long contract.

Gives up: accuracy and cost as the set grows. Past a few hundred pages, answers get worse and every question gets dearer.

Retrain the AI

Train a copy of the model on hundreds of examples of the answers you want.

Best for: a fixed format, tone or narrow task, such as sorting emails or writing in a house style. Often combined with looking things up.

Gives up: sources, permissions and easy updates. It can't say where a fact came from, anyone who can use it may get out what it learned, and every change means training again.

What goes wrong

  • Retraining to teach it your facts. It learns the style of your documents, then makes up the details.
  • Retraining on confidential documents. There's no keycard inside a model. Anyone who can ask it can try to get them out.
  • Needing to forget something. Removing one fact from a retrained model is an unsolved research problem (Maini and others, 2024). With the other two, you delete the document.
  • Handing it the whole drive. It fits, so people try it. Answers slip, and the bill grows with every question.
  • A retrained model that expires. When the AI company retires the model underneath, your retrained copy goes with it.

Questions to ask your team or vendor

  1. How much material are we talking about: tens of pages, hundreds, or thousands? And how often does it change?
  2. Does every answer show the passage it came from?
  3. How does it stop people getting answers from documents they may not see?
  4. When a policy changes, how soon do the answers change?
  5. If you propose retraining, what is it for: facts, or style and format? And what happens when the model underneath is retired?

Other names you may hear

Looking things up before answering, RAG (retrieval-augmented generation)
The system finds the relevant passages in your documents and hands them to the AI with the question.
Retraining, fine-tuning
Training a copy of the model further on your own examples, so it changes how it responds.
Handing it everything, long context
Sending the whole document set with every question.
Context window
How much the AI can read at once, measured in tokens.
Tokens
The units AI reads and is billed by, roughly three-quarters of a word each.
Prompt caching
A vendor discount when the same long text is sent again and again.
Context rot, lost in the middle
The drop in accuracy as the AI is given more to read.
Unlearning
Trying to remove something a model has learned. It doesn't work reliably yet.

Photo by Mikhail Nilov on Pexels