Build, buy or get help
Look it up, retrain the AI, or hand it everything: which fits which job

The short answer
For answers from your company's documents, have the AI look things up each time. It can show where every answer came from, respect who may see what, and stay current when a document changes.
Handing it everything at once suits a small, stable set of a few hundred pages. Retraining the AI changes how it writes, not what it knows, and the big AI companies are moving away from it.
Think of preparing a new hire to answer questions
You could give them the filing room and a good index, so they look up the right page each time someone asks. You could send them on a course, so they answer from memory. Or you could hand them the whole binder with every question and let them leaf through it.
Look it up, send them back to school, or hand them the binder.
Retraining teaches style, not facts
The AI companies say this themselves. OpenAI's guidance says to look things up when the model "lacks contextual knowledge" or needs "proprietary information", and to retrain when the tone, the format or the reasoning is wrong (OpenAI, optimizing accuracy).
Research agrees. A Microsoft team found looking things up "consistently outperforms" retraining for adding knowledge, old or new (Ovadia and others, 2023). A Google-led study found models learn new facts slowly through retraining, and the more new facts they learn that way, the more they make things up (Gekhman and others, 2024).
And retraining is becoming hard to buy. OpenAI is winding down its retraining service: it closed to new customers in 2026, and from 6 January 2027 nobody can start a new job (OpenAI deprecations). Anthropic's own service doesn't offer it. Google still does, for its smaller Gemini models (Google Cloud).
Handing it everything works, up to a point
Today's leading models can read about a million tokens at once (a token is roughly three-quarters of a word). That's a few thousand pages. Anthropic's advice: if your material is under about 500 pages, "you can just include the entire knowledge base" with the question (Anthropic, 2024).
The catch is that more isn't always better. Anthropic's own documentation says that as the input grows, "accuracy and recall degrade" (Anthropic). Researchers have found the same in every model they tested.
When the answer sat in the middle of a long input, the AI did worse than with no documents at all
It also costs more per question, because the AI reads everything every time. At list prices, 500 pages cost about 40 cents a question, or about 2 cents once the vendor's discount for repeated text kicks in. Looking things up sends only the few passages needed, at around a cent. What it really costs has the detail.
Which fits which job
Look it up each time
The system finds the few passages that answer the question and hands them to the AI. Also called RAG.
Best for: large or changing document collections, answers that must show their source, and documents only some people may see.
Gives up: simplicity. Someone has to build and maintain the search, and if it misses the right passage, the answer misses too.
Hand it everything
Send the whole document set with every question.
Best for: a small, stable set everyone may see, such as one policy manual or one long contract.
Gives up: accuracy and cost as the set grows. Past a few hundred pages, answers get worse and every question gets dearer.
Retrain the AI
Train a copy of the model on hundreds of examples of the answers you want.
Best for: a fixed format, tone or narrow task, such as sorting emails or writing in a house style. Often combined with looking things up.
Gives up: sources, permissions and easy updates. It can't say where a fact came from, anyone who can use it may get out what it learned, and every change means training again.
What goes wrong
- Retraining to teach it your facts. It learns the style of your documents, then makes up the details.
- Retraining on confidential documents. There's no keycard inside a model. Anyone who can ask it can try to get them out.
- Needing to forget something. Removing one fact from a retrained model is an unsolved research problem (Maini and others, 2024). With the other two, you delete the document.
- Handing it the whole drive. It fits, so people try it. Answers slip, and the bill grows with every question.
- A retrained model that expires. When the AI company retires the model underneath, your retrained copy goes with it.
Questions to ask your team or vendor
- How much material are we talking about: tens of pages, hundreds, or thousands? And how often does it change?
- Does every answer show the passage it came from?
- How does it stop people getting answers from documents they may not see?
- When a policy changes, how soon do the answers change?
- If you propose retraining, what is it for: facts, or style and format? And what happens when the model underneath is retired?
Other names you may hear
- Looking things up before answering, RAG (retrieval-augmented generation)
- The system finds the relevant passages in your documents and hands them to the AI with the question.
- Retraining, fine-tuning
- Training a copy of the model further on your own examples, so it changes how it responds.
- Handing it everything, long context
- Sending the whole document set with every question.
- Context window
- How much the AI can read at once, measured in tokens.
- Tokens
- The units AI reads and is billed by, roughly three-quarters of a word each.
- Prompt caching
- A vendor discount when the same long text is sent again and again.
- Context rot, lost in the middle
- The drop in accuracy as the AI is given more to read.
- Unlearning
- Trying to remove something a model has learned. It doesn't work reliably yet.