What is RAG? Retrieval-augmented generation explained
Retrieval-augmented generation (RAG) is a technique where a language model answers a question using documents fetched at query time rather than only what it learned during training. You retrieve the relevant material first, put it in the prompt, and let the model write the answer from it.
The point is not cleverness. It’s that a model with no access to your data will confidently invent an answer, and a model handed the right paragraph will usually get it right.
How RAG works
Four steps, in order:
- Index. Split your documents into chunks and store them, usually as vector embeddings that capture meaning rather than exact words.
- Retrieve. Convert the user’s question the same way and pull back the chunks most similar to it.
- Augment. Insert those chunks into the prompt alongside the question and an instruction to answer only from the supplied material.
- Generate. The model writes the answer. A well-built system cites which chunk each claim came from.
The interesting engineering sits almost entirely in step 2. Step 4 is a solved problem you buy from an API.
RAG or fine-tuning?
They solve different problems, and the choice is more decided than most vendors admit.
Use RAG when the answer depends on facts that change, that are private to your organisation, or that must be cited. Product documentation, policy manuals, contracts, support histories, anything with a last-updated date.
Use fine-tuning when you need to change behaviour: output format, tone, a domain vocabulary the base model handles badly, or a classification task with a fixed label set.
They compose. A fine-tuned model that follows your house citation format, answering from documents retrieved at query time, is a common production shape. If someone proposes fine-tuning to teach a model your company’s facts, ask what happens when those facts change next quarter.
The four ways RAG fails in production
Retrieval misses. The correct document exists, the search doesn’t find it. This is the most common failure by a wide margin, and it’s invisible unless you measure retrieval separately from generation. Most teams only look at final answers and conclude the model is bad.
Chunking destroys context. A chunk boundary lands mid-table or mid-clause, so the retrieved fragment is technically relevant and practically useless. Symptoms look like model hallucination but the cause is upstream.
Semantic search alone is not enough. Vector similarity handles paraphrase well and exact identifiers badly. A user searching for error code E-4471 or a specific SKU often gets nothing useful. Hybrid retrieval, keyword search combined with vectors, fixes this and is skipped surprisingly often.
No evaluation harness. Without a fixed set of questions and expected answers, every change to chunk size, embedding model or prompt is a guess. Teams end up tuning by vibe and cannot tell whether last week’s change helped.
When not to build RAG
If your entire corpus fits in a modern context window and doesn’t change often, put it in the prompt and skip the infrastructure. If your users ask ten predictable questions, write ten answers. If the underlying documents contradict each other, RAG will surface the contradiction rather than resolve it. Fix the documents first.
Go deeper
- RAG architecture — components, patterns and how to size them
- AI agents — when retrieval becomes one tool among several
- Prompt engineering — getting reliable output from the generation step
- AI development services — build a RAG system with senior engineers
Building RAG on your own documents?
Bring your corpus, your users’ actual questions, and your accuracy bar to a free 30-minute call. You get an engineer’s read on whether RAG is the right approach, what the retrieval layer needs to look like, and what it costs to run at your volume.
staffai.eu · Senior AI and data engineers from Eastern Europe, on T&M