RAG architecture: components, patterns and sizing
A production RAG application is a pipeline with six stages. Teams that skip stages usually skip the same two, reranking and evaluation, and spend the following quarter wondering why accuracy plateaued.
The reference pipeline
1. Ingestion. Pull documents from their sources and normalise them into text. Sounds trivial; it isn’t. PDFs with multi-column layouts, scanned contracts, tables inside slides and Confluence exports all degrade differently. Budget real engineering time here. It silently caps the quality of everything downstream.
2. Chunking. Split text into retrievable units. Fixed-size chunks with overlap are the default and are frequently wrong. Structure-aware splitting by section heading, by clause or by table preserves meaning that fixed windows destroy. Typical starting point is 300–800 tokens with 10–15% overlap, then tuned against your evaluation set.
3. Indexing. Embed each chunk and store it, alongside a keyword index. Also store metadata you’ll want to filter on: source, date, access level, document type. Retrofitting metadata later means reindexing the whole corpus.
4. Retrieval. Query both indexes and merge. Hybrid retrieval, dense vectors for meaning plus BM25 or equivalent for exact terms, outperforms either alone on almost every real corpus, because real users search for both concepts and identifiers.
5. Reranking. Take the top 20–50 candidates and reorder them with a cross-encoder that scores each against the query directly. This is the highest-return component most teams omit. It costs milliseconds and routinely moves relevant material from position 14 into the top 3.
6. Generation. Assemble the prompt, cap the context, instruct the model to answer only from supplied material and to say when the material is insufficient. Return citations by chunk ID.
Sizing decisions that matter
How many chunks to retrieve. More context is not better. Past roughly 10–15 chunks, models begin to lose material in the middle of long prompts, and your cost per query climbs linearly. Retrieve broadly, rerank hard, pass few.
Which embedding model. The difference between a mid-tier and a top-tier embedding model is usually smaller than the difference between good and bad chunking. Choose one, hold it constant, and tune everything else before you revisit it.
Where the vector store lives. Below a few million chunks, a Postgres extension handles it and saves you a service to operate. Above that, a dedicated vector database earns its keep. Choosing the dedicated option at 50,000 chunks is a common and expensive reflex.
Access control, the part that gets retrofitted
If different users may see different documents, filtering has to happen inside retrieval, not after generation. A model that has already read a restricted chunk has leaked it, regardless of what you strip from the response. Attach access metadata at indexing time and filter at query time.
This is the single most common architectural rework we’re called in to do, and it’s the one that costs most to fix late.
Evaluation is a component, not a phase
Build a set of 50–200 real questions with known-correct source documents. Measure retrieval and generation separately: did the right chunk come back, and did the model use it correctly. Without that split you cannot tell whether a bad answer is a search problem or a model problem, and you’ll tune the wrong half of the system.
Go deeper
- What is RAG? — the concept, and when to use it instead of fine-tuning
- AI agents — architectures where retrieval is one tool among several
- Data engineering services — the ingestion and pipeline layer underneath
- AI development services — build it with senior engineers
Reviewing an architecture, or starting one?
Bring your corpus size, user base and accuracy requirement to a free 30-minute call. You get a component-by-component read, the sizing decisions we’d make, and an honest cost estimate at your query volume.
staffai.eu · Senior AI and data engineers from Eastern Europe, on T&M