Retrieval is the default. Fine-tuning earns its place when the problem is how the model behaves rather than what it knows, and most teams reach for it about a quarter too early.
Both are sold to boards as “teaching the model our business”, which is why the decision gets made badly. They do different jobs. Retrieval-augmented generation puts your documents in front of the model at request time, so answers track whatever sits in the index today. Fine-tuning changes the weights, so the model learns form: output shape, register, domain vocabulary, a narrow classification call. A fine-tuned model that has never seen this month’s price list will still get this month’s prices wrong, fluently and without warning. Our explainer on how retrieval-augmented generation works covers the mechanics; this piece is about which one to fund.
What each costs to build and run
| Retrieval | Fine-tuning | |
|---|---|---|
| Time to a credible internal demo | 1 to 3 weeks | 3 to 8 weeks, mostly dataset work |
| Dominant effort | ingestion, chunking, permissions, ranking | labelled examples, training loop, serving |
| Content changes | reindex in minutes or hours | retrain in days |
| Marginal cost per request | higher: context inflates tokens 2x to 10x | lower: a smaller model can do the job |
| Typical failure | retrieves the wrong passage | answers confidently from stale weights |
The GPU bill is not the interesting number. A LoRA run over a few thousand examples on an 8B open-weight model costs tens of euros, sometimes low hundreds. The dataset construction, the held-out evaluation set and the regression suite around it absorb 60% to 80% of the budget, and those are engineer-days at the rates shown on our daily rates page, not compute.
When fine-tuning actually pays
- Unit economics. If you serve hundreds of thousands of requests a month, distilling a large-model behaviour into an 8B model can cut cost per request by 5x to 20x and take a second off p95 latency. Below roughly 50,000 requests a month the saving does not repay the build.
- Strict output conformance. Prompting gets structured extraction to about 90% schema compliance; fine-tuning is how you get the last nine points when a downstream system breaks on malformed output.
- Narrow classification or extraction where you already hold thousands of labelled examples from a legacy rules engine or a human review queue.
- Register you cannot describe in a prompt: regulated disclosure language, clinical phrasing, a house style with twenty years of precedent behind it.
When retrieval wins
- The corpus changes weekly or faster. Anything price, policy or inventory related.
- You must cite the source. Auditors and regulators want the paragraph, not the model’s word.
- Answers must respect per-user permissions. You can filter an index by document ACL. You cannot permission a weight.
- Breadth: tens of thousands of documents, two or three of which matter for any given question.
Build retrieval first even if you expect to fine-tune
The retrieval build produces the cleaned corpus and the evaluation set that any fine-tune depends on. The reverse is not true. Once you have an evaluation suite, you can separate knowledge failures from behaviour failures, and the fine-tuning question mostly answers itself. Teams that fine-tune first usually spend six weeks discovering that their real defect was a chunking strategy that split tables down the middle.
The choice changes who you hire
Retrieval work is data engineering with a language model on the end: ingestion, deduplication, permission propagation, ranking evaluation. Fine-tuning needs someone who has run a training job and then served a quantised model under load, which is a different CV. Hiring one and asking for the other is a common way to lose a quarter. We staff both sides, either with data engineers for the retrieval and pipeline work or ML engineers for training and serving.
Where renting engineers is the wrong call
If you cannot yet name the failure you are trying to fix, a rented team will bill you for discovery you could do internally in two weeks with one person and a notebook. If the honest answer is “better prompts and a better index”, you do not need a team at all, you need a fortnight. And if your board wants a fixed price with an outcome guarantee, time-and-materials staffing is the wrong instrument; the drivers behind AI development cost explain why fixed price gets quoted 30% to 50% higher on work this uncertain.
The decision rule is short. If the answer changes when your documents change, retrieve. If the answer changes when your standards change, fine-tune. If both, build retrieval now and revisit fine-tuning once the evaluation set exists.