Short answer: use retrieval (RAG) when the model needs information it does not have, such as your documents, records or anything that changes. Use fine-tuning when the model has the information but will not behave consistently: a fixed format, a house style, a narrow classification at high volume. If the knowledge is small and stable, skip both and put it in the prompt. Try them in order of cost: prompting, then long-context prompting, then retrieval, then fine-tuning, measuring each step against the same set of test examples.
Our guide to integrating an LLM into a product has a one-table overview of the three approaches. This article goes further into the decision itself.
What each approach actually changes
A language model produces its answer from two things: its weights, fixed during training, and the context, which is everything in the prompt for this request. Every technique changes one of the two.
| Changes | Mechanism | What the model gains | |
|---|---|---|---|
| Prompting | Context | Instructions and a few examples written into the prompt | A description of the task |
| Long-context prompting | Context | The full reference material placed in the prompt on every request | All of a small body of knowledge, every time |
| Retrieval (RAG) | Context | A search step selects relevant passages from your data and inserts them into the prompt | The relevant slice of a large body of knowledge, current as of the last index update |
| Fine-tuning | Weights | Further training on hundreds or thousands of example inputs with the desired outputs | A changed default behaviour: format, tone, task skill |
The distinction that settles most cases: knowledge belongs in the context, behaviour can be moved into the weights. Facts placed in the context can be updated, cited, filtered per user and removed. Facts pushed into weights can do none of those, and they are recalled unreliably, because training on a document teaches a model to write text that resembles the document more than it teaches the model to look up what the document says.
A useful analogy is a new employee. Fine-tuning is training: it shapes how they write and what they do by habit. Retrieval is giving them access to the filing system. Nobody tries to replace the filing system by having staff memorise it, and nobody expects the filing system to teach tone of voice.
Where fine-tuning happens in practice
Two routes exist. Several hosted model vendors offer fine-tuning of some of their models as a service: you upload examples, they run the training and host the result, usually at a different per-token price. Alternatively, an open-weight model is fine-tuned, often with a parameter-efficient method that trains a small set of additional weights, and then hosted by you or by an inference provider. Both routes produce a model that is tied to one base model version, which matters for maintenance, as discussed below.
Cost and maintenance profile
| RAG | Fine-tuning | |
|---|---|---|
| Up-front work | Document parsing, chunking, indexing, search quality, permission filtering, citations. Mostly software engineering | Collecting and cleaning examples, training runs, comparison against a prompted baseline. Mostly data work |
| Who does most of it | Engineers, with a domain expert checking answers | Domain experts producing and reviewing examples, plus someone who understands training and evaluation |
| Cost per request | Higher input cost: retrieved passages are sent on every call. A search step is added | Can be lower: shorter prompts, and possibly a smaller model. The tuned model may be priced above its base version or need dedicated hosting |
| When your data changes | Re-index the changed documents; minutes to hours, automated | Prepare new examples and retrain; days, with a full re-evaluation |
| When the model vendor retires the base model | Switch model, re-run the evaluation set, adjust prompts | Repeat the fine-tuning on a new base model, then re-evaluate. The tuned weights do not transfer |
| Switching vendors | Straightforward: the index and the pipeline are yours | Start again: the tuned model lives with one vendor or one model family |
| Ongoing attention | Index freshness, retrieval quality on new kinds of question, content hygiene | Drift between training examples and real inputs; periodic retraining |
The line founders most often miss is model retirement. Hosted models are withdrawn on published schedules. A prompt-and-retrieval system migrates with a few days of testing. A fine-tuned model has to be rebuilt, which means the example set, the training procedure and the evaluation all need to be kept in working order for as long as the feature exists. Fine-tuning is therefore a standing commitment. Budget figures for building either kind of feature are in our breakdown of AI integration costs.
Data requirements
For retrieval
- The source material itself, in a form that can be parsed. Clean, well-structured documents give good retrieval. Scanned PDFs, slide decks with little text and contradictory versions of the same policy give poor retrieval whatever the technology.
- Metadata: title, section, date, owner, and who may see it. Permission rules must exist as data before they can be enforced in search.
- An evaluation set: 50 to 200 real questions, each with the passage that contains the answer and the answer you would accept.
- An owner for the content. If nobody is responsible for keeping the documents correct, the assistant will quote outdated ones with complete confidence.
For fine-tuning
- Input-output pairs that show exactly the behaviour you want. As a planning figure, a few hundred for a narrow format or classification task, a few thousand for more varied behaviour.
- Consistency. If two reviewers would write different “correct” outputs for the same input, settle the guideline first. The model learns whatever inconsistency is in the examples.
- Coverage of the real distribution of inputs, including the awkward ones, and examples of when to refuse or defer.
- A held-out test set that is never used in training.
- Rights and privacy clearance. Personal data in training examples is much harder to remove from a model than from an index. Deleting a document from a retrieval index takes effect at once; removing its influence from trained weights means retraining.
If you have documents but no examples of ideal outputs, you are equipped for retrieval and not for fine-tuning. If you have thousands of past human decisions or edited drafts, fine-tuning becomes possible, and so does a very good evaluation set, which is worth having in either case.
Latency
| Approach | Effect on response time |
|---|---|
| RAG | Adds a search step before the model call: typically tens to a few hundred milliseconds for embedding the query, searching and reranking. Adds input tokens, which lengthens the time before the first word appears. Both are usually small next to generation time |
| Long-context prompting | Large prompts take noticeably longer to process unless the vendor’s prompt cache is warm, in which case the repeated prefix is processed much faster |
| Fine-tuning | Often reduces latency: instructions and examples no longer need to be in every prompt, and a smaller model may be good enough. With self-hosting, latency depends on the hardware you pay for |
Latency is a legitimate reason to consider fine-tuning for narrow, high-volume, real-time tasks such as classifying each incoming message or reformatting each record. It is rarely a reason to avoid retrieval.
How often the knowledge changes
| Rate of change | Examples | Fits |
|---|---|---|
| Minutes to days | Tickets, orders, inventory, prices, account status | Retrieval, or a direct lookup through a tool call to your database or API. Never fine-tuning |
| Weeks to months | Help-centre articles, policies, product documentation | Retrieval with automated re-indexing; long-context prompting if the material is small |
| Rarely | Brand voice, output schema, labelling rules, the structure of a good summary in your domain | Prompting first; fine-tuning if prompting cannot hold the behaviour steady |
Note that live operational data often needs neither technique. “What is the status of order 4417?” is a database query, which the model can request through a tool call and then phrase for the user. Searching an index for it would be slower and less accurate.
Evaluating each approach
Neither approach should be adopted, or rejected, without numbers from your own examples. What you measure differs.
Retrieval: two separate scores
- Retrieval hit rate. For each test question, is the passage that contains the answer among the top results? If not, no model can answer correctly, and the fix is in parsing, chunking or search, not in the prompt.
- Answer quality given good retrieval. Is the answer correct, is every claim supported by the cited passages, and does the system decline when the sources do not contain the answer?
Separating the two tells you where to spend effort. Teams that look only at final answers tend to rewrite prompts for weeks when the real fault is that the right passage was never retrieved.
Fine-tuning: always against a baseline
- Build the strongest prompted baseline first, with clear instructions and a few examples on a capable model, and score it on the held-out test set.
- Score the fine-tuned model on the same set. If the gain is small, the maintenance commitment is not justified.
- Check for regressions outside the training distribution: unusual inputs, refusals, and any general ability the feature still relies on. Narrow training can degrade behaviour elsewhere.
- Record cost and latency per request for both, because those are usually the reasons for fine-tuning in the first place.
In both cases, keep the evaluation set and re-run it on every change. The process for building one is described in the integration guide linked above.
Long-context prompting: the third option
Current hosted models accept very large prompts, large enough for a full handbook, a contract set or a product manual. That makes a third approach practical: skip retrieval and send all of the reference material with every request, relying on the vendor’s prompt caching to keep the cost down.
When it works well
- The material is small, roughly up to a few hundred pages, and fits comfortably within the model’s limit with room for the conversation
- It changes rarely, so the cached prefix stays valid
- Every user may see all of it, so there are no per-user filtering rules
- Questions need reasoning across the whole document, which is where chunk-based retrieval is weakest
- Request volume is low to moderate, or steady enough to keep the cache warm
Where it stops working
- The collection is larger than the context window, or grows without limit
- Different users may see different documents
- Requests are sparse, so the cache expires between them and every call pays the full price
- Accuracy on specific details falls as prompts get very long. Test this on your own material
- You need precise citations to a passage, which retrieval gives you as a by-product
An illustrative cost comparison
The prices below are placeholders chosen to make the arithmetic easy, not any vendor’s price list. Assume input costs $3.00 per million tokens, and cached input is billed at one tenth of that. The reference material is 150,000 tokens. A retrieval pipeline would instead send about 4,000 tokens of selected passages. Output cost is the same in all three cases and is left out.
| Input tokens per request | Input cost per request | At 10,000 requests per month | |
|---|---|---|---|
| Long context, no cache hits | 150,000 | $0.45 | $4,500 |
| Long context, cache always warm | 150,000 (cached rate) | $0.045 | $450 |
| Retrieval | 4,000 | $0.012 | $120 |
At 300 requests a month the long-context bill is between $13.50 and $135, and building a retrieval pipeline to save that would take years to pay back. At 10,000 requests a month the gap is large enough to fund the pipeline. Put your own token counts and your vendor’s current prices into the same sums before deciding.
Decision table by use case
| Use case | Start with | Why |
|---|---|---|
| Customer-support assistant answering from help articles | RAG | Content changes, answers need citations, and the collection outgrows a prompt |
| Internal knowledge search across departments | RAG with permission filtering | Different staff may see different documents; only retrieval can enforce that |
| Questions about one policy document or one contract | Long-context prompting | Small, stable, and benefits from reading the whole text |
| Account, order or booking status | Tool call to your API or database | Live structured data; neither RAG nor fine-tuning |
| Classifying or routing messages at high volume | Prompting; then fine-tune a small model if volume justifies it | Narrow, stable task with abundant labelled history; cost and latency matter |
| Extracting fields from documents into a fixed schema | Prompting with structured output and validation in code | Vendor structured-output features handle most format problems; fine-tune only for persistent errors on unusual layouts |
| Writing in a specific brand voice | Prompting with a style guide and examples | Usually sufficient. Fine-tune if the voice must hold across very high volume with short prompts |
| Drafting replies that follow company policy | RAG for the policy, prompting for the tone | Policy is knowledge; tone is behaviour, and prompts usually handle it |
| Specialist domain with its own vocabulary and conventions | RAG first; consider fine-tuning in addition | Retrieval supplies the facts; tuning can help the model read and write the domain’s shorthand |
| Running on a device or fully inside your own network | Small open-weight model, often fine-tuned, plus local retrieval | A small model needs help to match a large hosted one on the specific task |
| Agent that uses your tools in a particular sequence | Prompting with clear tool descriptions | Fine-tune only after a measured, persistent failure to choose tools correctly |
Decision rules in order
- Write the task down and collect test examples. No approach can be judged without them.
- Prompt a capable hosted model. Measure.
- If failures come from missing information: place the material in the prompt if it is small and stable, otherwise build retrieval. If the information is live structured data, give the model a tool.
- If failures are about format, use structured output and validate in code before anything else.
- If failures are about consistency or tone despite good prompts, or the unit cost or latency of a large model is the obstacle on a narrow task, and you have hundreds of clean examples, evaluate fine-tuning against the prompted baseline.
- If none of these reaches the accuracy the task needs, reconsider the task: narrow it, add human review, or accept that it is not ready to automate. Our guide to building an AI product covers when AI is the wrong tool.
Combining them
The approaches are layers, not rivals, and mature systems often use several:
- Fine-tuned model plus retrieval. The tuning fixes format, tone or domain vocabulary; retrieval supplies current facts. The training examples should include retrieved passages in the input, so that the model learns to answer from them and to decline when they are insufficient.
- Tuning the retrieval side. When search struggles with specialist vocabulary, adapting the embedding or reranking model to your domain can improve retrieval more than any change to the generating model.
- Routing. A small, possibly fine-tuned model classifies each request, and only the ones that need it go to a large model with retrieval.
- Retrieval plus tools. Documents come from the index, live values come from API calls, and the prompt tells the model which to trust for what.
Add one layer at a time and measure after each. A system with retrieval, a tuned model and a router has three places for quality to go wrong and needs an evaluation for each.
Common misconceptions
| Belief | What is closer to the truth |
|---|---|
| “Fine-tuning on our documents will teach the model our business” | It teaches the model to sound like your documents. Recall of specific facts is unreliable, there are no citations, and updates need retraining |
| “RAG stops hallucination” | It reduces it when retrieval finds the right passage. When retrieval returns the wrong passage, the model answers confidently from it. Measure retrieval, require citations, allow “I do not know” |
| “Fine-tuning is more private because the data stays in the model” | The reverse is nearer the mark. Training data can sometimes be drawn out of a model, it cannot be filtered per user, and it cannot be deleted without retraining. An index can be permissioned and purged |
| “RAG needs a dedicated vector database” | It needs search. A PostgreSQL extension covers vector search for most collections, and keyword search alongside it matters as much |
| “Long context made retrieval obsolete” | For small, stable, unrestricted material, often yes. For large, changing or permissioned collections at volume, no |
| “Fine-tuning is a one-off cost” | It recurs with every base-model retirement and every meaningful change in your inputs |
| “Serious AI products train their own models” | Most useful products are built on hosted models with good retrieval, tools and evaluation. Training is justified by a measured gap, not by ambition |
| “Bigger chunks and more passages mean better answers” | Past a point, extra context dilutes the relevant passage and raises cost. Fewer, better-ranked passages usually win |
How BBR approaches the choice
BBR integrates hosted models and builds the systems around them: retrieval pipelines, agents, guardrails and evaluation. We do not train foundation models. When we scope an AI feature, the first deliverable is an evaluation set and a prompted baseline, because those two things decide whether retrieval is needed and whether anything beyond it is worth considering. In most business cases the answer is prompting plus retrieval plus ordinary engineering. If the numbers say a task needs custom model training, that is work for a machine-learning specialist, and we will tell you so.
If the feature you have in mind is a support or knowledge assistant, our AI chatbot cost guide prices the retrieval route block by block. The scope and process we follow are on the AI development service page.