Fine-tuning vs. RAG: choosing, data, and costs

Choose RAG when a model needs to consult private or recent information, or provide sources for its answers; consider fine-tuning when you need lasting improvements in a specialized behavior using corrected examples. Start by measuring the errors of a model with clear instructions: the approaches can work together, but neither replaces reliable data or independent evaluation.

What each approach actually changes

RAG, or retrieval-augmented generation, adds a retrieval step before the answer. The system selects passages from a document collection, then passes them to the model along with the question. In this common architecture, the generating model’s parameters remain unchanged.

Fine-tuning continues the training of an existing model. In supervised fine-tuning, examples pair an input with the desired output: classifying a request, extracting fields, or writing a response that follows a business convention.

The distinction between knowledge and behavior is a useful starting point, but it is not an absolute boundary. Training can cause a model to memorize facts; examples placed in the context can change its behavior. The practical question is where to put information so that you can control it, update it, and verify how the system uses it.

The comparison in the AWS documentation recommends starting with RAG when querying an organization’s own documents. It also notes that the approaches can be combined.

To explore the available components, see the language model profiles. Choosing a base model that already performs well on your task can avoid expensive customization.

Decision table: information, behavior, and constraints

Criterion	RAG	Fine-tuning
Main objective	Provide access to selected information	Adapt responses to a task or behavior
Starting data	Usable documents and metadata	Representative examples with expected outputs
Updates	Update sources and their index	Prepare and evaluate another training run
Document references	Possible when passages retain their provenance	Not automatically supplied by training
Access control	Must be enforced during document selection	Difficult to apply individually to memorized facts
Initial cost	Preparation, ingestion, retrieval, and integration	Annotation, training, evaluation, and deployment
Cost per request	Retrieval, supplied context, and generation	Custom model inference and any hosting costs
Typical failure	The useful passage never reaches the model	The learned behavior generalizes poorly

Frequently changing documentation, internal procedures, and product specifications point toward retrieval. A stable, repetitive classification task with clearly defined categories may justify specialized training.

Traceability needs a qualification: retrieving a source does not guarantee that the answer follows it. Keep the passage used, check that it supports the claim, and distinguish a relevant reference from a link simply appended to the answer.

Neither RAG nor fine-tuning is necessary for every task. If a few documents fit within the context and access is authorized, passing them directly to the model may be enough. For an exact calculation or transactional data, a deterministic query may be preferable.

Diagnose errors before choosing

Build a set of representative requests and observe a simple baseline: clear instructions, relevant examples, and a suitable model. Then classify errors by their likely cause, instead of collapsing them into a single poor overall score.

One useful diagnostic is to give the model the correct passage manually. If it answers correctly, the problem probably lies in access to information. If it still fails despite complete, readable evidence, examine the instructions, the model’s capabilities, or the task’s complexity.

Distinguish these cases in particular:

Missing information: the system does not have the answer.
Retrieval failure: the answer exists, but the useful passage is not retrieved.
Misuse of evidence: the right passage is present but misinterpreted.
Inconsistent behavior: the model understands the task but applies a recurring convention incorrectly.
Poorly defined task: the evaluators themselves disagree on the expected output.

Fine-tuning is worth testing when behavior remains unreliable despite adequate instructions and context. It does not directly fix a document extraction process that has lost the columns of a table.

For strict output formats, first test schema-constrained generation if your inference system supports it, followed by application-level validation. Training a model to produce a structure does not remove the need to check fields, types, and permitted values.

The data you need for RAG

A folder full of files is not yet a usable knowledge base. Identify authoritative documents, remove duplicates, and retain effective dates, document versions, provenance, and access permissions.

Preparation depends on the content. A table requires the relationships between cells to remain intact; a procedure must retain its conditions and exceptions. Splitting documents mechanically can separate a rule from a restriction that changes its meaning.

An embedding represents content numerically to support similarity search. It does not check whether that content is valid or current. Depending on the requests, a search that combines semantic similarity, keywords, and filters may work better than semantic search alone.

Also prepare reference questions paired with the expected passages. Include industry acronyms, exact identifiers, ambiguous questions, and requests for which no answer is available. This lets you evaluate document selection before judging the writing.

The RAG reference guide explains chunking and indexing in more detail. When choosing a solution, our selection of document knowledge base and RAG tools helps you examine the components without confusing a retrieval database with a complete application.

Plan how to remove outdated documents as well. Adding a new procedure without disabling the old one leaves the system choosing between contradictory texts that may otherwise look almost identical.

The data you need for fine-tuning

For supervised training, show what the model should produce under actual operating conditions. Each example needs a representative input and an output corrected according to an explicit rule. Collecting historical responses without reviewing them can teach the model a service’s inconsistencies.

Coverage matters as much as volume. A dataset containing only easy requests tells you little about exceptions. Include incomplete inputs, unusual wording, closely related categories, and situations where the correct response is to ask for clarification.

Separate training, validation, and test data. Keep variations of the same case from appearing across these sets: the model might recognize a case instead of learning the task. Depending on the application, you may need to split the data by customer, document, or time period.

There is no universal minimum number of examples. Start with a carefully checked batch, increase its diversity, and track performance on the independent test set. If improvements plateau, examine the annotations before buying more compute.

Synthetic examples can supplement your data, but they need verification. A generator can reproduce the same error in many different phrasings. Record where examples came from and distinguish those reviewed by a specialist from those that remain automatically generated.

Finally, write annotation instructions that several people can apply consistently. Disagreement over the expected answer often reveals a problem with the task definition that training will not solve.

Full fine-tuning, adapters, and deployment

Full fine-tuning adjusts all the parameters selected as trainable for the model. Adapter methods limit which parameters change. The LoRA technical documentation explains how this approach freezes the base weights and learns smaller adaptation matrices.

That reduction concerns trainable parameters; it does not mean the base model disappears. Training and inference still need to load the necessary components, with memory requirements depending on the model, context, and configuration.

An adapter must also remain compatible with its base model. Archive the exact model identifier, training settings, data used, and evaluation results. An adapter file on its own is not enough to reproduce an application.

With a hosted API, check which models support customization, which formats are accepted, and what deployment conditions apply when you begin the project. A model being available for inference does not establish that it supports fine-tuning.

For a deployment you manage yourself, include licensing, maintenance, and computing capacity in the decision. The technical ability to train does not settle data usage rights or operational constraints.

Above all, the chosen model must perform the task successfully after customization. A falling training loss shows that optimization is progressing on the training examples; on its own, it does not demonstrate better performance on new requests.

Calculate the full cost beyond tokens

RAG adds expenses for extraction, initial indexing, updates, storage, retrieval, optional reranking, and context sent to the generator. Fine-tuning adds example preparation, training experiments, evaluation, and deployment of the resulting model.

Human work belongs in the budget. Cleaning a corpus or resolving conflicting annotations can cost more than model calls. Include fresh evaluations when a procedure, category, or base model changes.

Cost item	RAG	Fine-tuning
Setup	Document cleanup and retrieval pipeline	Task definition and annotation
Initial compute	Extraction and indexing	Training runs and experiments
Operations	Retrieval, storage, and generation	Adapted model inference and hosting
Maintenance	Synchronizing and removing sources	New examples and retraining
Quality control	Retrieval, evidence, and answers	Generalization and regressions

Billing units depend on the provider. The Amazon Bedrock customization documentation describes, among other items, training charges based on corpus tokens multiplied by training passes, along with model storage. These arrangements are not a universal pricing model.

Compare costs over the same time horizon:

Total cost = preparation + experimentation + operations + maintenance + error handling.

The final item includes manual rework and repeated requests. The LLM pricing calculator helps estimate usage costs; add your infrastructure and staffing expenses to that amount.

A numerical example and the break-even point

Consider a hypothetical cost scenario, with no performance measurements: an application handles 10,000 requests per month. Each request contains 1,000 tokens of instructions and question text, receives 4,000 tokens from documents, and produces 500 tokens.

To isolate generation costs, use the pricing for GPT-6.1 Sol: $2 per million input tokens and $10 per million output tokens, as recorded by the site in October 2026. This example makes no assumption that fine-tuning is available for this model.

Under these assumptions, with no caching, discounts, or additional calls:

Monthly input: 50 million tokens, costing $100.
Monthly output: 5 million tokens, costing $50.
Total generation: $150, including $80 attributable to document passages.

This amount excludes retrieval, indexing, hosting, and maintenance. It illustrates why you should measure the length of useful passages instead of routinely sending more documents.

A customized model might reduce some repeated examples or instructions. It does not let you remove documents needed for recent facts while preserving the same expected reliability.

Calculate the break-even point as follows: additional fixed customization cost divided by net savings per successfully handled request. Those savings must account for corrections, retries, and hosting. If they are zero or negative, higher volume will not recover the training cost; training must then be justified by a measured improvement in quality.

Example choices and a combined architecture

The following situations are hypothetical design examples, not customer cases or observed results.

Document support assistant. It needs to explain the terms that apply to a particular product. RAG can select documentation for that product and its validity period. The decisive test is whether the system uses the correct procedure, handles its exceptions, and supports its answer with evidence.

Incoming request classification. Each message needs a stable category from an internal classification system. After testing instructions and examples, you can evaluate fine-tuning on recurring category confusions. If the categories change frequently, the cost of relabeling examples becomes a deciding factor.

Writing to a business convention. The system needs to consult current procedures, then organize its response in a specific way. RAG supplies the facts; training may improve how the model uses them. The examples must then include the document context that will actually be available.

The paper RAFT: Adapting Language Model to Domain Specific RAG studies such a combination, with training that includes relevant documents and distractors. It offers a possible design approach without guaranteeing an improvement on your corpus.

Before adopting this architecture, compare it with RAG alone. It adds components and data that need maintenance. For a related requirement, the article on AI agent memory distinguishes retaining information across sessions from training the model.

Evaluate retrieval and answers separately

An overall average can hide which component is failing. For RAG, first measure whether the expected evidence is retrieved. Then evaluate whether the answer follows that evidence, addresses the request, and acknowledges missing information.

The Microsoft documentation on RAG evaluators distinguishes retrieval quality, groundedness, relevance, and completeness, among other dimensions. These dimensions help separate a faithful but incomplete answer from a detailed answer containing unsupported claims.

For fine-tuning, measure the intended task: correct classification, exact field values, handling of exceptions, or writing quality assessed against a rubric. Also check the behaviors you did not intend to change.

Compare the base model, RAG alone, the customized model, and, if necessary, their combination on the same independent test set. Keep conditions comparable and record configuration changes.

Include contradictory documents, malformed inputs, and requests outside the system’s scope. A sound evaluation also tests abstention: answering every request is not a quality objective.

Finally, measure latency and cost per request resolved according to your criteria. A system with cheaper individual calls can cost more if it forces users to try again. The guide to evaluating an AI agent explains how to build a test set and track regressions.

Privacy, maintenance, and the final decision

In a RAG system, enforce permissions before passing passages to the model. An interface-level filter is insufficient if retrieval can access another user’s documents. Indexes, caches, and logs also belong within the scope of protection.

Retrieved text remains data, even when it contains instructions. Define its role and prevent instructions found inside a document from changing application permissions. Training should not be treated as sufficient protection against this kind of manipulation.

For fine-tuning, examine which data could be memorized, who can query the model, and how you would replace it. Avoid training on secrets or information whose access permissions vary by user.

Plan a rollback to the previous configuration for every change. Version the corpus, index, instructions, and adapted model so that you can explain a regression. An apparent improvement becomes difficult to verify when several components change at once.

Key takeaways

Start by diagnosing errors with a simple baseline and an independent test set.
Choose RAG to consult information that needs to remain updatable and verifiable.
Evaluate fine-tuning for specialized behavior demonstrated through consistent examples.
Combine the approaches only when measured gains justify their maintenance.
Budget for preparation, operations, and errors, then compare cost per successfully handled request.

Frequently asked questions

Can you use RAG without a vector database?

Yes. Keyword search, a relational database query, or a combination of methods can select the information passed to the model. A vector database is a retrieval option, not a requirement for a system to qualify as RAG.

How many examples do you need for fine-tuning?

There is no universal number that guarantees results. The useful amount depends on the variety of inputs, the difficulty of the task, and the model you choose. Build progressively larger batches and check whether performance on independent cases continues to improve.

Does fine-tuning eliminate hallucinations?

No. It can improve a specific behavior, but it can also reproduce errors and biases in the training examples. A more polished answer can still be factually wrong.

Can you remove information learned through fine-tuning?

Removing information from the training file does not change a model that has already been trained. You need a replacement or retraining procedure, followed by checks that it worked. Do not treat model weights as a document database where each piece of information can be deleted separately.

Do you need an open-weight model for fine-tuning?

Not necessarily: some APIs offer hosted customization for supported models. Access to the weights gives you more control over training and deployment, subject to the license. Also compare maintenance requirements, computing capacity, and migration options.

Fine-tuning vs. RAG: how to choose, costs, data requirements, and examples