What is RAG? How to give an AI your documents so it answers accurately

RAG (retrieval-augmented generation) means having the AI look up the relevant passages in your documents before it answers, then asking it to rely only on them. It is the most common way to get an assistant that knows your procedures, contracts or product catalog without retraining the model.

The problem RAG solves

A large language model knows a lot, but it has two blind spots. First, its knowledge stops at a knowledge cutoff: it has no idea what happened after training. Second, it has never seen your private documents, whether that is your company handbook, your product sheets or your course notes.

Ask it about those topics and it may answer anyway, confidently and wrongly: that is a hallucination. RAG fills both gaps in a simple way. Instead of relying on the model’s memory, you bring it the right pages at the right time.

The term was popularized by a research paper by Patrick Lewis and co-authors, posted on arXiv in May 2020. The idea then spread with conversational assistants.

How RAG works, step by step

A RAG system works in two phases: a preparation phase, done once and then kept up to date, and a processing phase for every question.

Preparing the documents

1. Collect the useful documents: PDFs, web pages, spreadsheets, a knowledge base.
2. Split each document into reasonably sized pieces, say a few paragraphs each: this is chunking. A chunk that is too long buries the information; one that is too short loses its context.
3. Convert each chunk into an embedding, a list of numbers that represents its meaning. Two texts about the same thing have nearby embeddings, even if they use different words.
4. Store these embeddings in a vector database, built to quickly find the vectors closest to another one.

Handling a question

1. The user’s question is also converted into an embedding.
2. The system looks for the chunks with the closest meaning: this is semantic search.
3. The best excerpts are inserted into the prompt, with an instruction such as “Answer only from the excerpts provided and cite them; if the answer is not there, say so.”
4. The model writes its answer based on those passages. This is called grounding: the answer is tied to sources you can check.

Here is an example. A small business wants an internal assistant for HR questions. An employee asks: “How many days off do I get when I move house?” The system finds the paragraph of the leave policy that covers it, passes it to the model, and the answer quotes that paragraph. If the policy changes, you just update the database; the model itself stays the same.

Check yourself A RAG assistant says it cannot find the information, yet the document is in the database. What is the most likely cause?

The search did not bring back the right passage. The relevant chunk may be badly split (the information cut in half), worded very differently from the question, or ranked too low among the results. It is almost always the retrieval step that needs work, not the model.

Improving search quality

In RAG, answer quality depends first on retrieval quality. If the right excerpt never reaches the model, even the best model cannot get it right. Four techniques come up often.

• Hybrid search combines meaning-based search with keyword search. It helps with proper names, product codes or legal references that embeddings sometimes handle poorly.
• Reranking first retrieves a large set of candidates, then reorders them with a specialized model to keep only the most relevant.
• Metadata (date, department, document type) lets you filter: there is no point searching 2019 archives for a 2026 procedure.
• Query rewriting. A short or vague question (“what about contractors?”) finds poor matches. Having the model rewrite the question first, using the conversation history, produces a fuller query and better search results.

How well the documents are maintained matters as much as the technique. Outdated, contradictory or badly scanned files lead to inconsistent answers. Assign someone to own the knowledge base, as you would for an intranet.

RAG, fine-tuning or long context: which one?

RAG is not the only way to give an AI knowledge. Two alternatives come up often: fine-tuning, which changes the model through extra training, and long context, which means sending everything to the model at once.

Approach	How it works	Best for	Limits
RAG	Retrieve excerpts and attach them to the question	Large, changing information that must be cited	Only as good as the search
Fine-tuning	Retrain the model on your examples	Consistent style, format, domain vocabulary	Costly to update, does not cite sources
Long context	Send all documents with every question	Small collections, one-off review of a file	Expensive when repeated, diluted attention

Context windows have grown a lot. According to models.dev, as of October 22, 2026, several recent models accept around one million tokens, as the chart below shows.

Context window (tokens)

• Claude Sonnet 5.51 M
• Gemini 4 Argon1.05 M
• GPT-6.1 Sol1.05 M
• Mistral Large 41.05 M
• Grok 4.7500 k
• Kimi K2.6262 k

To picture what these numbers mean in pages of text, try our context window tool.

Tools for building RAG

You do not have to code everything yourself. Several families of tools exist, from low-code to code.

Tool	Type	License	MCP	Multi-agent	Level
LlamaIndex	Code	MIT	Yes	Yes	Intermediate
Haystack	Code	Apache-2.0	Yes	Yes	Intermediate
LangChain	Code	MIT	Yes	Yes	Intermediate
Dify	Low-code	Dify Open Source License	Yes	—	Intermediate
n8n	Low-code	fair-code	Yes	Yes	Intermediate

• LlamaIndex started in November 2022 as GPT Index, precisely to connect a language model to your own data. According to our profile, the company now focuses mainly on LlamaParse, its commercial document-parsing platform, while the framework remains open source.
• Haystack, from deepset, builds explicit pipelines: chunking, embeddings, search, model call. Version 3 came out in July 2026, according to our profile.
• Dify takes a visual approach: you upload documents, the platform splits and indexes them, and you build a chatbot that answers from them.

If you want to go further and turn RAG into a tool for an agent that decides when to search, our guide on how to build an AI agent, from no-code to code walks through the full process.

Where to start: a five-step method

A useful first RAG project does not need to be ambitious: a small, well-managed scope beats a large, messy knowledge base.

1. Pick a narrow scope. One topic, such as leave questions or a product catalog, rather than all of the company’s documentation.
2. Clean up the documents. Remove outdated versions and duplicates, make sure scanned PDFs are readable as text, and record the date of each document.
3. Write your reference questions. About twenty real questions from real users, with the correct answer and the document where it lives. This small test set will guide every adjustment.
4. Build a simple first version. Paragraph-level chunking, semantic search, and an instruction that requires citing sources and admitting when information is missing.
5. Measure, then improve. Find the questions that fail and trace the cause: missing document, poor chunking, badly ranked passage, or a model that strays from the excerpts. Fix one thing at a time and rerun your tests.

Resist the urge to stack advanced techniques on day one. A clean knowledge base and a clear instruction often do more than a sophisticated setting.

What RAG costs

RAG has three cost items: creating embeddings (once, then with every update), hosting the vector database and, above all, the tokens sent to the model with each question. Because only a few excerpts are passed along, the cost per question is usually far lower than sending the whole document. Our guide to AI API pricing explains the math and the levers that cut the bill.

Limits to keep in mind

Other risks are less visible.

• Security. A document in the database may contain hidden instructions written to hijack the assistant: this is a form of indirect prompt injection. Our guide to AI agent security covers the safeguards.
• Access rights. An employee should not be able to get, through the assistant, a document they are not allowed to read. Permissions must be enforced at search time, not just on the intranet.
• Privacy. Excerpts are sent to the model provider. For personal or sensitive data, check the provider’s terms, or consider a model you host yourself.
• Measurement. Without a set of reference questions, you cannot tell whether a change improves or degrades answers. Our guide on how to evaluate an AI agent offers a method for in-house testing.