Chunking: splitting documents so AI can actually find them
Chunking means splitting long documents into small, coherent passages, called chunks, before indexing them. It is a quiet step in a RAG pipeline, but it often decides how good the answers are.
Data as of October 22, 2026
Key figures
| Level | Advanced |
|---|
What it is
Put simply: you can’t search a long manual efficiently as a single block, so you cut it into index cards. In a RAG pipeline, each chunk is converted into an embedding and stored in a vector database; when a question comes in, only the closest chunks are sent to the model. Size is measured in tokens, and our token estimator gives you a feel for how big a passage really is.
Main strategies
- Fixed size: cut every N tokens, often with some overlap. Simple and fast, but sometimes cuts mid-sentence.
- Recursive splitting: try to cut at headings first, then paragraphs, then sentences, until a maximum size is respected.
- Structure-aware: follow HTML or Markdown sections, the clauses of a contract, the functions in a code file.
- Semantic: cut where the topic changes, detected by shifts in meaning between consecutive sentences.
The orphan chunk trap
The most common problem is lost context. Anthropic gives the example of a passage saying revenue grew compared with the previous quarter, without stating which company or which period. On its own, that chunk is almost impossible to retrieve. In its Contextual Retrieval study (September 2024), Anthropic proposes having a model write a short situating context that is prepended to each chunk before indexing, which substantially reduced retrieval failures on its test sets.
A concrete example
A health insurer indexes its policy terms. Split at a fixed size, the “exclusions” clause ends up separated from its heading, so a question about uncovered dental care retrieves a passage with no context. Re-split clause by clause, with the section title repeated at the top of each chunk, the same questions retrieve the right clause.
Common mistakes
- Picking a size at random without measuring: test several settings on a question set.
- Chunks that are too big: they dilute meaning and waste the context window.
- Chunks that are too small: they lose the context needed to understand them.
- Forgetting metadata: source, date and section help with filtering and citation.
Related terms: RAG, hybrid search, embedding, vector database.
Related terms
- RAG (retrieval-augmented generation) · A technique that first retrieves relevant passages from your documents, then hands them to the model so its answer is based on them.
- Embedding · A list of numbers (a vector) that captures the meaning of a text or image, so content can be compared by how close in meaning it is.
- Vector database · A database that stores embeddings and quickly finds the ones closest to a query, which means the most relevant pieces of content.
- Context window · The maximum amount of text (in tokens) a model can read and produce at once, including its own answer.
- Hybrid search · Search that combines exact keyword matching with meaning-based (embedding) search, then merges both result lists into a single ranking.
- Token · A small chunk of text (a word, part of a word or a symbol) that an AI model reads and writes; the unit for limits and pricing.
Try it
Full glossary
Frequently asked questions
What chunk size should I use?
There is no universal ideal. Anthropic notes that chunks are usually no more than a few hundred tokens. The right choice depends on your documents and should be validated by testing retrieval on real questions.
What is chunk overlap for?
Repeating a few sentences at the boundary between two chunks prevents an idea split in half from being lost. The downside: more duplicates in the index and slightly more storage.
Do long-context models make chunking unnecessary?
For a handful of documents, sending the full text can be enough. For a large document base, splitting and selecting is generally cheaper and more precise, because the model reads less irrelevant text.
How should tables or code be chunked?
By respecting their structure: a table should stay whole with its headers, and a code function should not be cut in the middle. Structure-aware splitters beat fixed-size splitting here.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.