Long context: getting an AI to read entire documents
Long context refers to a language model's ability to process very large amounts of text in one go, such as a book, a document collection or an entire code repository. The term also covers the architecture and training techniques that make this possible, and the methods used to check that the model genuinely uses all that text.
Data as of October 22, 2026
Key figures
| Level | Advanced |
|---|
A plain definition, then the technical one
A long-context model can keep an entire document “in view” while it answers, rather than just an excerpt. In practice that means a very large context window, and a model trained to actually use it.
Three technical hurdles had to be cleared:
- The cost of attention. In a standard Transformer, the attention mechanism compares every token with every other token, so computation grows with the square of the length.
- Memory. The KV cache, which stores information about tokens already read, grows with the length of the text and quickly fills GPU memory.
- Positions. A model trained on short texts does not know how to interpret positions it has never seen. Methods such as Position Interpolation (Chen et al., 2023) and YaRN (Peng et al., 2023) extend the window of models that use RoPE, with only limited extra training.
What evaluations show
The paper “Lost in the Middle: How Language Models Use Long Contexts” (Liu et al., 2023) found that performance is often best when the relevant information sits at the start or end of the context, and degrades when it is buried in the middle. In 2024, the RULER benchmark argued that the “needle in a haystack” test, which asks a model to find one hidden sentence, only measures a superficial kind of understanding: harder tasks such as following chains of references or aggregating scattered information reveal a usable length that is often shorter than the advertised one.
A concrete example
You give an assistant your building’s bylaws, the minutes of several owners’ meetings and a contractor’s quote, then ask: “Have these works already been approved?” A long-context model can cross-reference all of it without you splitting the files first. For a reliable answer, ask it to quote the exact passages, and check them.
Common misconceptions
- “The bigger the window, the better.” A huge window that is poorly used is worth less than a smaller one the model handles well.
- “Long context makes RAG obsolete.” RAG remains useful for large knowledge bases, for cost control and for traceable sources.
Our context window tool helps you estimate whether your documents fit in a given model’s window.
Related terms
- Context window · The maximum amount of text (in tokens) a model can read and produce at once, including its own answer.
- RoPE (rotary position embedding) · A technique that tells a Transformer where each token sits by rotating its vectors through an angle that depends on that position.
- KV cache · Memory where a model keeps the work already done on previous tokens so it does not have to redo it for every new word.
- RAG (retrieval-augmented generation) · A technique that first retrieves relevant passages from your documents, then hands them to the model so its answer is based on them.
- Attention mechanism · The technique that lets a model weigh, for each word it processes, how relevant every other word in the text is.
- Context engineering · The practice of deciding what goes into a model's context window at each step: instructions, tools, documents, memory and history.
Try it
Full glossary
Frequently asked questions
If a model accepts a huge context, does it really read everything?
It processes all of the text, but does not make equally good use of every part. Studies have shown that information placed in the middle of a long context is used less effectively than information at the start or end. The advertised maximum does not tell you the length that is actually useful.
Long context or RAG: which should I choose?
Long context is simple and effective for analysing a closed set of documents in one pass. RAG is still the better fit for a very large, changing knowledge base, for keeping the cost of each query down, or for citing sources precisely. The two are often combined.
Why is long context expensive?
Every token you send is billed, and both the attention computation and the memory required grow with length. Prompt caching cuts the bill when you resend the same long document repeatedly.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.