Transformer: the architecture behind large language models
The Transformer is a neural network architecture, in other words a blueprint for building an AI model, introduced by Google researchers in 2017. Almost every large language model today, from GPT to Claude and Gemini, is built on it.
Data as of October 22, 2026
Key figures
| Level | Intermediate |
|---|---|
| Origin | 2017 |
| Introduced by | Vaswani et al. (Google) |
A plain definition
A neural network can be wired up in many ways. The Transformer is one particular layout, and its key component is the attention mechanism: to process each word, the model looks at every other word in the text and decides which ones matter most.
Earlier language models read text one word at a time, in order, like running a finger along a line. A Transformer can examine all the words in parallel. That change had two big effects: it captures links between words that are far apart much better, and it trains far more efficiently on graphics processors, which excel at doing many calculations at once.
Where it came from
The architecture was introduced in June 2017 in the paper “Attention Is All You Need” (Vaswani et al., Google). The title sums it up: a model based solely on attention, dropping the recurrent machinery used until then. Its first target was machine translation. By 2018, OpenAI with GPT and Google with BERT had shown that a Transformer pre-trained on huge amounts of text could then be adapted to many tasks. You can place these milestones on the site’s AI timeline.
A worked example
Take the sentence: “The cup fell off the table because it was fragile.” What does “it” refer to? To a person, obviously the cup. A Transformer makes that link by giving the word “it” a high attention weight towards “cup” and a low one towards “table”. Swap “fragile” for “wobbly” and the attention shifts to “table”. This ability to connect words by meaning is what makes assistants’ replies read so naturally.
Why it matters
Knowing that large language models are Transformers helps explain their strengths and limits. They split text into tokens, and the cost of attention climbs steeply as the text gets longer, which is one reason context windows are limited. Variants such as mixture of experts rework parts of the Transformer to make it more efficient without changing the core idea.
Common myths
- “Transformer is a product name.” It is a public architecture, described in a research paper and used by nearly every AI lab.
- “A Transformer understands text the way we do.” It computes statistical relationships between tokens; the result looks like understanding, but the mechanism is very different from ours.
- “All Transformers are equal.” Size, training data and tuning produce very different models from the same basic architecture.
Related terms
- Attention mechanism · The technique that lets a model weigh, for each word it processes, how relevant every other word in the text is.
- Neural network · Software made of many small connected computing units that learns by adjusting the strength of the links between them.
- Large language model (LLM) · A program trained on huge amounts of text to predict what comes next in a text; it is the engine behind AI assistants.
- Pre-training · The first training phase of a language model: it learns to predict the next piece of text across a huge corpus, with no specific task.
- Token · A small chunk of text (a word, part of a word or a symbol) that an AI model reads and writes; the unit for limits and pricing.
- Mixture of experts (MoE) · An architecture in which a model holds many expert sub-networks but only switches on a few of them for each token it processes.
Full glossary
Frequently asked questions
What does the “T” in GPT stand for?
GPT stands for “Generative Pre-trained Transformer”: a generative Transformer, pre-trained on large amounts of text. The model's very name tells you its architecture.
Was the Transformer invented for chatbots?
No. The 2017 paper presented it first for machine translation. Only later was it used to generate text at scale, and then for images, audio and code.
Are there alternatives to the Transformer?
Yes. Researchers are exploring other architectures, sometimes combined with the Transformer in hybrid models. For now, the Transformer remains the dominant basis for large language models.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.