Pre-training: where a language model learns language
Pre-training is the first and most expensive stage in building a large language model: the model reads a vast amount of text and learns to predict what comes next. It is where the model picks up most of its knowledge and its command of language.
Data as of October 22, 2026
Key figures
| Level | Intermediate |
|---|---|
| Origin | 2018 |
| Introduced by | Radford et al. (OpenAI) |
The definition
Pre-training a model means having it read gigantic amounts of text (web pages, books, code, articles) while setting it the same exercise over and over: guess the next token. The model makes a guess, the guess is compared with the real text, and its parameters are nudged so it does slightly better next time. Repeated an enormous number of times, this simple exercise forces it to absorb grammar, vocabulary, facts about the world and patterns of reasoning.
It is called “pre”-training because it is only the first step. The result, known as a base model, can continue a text but cannot yet hold a proper conversation or follow instructions well.
A worked example
Give a base model the opening “Pancake batter recipe: in a large bowl, add” and it will carry on plausibly with flour and eggs. Ask it “Can you give me a pancake recipe?”, however, and it may produce a list of similar questions, as if it were a forum page. It is the next stage, with fine-tuning and RLHF, that turns it into an assistant that actually answers.
Where the approach came from
The idea of pre-training a model on unlabelled text before adapting it to a task was popularised in 2018, notably by OpenAI’s paper “Improving Language Understanding by Generative Pre-Training” (Radford et al.), the origin of the GPT name, and by Google’s BERT the same year. You can find these milestones on the AI timeline.
Why it matters
Pre-training explains several things you notice day to day. The quality and variety of the training data shape what a model knows, and its biases. The point where that data ends sets the knowledge cutoff. And the training compute required is so large that only a few labs pre-train the biggest models; the site’s models list shows who publishes them.
Common myths
- “The model has memorised the internet.” It has extracted statistical patterns; it can reproduce some passages, but it does not hold a searchable copy of the web.
- “More data is always better.” The quality and filtering of data matter as much as the volume.
- “Pre-training alone makes an assistant.” Without the post-training stages, the model does not follow instructions well.
Related terms
- Training data · The examples (text, images, audio and more) an AI model learns from; they shape what the model knows and where it goes wrong.
- Fine-tuning · Continuing the training of an already trained model on a small, targeted set of examples to adapt it to a task, style or output format.
- RLHF (reinforcement learning from human feedback) · A training method where people compare model answers, and the model learns to favour the responses humans judge helpful and safe.
- Knowledge cutoff · The date after which an AI model received no more training data, so it knows nothing about later events.
- Training compute (FLOP) · The total number of arithmetic operations, measured in FLOP, needed to train an AI model.
- Scaling laws · Measured relationships that predict how a model's performance improves with its size, its training data and its compute.
Full glossary
Frequently asked questions
Does a model keep learning while I chat with it?
Not in real time. Its core knowledge is frozen at the end of pre-training. Some providers may use conversations to train future versions, depending on their terms and your privacy settings.
Why doesn't my assistant know about recent news?
Because its pre-training data stops at a certain date, known as the knowledge cutoff. For anything newer, the assistant has to rely on web search or on documents you give it.
What's the difference between pre-training and fine-tuning?
Pre-training teaches the model language and general knowledge from broad text. Fine-tuning, which is shorter and cheaper, then adjusts that model for a specific task, style or behaviour.
Can I pre-train my own model?
For a large model, that is out of reach for most organisations because it takes enormous amounts of data and computing power. It is common, however, to start from an open-weights model that has already been pre-trained and fine-tune it.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.