Tokenizer: the tool that chops text up for AI models
A tokenizer is the program that splits your text into small pieces called tokens, then turns each one into a number, the only form a language model can process. It decides how many tokens your message contains, and therefore what it costs and how much room it takes.
Data as of October 22, 2026
Key figures
| Level | Intermediate |
|---|---|
| Origin | 2015 |
| Introduced by | Sennrich, Haddow et Birch (sous-mots BPE) |
What a tokenizer does
A large language model doesn’t read letters or words: it reads sequences of numbers. The tokenizer translates in both directions. On the way in, it splits the text into tokens and gives each one an ID from a fixed vocabulary; on the way out, it turns the IDs the model produces back into readable text.
Most modern tokenizers use subword splitting. The idea, popularized by Sennrich and his co-authors in 2015 for machine translation, is simple: frequent fragments become tokens in their own right, and rare words are rebuilt from pieces. The documentation for tiktoken, OpenAI’s tokenizer, gives the example of “encoding” split into “encod” + “ing”. It also notes that, on average, a token corresponds to about 4 bytes of text.
A concrete example
A sentence like “The cat sleeps.” fits in a handful of tokens: short, common words. A word like “antidisestablishmentarianism” gets broken into several pieces. The upshot: two texts that look the same length can have different token counts, and text in languages other than English often produces more tokens than its English equivalent. Try it with our token estimator.
Why it matters
The tokenizer sets the unit of measure for almost everything. APIs bill by price per million tokens, and the context window is measured in tokens. Changing the tokenizer therefore changes the bill for the same text. Anthropic, for example, states in its documentation that its models from Claude 4.7 onward use a new tokenizer that produces about 30% more tokens for the same text, and recommends recounting prompts against the target model.
Common mistakes and myths
- “1 token = 1 word.” It is a rough approximation that varies by language and model.
- “Token counts carry over between models.” They don’t: compare costs by recounting with each tokenizer.
- “The model sees letters.” It sees tokens. That is one reason why counting the letters in a word or spelling it backwards can trip it up.
Related terms: token, context window, price per million tokens, embedding.
Related terms
- Token · A small chunk of text (a word, part of a word or a symbol) that an AI model reads and writes; the unit for limits and pricing.
- Price per million tokens · How AI APIs bill usage: one rate per million tokens the model reads as input, and another, usually higher, per million it generates.
- Context window · The maximum amount of text (in tokens) a model can read and produce at once, including its own answer.
- Large language model (LLM) · A program trained on huge amounts of text to predict what comes next in a text; it is the engine behind AI assistants.
- Embedding · A list of numbers (a vector) that captures the meaning of a text or image, so content can be compared by how close in meaning it is.
Try it
Full glossary
Frequently asked questions
Is a token the same as a word?
Not quite. A common word may fit in a single token, while a rare or long word is split into several pieces. Punctuation and spaces count too.
Do all models use the same tokenizer?
No. Each model family has its own, and it can change from one generation to the next. The same text will not produce the same token count everywhere.
How do I find out how many tokens a text has?
Providers offer counting tools: the tiktoken library for OpenAI models, and a free token-counting endpoint at Anthropic. For a quick ballpark figure, use our token estimator.
Where does subword tokenization come from?
The most widespread approach, BPE (Byte Pair Encoding) applied to subwords, was popularized in language processing by a paper from Rico Sennrich, Barry Haddow and Alexandra Birch, posted to arXiv in August 2015.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.