Token: the basic unit an AI reads

A simple definition

Before reading your text, an AI model chops it into pieces using a tokenizer. Short, common words often make up a single token; long or rare words are split into several. Spaces, punctuation and numbers count too.

A simplified example: “unbelievably” may be split into a few pieces, while “the” counts as just one. The exact split varies from one model to another, which is why the same text does not have the same token count everywhere.

The model then writes its answer token by token, which is why you see the text appear progressively on screen, a bit like someone typing in real time.

Why it matters to you

Tokens are the unit of measurement for three very concrete things:

• Maximum length: every model has a context window, a limit in tokens that covers your request, attached files, the conversation history and the answer itself.
• Price: developer services charge by the price per million tokens, often with one rate for tokens sent and another for tokens generated.
• Quotas: some plans cap usage based on the volume processed.

A workplace example: you want an automated tool to summarize around a hundred meeting reports every week. Estimating the token count tells you whether the documents fit within the model’s limit and roughly what it will cost. The token estimator does this rough calculation in your browser, and the agent cost calculator turns those volumes into a budget in dollars.

Common myths

• “One token equals one word.” No. A token is often less than a word, and the ratio changes with the language.
• “Tokens are a kind of currency.” In this context they have nothing to do with crypto tokens: they are simply units of text.
• “Images don’t count.” Multimodal models also convert images and files into tokens, and those take up room within the limit.