Tokens

Text → numbers the model can see

Sources: Neural Machine Translation of Rare Words with Subword Units — Sennrich et al., 2016; OpenAI token-counting guide

A neural network can only operate on numbers, not text. So before any actual model work happens, every prompt you type goes through a small program that turns text into a list of integers, or whole numbers. That program is called a tokenizer, the chunks of text it produces are called tokenstokenThe atomic unit of text the model sees. Roughly a word-fragment — “tokenization” is a piece of text → list of token IDs.See in glossary →, and the fixed list of all possible tokens the model knows about is called its vocabularyvocabularyThe fixed set of tokens a model knows about. Modern LLMs have ~32k–200k entries.See in glossary →.

For modern LLMs the vocabulary has somewhere between 32,000 and 200,000 entries. Each entry has an integer index (its token IDtoken IDAn integer index into the vocabulary that uniquely identifies a token.See in glossary →) and that integer is what flows into the rest of the network. From the model’s point of view, your prompt is not “Hello, world” but [9906, 11, 1917].

Why not just use characters? Or whole words?

If we tokenized one character at a time, every prompt would be very long. A 500-word email might become 3,000 tokens, and since the cost of inference grows with sequence length, that gets expensive fast. The model would also have to learn that c, a, t next to each other means something, instead of being told “this is the word cat” up front.

If we tokenized whole words, common words would be easy, but English alone has hundreds of thousands of them. A word the tokenizer had never seen (a typo, a new product name, a piece of code) would typically be mapped to a generic “unknown” token, losing its spelling and identity. Worse, modern LLMs are expected to handle every major language at once: a vocabulary that covered the words of English, Mandarin, Spanish, Hindi, Arabic, Japanese, Russian, and the other 90+ languages users actually type would balloon into the millions of entries, and still miss every word never written down before.

Many modern LLMs use subword tokenizationsubword tokenizationSplitting text into units that can be whole words or smaller word pieces, so unfamiliar words can be represented using known pieces.See in glossary →, which can split words into smaller pieces. A bytebyteA group of eight bits, capable of representing 256 distinct values. A bit is a single binary digit: zero or one.See in glossary → is a group of eight binary digits used to store data. In a byte-level byte-pair encoding (BPE)BPEByte-Pair Encoding — the most common tokenization algorithm. It merges frequent byte pairs into tokens.See in glossary → tokenizer, which builds tokens by repeatedly joining frequent adjacent pieces, such as cl100k_base, common words may get a single token, less common words split into pieces, and arbitrary text remains encodable by falling back to byte-level pieces. Other tokenizers use related algorithms such as WordPiece or Unigram, so their exact behavior differs.

Try it

Below is a real tokenizer running in your browser: cl100k_base, a 100,000-token encoding used by several OpenAI models, including GPT-4 and GPT-3.5 Turbo. It is not the tokenizer for every OpenAI model (GPT-4o, for example, uses o200k_base) but it is a useful concrete example. Type anything, or pick one of the sample inputs. Watch how the same text gets chopped up into different numbers of tokens depending on whether it’s common English, code, a long word, or another language.

A few things worth noticing as you play with it:

• Common words are often one token. “The”, “and”, “world”: each is a single chip. Their numerical IDs are just indices into the vocabulary; they do not encode meaning.
• Rare words get split. Try “antidisestablishmentarianism”: it breaks into 5–6 pieces. The model still sees a meaningful sequence; it just has to do a bit more work.
• Code token counts are tokenizer-dependent. Punctuation, whitespace, and unusual identifiers can create more token boundaries, while common pieces such as def and return are single tokens in cl100k_base. Compare the actual count instead of assuming code always uses more tokens than prose.
• Token counts vary across languages. A tokenizer may split some scripts or languages more finely than English, particularly when it lacks common character sequences. But this is tokenizer-specific: modern multilingual vocabularies can encode many non-Latin strings efficiently. Compare the token count for the text you are using rather than assuming all non-English prompts cost more.
• Emoji and rare Unicode can hit the byte fallback: one token per byte of UTF-8.

What the model actually receives

By the end of the tokenizer’s work, your prompt (a string) has become a flat list of integers, each between 0 and vocab_size - 1. Those integers will be the very first thing the neural network sees. The model has no idea they used to be letters; as far as it knows, the world is just a sequence of integer indices.

Of course, integers alone don’t capture meaning. The token ID for “king” and the ID for “queen” are just two arbitrary numbers; they don’t tell the network that those words are related. The next step is to turn each integer into a much richer object that can carry information about meaning: a vectorvectorAn ordered list of numbers, such as the measurements describing one example or the learned values representing a token.See in glossary →, an ordered list of numbers. A learned vector representing a token is called an embeddingembeddingA dense vector representation of a token (typically d=2k–8k floats). Similar tokens get nearby vectors.See in glossary →.