TechCompare LogoTechCompare
Showing centimeters. Press to switch to inches.Showing inches. Press to switch to centimeters.

AI

Byte Pair Encoding (BPE)

The tokenization algorithm used by GPT, Llama, and many other LLMs. BPE builds a vocabulary by iteratively merging the most frequent adjacent byte pairs in a training corpus, so common sub-words become single tokens and rare words split into multiple tokens. The tokenizer is what determines exact token counts for a given input.

Building a vocabulary by merging

BPE starts small and grows. The recipe is short:

  • Split every word in the training text into single characters. Those characters are the starting vocabulary.
  • Count every adjacent pair of symbols. Say the corpus has "low" five times, "lowest" twice, "newer" six times and "wider" three times. The pair "e" + "r" wins, because it sits inside both "newer" and "wider".
  • Merge that pair into one new symbol, "er", and add it to the vocabulary. "newer" is now n, e, w, er.
  • Recount and merge the next most common pair. Repeat until the vocabulary reaches the size you picked.

The ordered list of merges is the tokenizer. To encode new text, it splits the text into characters and replays the merges in the order they were learned. Sennrich and colleagues borrowed this idea from data compression in 2015 so translation models could handle rare words, and it's now behind most LLM tokenizers.

Why byte level BPE has no unknown tokens

Plain BPE has a gap: a character the training text never contained has no symbol. Byte level BPE closes it by starting from the 256 possible byte values instead of characters. Any text in any language, emoji included, is a string of bytes, so it can always be encoded, even if the worst case is one token per byte.

GPT-2's tokenizer is the standard example. Its vocabulary has 50,257 entries: 256 byte values, 50,000 learned merges and one special end of text token. Hugging Face lists Llama, Gemma and Qwen2 among the models that use BPE as well. OpenAI's tiktoken describes the method as reversible and lossless, so the tokens always decode back to the exact original text.

What it means for your token count

The merges come from the training corpus, so they favor whatever the corpus had plenty of. Common English words become one token. Words from smaller languages, rare names and long numbers fall back to shorter pieces, and in the worst case to single bytes. That's how Petrov and colleagues measured the same text taking up to 15 times as many tokens in one language as in another.

It's also why two models can count the same prompt differently. Each family trains its own merges on its own data and picks its own vocabulary size, and a bigger vocabulary generally covers the same text in fewer tokens. If a bill or a context limit depends on the count, use the tokenizer that matches the model.

Work it out yourselfOpen the LLM Token Counter

Read next

BPE tokenization on real text OpenAI vs Llama tokenizers Token counts across languages

Related terms

LLM tokenThe unit a large language model reads and bills text in. A token is usually a common word, part of a longer word, or a piece of punctuation, averaging about four characters or three-quarters of a word per token in English. Each model family has its own tokenizer, so the same input produces different token counts across GPT, Llama, Claude, and Gemini.UTF-8The most common way to store Unicode text as bytes. It writes each code point as one to four bytes, and plain ASCII text stays one byte per character, so any ASCII file is already valid UTF-8.Context windowThe most text a language model can handle in one request, counted in tokens. The prompt, the conversation so far and the model's reply all share it, so a long input leaves less room for the answer.

Sources

Back to the glossary