Byte Pair Encoding: how LLM tokenizers learn to split text into tokens

How this is calculated

BPE training works like this: start with every unique character in the training data as a token. Count all adjacent token pairs. Merge the most frequent pair into a new token. Repeat thousands of times until the desired vocabulary size is reached (e.g. 100K or 200K tokens). The resulting vocabulary handles any text, even words never seen during training, because any word can be broken into known subword tokens. This is why misspellings, code, and invented words still produce reasonable tokenization. BPE's main competitor is Unigram (used by SentencePiece in some configurations), which starts with a large vocabulary and prunes it. BPE is more common in the latest generation of models (GPT-5, Llama 4) because it handles whitespace and code more naturally.

Verdict

The algorithm starts with every unique character as a token, counts adjacent pairs across the corpus, and merges the most frequent pair into a new token, repeating until the vocabulary hits its target size. That bottom-up training is why a word never seen during production can still tokenize cleanly into known subwords, which covers misspellings, invented terms, and code. Unigram, the SentencePiece alternative, prunes a large vocabulary instead and shows up less in the current generation of models.