What are tokens in LLMs? How language models break text into pieces
Tokens are how LLMs see text. Understanding them helps you estimate costs, stay inside context windows, and write better prompts. The 4-chars-per-token rule of thumb is good enough for back-of-the-envelope estimates. For exact counts, use a tokenizer.
A token is the atomic unit of text that a language model processes. It's not quite a word and not quite a character. Common words like 'the' are usually one token each. Longer or less common words like 'tokenization' might be split into two or three tokens. As a rule of thumb, one token is roughly 4 characters of English text, or 100 tokens is roughly 75 words.
By TechCompare · Updated
How this is calculated
Tokens exist because language models don't understand text directly. They understand sequences of numbers. A tokenizer converts text into tokens, and each token maps to a numeric ID in the model's vocabulary. The model processes these IDs, not the raw text. Different model families use different tokenizers with different vocabularies, which is why the same sentence can produce a different token count for GPT-5 vs Claude vs Llama 4. OpenAI's o200k_base tokenizer (used by GPT-5 and GPT-4o) has a vocabulary of 200,000 tokens and is more efficient than older tokenizers like cl100k_base, typically producing fewer tokens for the same input.
Verdict
A token is the unit a language model actually reads, and each one maps to a numeric ID in the model's fixed vocabulary. Common words compress to a single token, while rare or longer ones split into subwords, which is why 'the' and 'tokenization' cost different amounts. The o200k_base tokenizer in GPT-5 and GPT-4o builds a 200,000-entry vocabulary, so identical text produces a different count than it does on Claude or Llama 4.
More Tokens scenarios
Related guides
Frequently asked questions
What is a token in an LLM?
Why do language models use tokens instead of words?
How many words is 1,000 tokens?
Related tools
LLM API Pricing Calculator
Compare API costs across major models (OpenAI, Anthropic, Google) with prompt caching.
Use tool ➜LLM VRAM Calculator
Calculate the VRAM needed to run or fine-tune any LLM at any quantization.
Use tool ➜JSON Formatter
Validate, format, and minify JSON data with syntax highlighting.
Use tool ➜