What a tokenizer hands the model
A model never sees letters or words. A tokenizer cuts your text into pieces first, then swaps each piece for an ID number, and those IDs are all the model gets. A piece can be a whole word like "the", a chunk like "ization", a punctuation mark or even a single byte. Common words usually get one token. Rare words, names and long numbers get split into several.
Tokens are short. OpenAI's tiktoken library puts the average at about 4 bytes, which is roughly 4 characters of plain English. As a rule of thumb, 100 tokens come to about 75 words, so a 128,000 token window holds around 96,000 words. Code, other languages and text full of numbers or symbols tend to split into more tokens per word, so treat the rule as a starting point.
Why the same text gets different counts
Every model family ships its own tokenizer with its own vocabulary. One paragraph can count differently from one model to the next, so a count from one provider's tokenizer is only an estimate for another's. Three things move it most:
- Language. A tokenizer trained mostly on English spends its vocabulary on English. Petrov and colleagues found the same text can take up to 15 times as many tokens in one language as in another.
- Code and numbers. Indentation, symbols and long digit strings tend to break into short pieces.
- Tokenizer version. A newer model can use a different vocabulary than the one before it, so an old count may not carry over.
Run your real prompts through the tokenizer of the model you'll call. When you can only estimate, keep some headroom. The Token Counter does the counting side by side.
Tokens as a budget
Hosted APIs bill in tokens. Providers price input tokens (your prompt, the chat history and any attached files) and output tokens (the reply) separately, so a short question with a long answer and a long document with a short answer can land on very different bills. The LLM Pricing Calculator takes both counts and the current rates, which change often enough that this page doesn't list them.
Tokens also fill the context window, and the reply comes out of the same budget. Put a 100,000 token document into a 128,000 token window and 28K tokens remain for your instructions, the model's answer and any follow-up. A 2,000 token system prompt sent with each of 10,000 requests adds up to 20,000,000 input tokens before anyone asks a question.