Tokenization
The process of converting text into a sequence of integers that an LLM can process, where each integer represents a token - typically a subword fragment, full word, or punctuation character.
LLMs don't process raw text. Before inference, text is converted into a sequence of integers using a tokenizer. Each integer maps to a "token" - typically a subword fragment, full word, or punctuation character, depending on frequency in the training corpus.
How tokenizers work
Modern LLMs use Byte Pair Encoding (BPE), SentencePiece, or other subword tokenization algorithms trained on large text corpora. Common words like "the", "is", and "of" become single tokens. Rare words and technical terms split into fragments: "tokenization" might become ["token", "ization"]. Code identifiers, URLs, and non-English text are often less efficient, consuming more tokens per character.
Why token count matters
Most LLM APIs charge by token consumption, not by characters or words. Context windows are measured in tokens: current flagship models such as Claude Opus 5.5, GPT-6.1 Sol and Gemini 4 Argon accept about 1 million tokens, and Gemini 4 Argon counts input, thinking and output against one shared 1,048,576-token window. Output length limits are specified in tokens. A rough rule for English text: 1 token is about 0.75 words or 4 characters, but the ratio varies by tokenizer. Code and non-English text typically consume more tokens per equivalent content.
Counting tokens
OpenAI provides tiktoken, an open-source tokenizer library for its GPT models. Anthropic offers a token counting endpoint (count_tokens) for Claude models, and the Gemini API has a countTokens method. Count with the same provider and model you will call, because tokenizers change between model generations: Anthropic says Claude Haiku 5.5, which uses the tokenizer introduced with Claude Opus 4.7, produces roughly 30 percent more tokens than Claude Haiku 4.5 for the same text. For reference: a 10-page PDF is roughly 5,000 to 8,000 tokens; a full novel runs around 100,000 to 150,000 tokens.
Token efficiency and model behavior
Some tasks inherently consume more tokens. Chain-of-thought prompting generates intermediate reasoning tokens before producing the final answer. Models designed for extended reasoning allocate additional tokens to internal processing, trading token consumption for improved accuracy on complex problems.
Related terms
Models relevant to Tokenization
Claude Haiku 5.5
Anthropic's fastest, lowest-cost current model at $0.10 per million input tokens with a 1M context window
View model →GPT-6.1 Sol
Near-Astra intelligence for a fifth of the price
View model →Gemini 4 Argon
Google's frontier model for real-world coding, enterprise knowledge work, and cybersecurity defense
View model →