Tokenization
How text becomes tokens, BPE, and counting tokens
Skills taught: llms.tokenization
What Are Tokens?
LLMs do not operate on characters or words. They use sub-word tokenization (Byte-Pair Encoding, or BPE) which breaks text into variable-length chunks from a fixed vocabulary. English averages ~0.75 words per token, so 1000 words ≈ 1333 tokens. API pricing is charged per token.
- BPE (Byte-Pair Encoding)
- A tokenization algorithm that builds a vocabulary by iteratively merging the most frequent character pairs. 'tokenization' → ['token', 'ization'] or ['token', 'iz', 'ation'] depending on the vocabulary.
- Vocabulary
- The fixed set of all possible tokens a model knows. GPT-4 uses ~100,257 tokens. Each token is an integer ID; the model never sees raw text — only these integers.
import tiktoken # pip install tiktoken
enc = tiktoken.encoding_for_model("gpt-4") # cl100k_base
# Count tokens for a string
text = "Hello, world!"
tokens = enc.encode(text)
print(f"Tokens: {tokens}") # [9906, 11, 1917, 0]
print(f"Count: {len(tokens)}") # 4
# See the actual text pieces
for token_id in tokens:
print(f" {token_id:6d} -> '{enc.decode([token_id])}'")
# Token-heavy example: code vs prose
code = 'def fibonacci(n): return n if n <= 1 else fibonacci(n-1) + fibonacci(n-2)'
print(f"\nCode: {len(enc.encode(code))} tokens for {len(code)} chars")
prose = "The quick brown fox jumps over the lazy dog near the river"
print(f"Prose: {len(enc.encode(prose))} tokens for {len(prose)} chars")
Counting Tokens via the Anthropic API
import anthropic
client = anthropic.Anthropic()
# Count tokens BEFORE making a full API call (saves money on large prompts)
count_response = client.messages.count_tokens(
model="claude-opus-4-5",
system="You are a helpful assistant that answers concisely.",
messages=[
{"role": "user", "content": "Explain the difference between TCP and UDP in simple terms."}
]
)
print(f"Input tokens: {count_response.input_tokens}") # e.g., 35
# After a real call, usage is included in the response
response = client.messages.create(
model="claude-haiku-5-5",
max_tokens=200,
messages=[{"role": "user", "content": "What is 2+2?"}]
)
print(f"In: {response.usage.input_tokens}, Out: {response.usage.output_tokens}")
Token Counts by Language and Content Type
- English prose: ~0.75 words/token (most efficient)
- Python/JS code: ~1 token per 3-4 characters (more expensive)
- Chinese/Japanese: 1-2 chars per token (more expensive than English)
- Emojis: 1-3 tokens each
- Numbers: each digit is often its own token (123 = 3 tokens)
- Whitespace and newlines: each takes tokens — compact prompts cost less
Tip: Code uses more tokens per character than English. A Python function of 100 characters may use 50-80 tokens. When building RAG systems, account for this when setting chunk sizes.
Warning: Tokenizers differ between providers: Claude, GPT-4, and Llama use different tokenizers. Never assume token counts transfer between providers. Always use the provider's own tokenizer or counting API.
import tiktoken
def estimate_cost_gpt4(text: str, is_input: bool = True) -> float:
"""Rough cost estimate for GPT-4o at Nov 2026 pricing."""
enc = tiktoken.encoding_for_model("gpt-4")
n_tokens = len(enc.encode(text))
# GPT-4o: $5/1M input tokens, $15/1M output tokens
price_per_million = 5.0 if is_input else 15.0
cost = (n_tokens / 1_000_000) * price_per_million
return cost, n_tokens
system_prompt = "You are a JSON-only API. Respond with valid JSON."
user_msg = "Extract the name and email from: John Smith <john@acme.com>"
cost, tokens = estimate_cost_gpt4(system_prompt + user_msg)
print(f"Input: {tokens} tokens, estimated cost: ${cost:.5f}")
Frequently Asked Questions
Does tokenization differ between models?
Yes. Claude, GPT-4, and Llama use different tokenizers. Never assume token counts transfer between providers. Always use the provider's own counting API (e.g., Anthropic's count_tokens endpoint).
How do I estimate the cost of a large batch job?
Use tiktoken to count tokens for a sample of your data, multiply by your dataset size, then multiply by the model's per-token price. The count_tokens API endpoint also works without incurring generation costs.
What is the max tokens parameter for?
max_tokens sets the maximum number of tokens the model is ALLOWED to generate in its response. It does not control input length. Setting it too low truncates responses mid-sentence.