The same English text now counts 33% more tokens on newer Claude models: a measured tokenizer ledger
We sent the same 33 fixed texts to the token counters of 13 Claude models, 11 Gemini models and 2 Cohere models, and counted them offline for OpenAI's encodings. From Claude Opus 4.6 to Claude Opus 4.7, English prose needs 33.3% more tokens and technical English 41.0% more, at an unchanged list price; Russian, Chinese and Arabic need about the same. Measured, not claimed, with every raw response archived.
In this article
The headline, measured
We counted the same 33 texts with each model's own token counter. The step from Claude Opus 4.6 to Claude Opus 4.7 is the clearest case: the list price per token did not change, but English prose needed 33.3% more tokens, technical English 41.0% more, JavaScript 23.7% more, and Russian, Chinese and Arabic about 0%.
The same 558-word English text counted by each model's own token counter. List price per token did not change in this step.
The same step changed token counts by about nothing for all three non-Latin scripts we measured.
13 Claude models, 11 Gemini models, 2 Cohere models and 2 OpenAI encodings, on 33 fixed texts.
Every one is in measurements.json; every raw API response is archived with a SHA-256 and read time.
Bars: tokens the newer model needs for every 100 tokens the older model needs, by text type, Claude Opus 4.6 to Claude Opus 4.7. A bar above 100 is inflation.
Every step inside each Claude model line
Mean of the 11 text types, a 95% bootstrap interval over the types, the English prose result, the change in list price, and what the two together do to the cost of English input. Steps inside one tokenizer generation show exactly zero.
| Step within a model line | Tokens, mean of 11 text types | 95% interval | English prose | List price, input | Effective cost of English input |
|---|---|---|---|---|---|
| Claude Opus 4.5 to Claude Opus 4.6 same tokenizer | 0.0% | n/a | 0.0% | 0.0% | 0.0% |
| Claude Opus 4.6 to Claude Opus 4.7 | +16.0% | +8.2% to +24.6% | +33.3% | 0.0% | +33.3% |
| Claude Opus 4.7 to Claude Opus 4.8 | -0.0% | -0.1% to 0.0% | 0.0% | 0.0% | 0.0% |
| Claude Opus 4.8 to Claude Opus 5 same tokenizer | 0.0% | n/a | 0.0% | 0.0% | 0.0% |
| Claude Opus 5 to Claude Opus 5.5 same tokenizer | 0.0% | n/a | 0.0% | -20.0% | -20.0% |
| Claude Sonnet 4.6 to Claude Sonnet 5 | +16.0% | +8.2% to +24.5% | +33.3% | -33.3% | -11.2% |
| Claude Sonnet 5 to Claude Sonnet 5.5 same tokenizer | 0.0% | n/a | 0.0% | 0.0% | 0.0% |
| Claude Haiku 4.5 to Claude Haiku 5.5 | +16.0% | +8.2% to +24.5% | +33.3% | n/a | n/a |
| Claude Fable 5 to Claude Fable 5.1 same tokenizer | 0.0% | n/a | 0.0% | 0.0% | 0.0% |
What it does to your bill
Tokens per 1,000 English words and the cost of those tokens at the provider-stated list prices archived by AI Price Watch. A model whose price was not parsed from the archived table shows n/a rather than a guess. Output cost assumes the answer is the same number of words as the input and is labelled as an approximation.
| Model | Tokenizer generation | Tokens per 1,000 English words | List input, $ per million tokens | Input cost per 1,000 words | List output, $ per million tokens | Output cost per 1,000 words, if the answer is the same length |
|---|---|---|---|---|---|---|
| Claude Haiku 5.5 | 3 | 1,606 | n/a | n/a | n/a | n/a |
| Claude Sonnet 5.5 | 3 | 1,606 | $2 | $0.00320 | $10 | $0.0161 |
| Claude Opus 5.5 | 3 | 1,606 | $4 | $0.00640 | $20 | $0.0321 |
| Claude Fable 5.1 | 3 | 1,606 | $10 | $0.0161 | $50 | $0.0803 |
| Claude Opus 5 | 3 | 1,606 | $5 | $0.00800 | $25 | $0.0401 |
| Claude Sonnet 5 | 3 | 1,606 | $2 | $0.00320 | $10 | $0.0161 |
| Claude Fable 5 | 3 | 1,606 | $10 | $0.0161 | $50 | $0.0803 |
| Claude Opus 4.8 | 3 | 1,606 | $5 | $0.00800 | $25 | $0.0401 |
| Claude Opus 4.7 | 2 | 1,606 | $5 | $0.00800 | $25 | $0.0401 |
| Claude Sonnet 4.6 | 1 | 1,203 | $3 | $0.00360 | $15 | $0.0180 |
| Claude Opus 4.6 | 1 | 1,203 | $5 | $0.00600 | $25 | $0.0301 |
| Claude Opus 4.5 | 1 | 1,203 | $5 | $0.00600 | $25 | $0.0301 |
| Claude Haiku 4.5 | 1 | 1,203 | $1 | $0.00120 | $5 | $0.00600 |
Uses the measured tokens per 1,000 words of the longest English passage and the provider-stated list prices in this ledger. It ignores caching, batch discounts and reasoning tokens.
The same text across vendors
One 558-word English passage, counted by each tokenizer. Different vendors split the same words into different numbers of tokens, which is why a price per token cannot be compared across vendors without it. OpenAI figures are offline counts of the text alone with the tiktoken library.
| Vendor | Tokenizer | Tokens for the same 558-word text | Against the shortest count |
|---|---|---|---|
| Cohere | command-a-plus-05-2026 | 627 | 0.0% |
| gemini-2.5-flash-lite and 9 other Gemini models with identical counts | 637 | +1.6% | |
| OpenAI (offline tiktoken) | o200k_base | 639 | +1.9% |
| OpenAI (offline tiktoken) | cl100k_base | 644 | +2.7% |
| Cohere | command-a-03-2025 | 649 | +3.5% |
| gemini-4-argon | 664 | +5.9% | |
| Anthropic | Claude tokenizer generation 1 | 671 | +7.0% |
| Anthropic | Claude tokenizer generation 2 | 896 | +42.9% |
| Anthropic | Claude tokenizer generation 3 | 896 | +42.9% |
10 of the 11 Gemini models we could count (from 2.5 to 3.8) returned identical counts on every text, so those steps show no tokenizer inflation; gemini-4-argon counted the same English passage differently (see the table). Cohere's newer model counted -13.2% against its predecessor, and OpenAI's newer encoding -11.9% against the older one: the direction is not always up.
Which Claude models share a tokenizer
Models are grouped by identical counts on all 33 texts, not by name.
| Tokenizer generation | Models whose counts are identical on all 33 texts |
|---|---|
| 1 | Claude Haiku 4.5, Claude Opus 4.5, Claude Opus 4.6, Claude Sonnet 4.6 |
| 2 | Claude Opus 4.7 |
| 3 | Claude Opus 4.8, Claude Fable 5, Claude Sonnet 5, Claude Opus 5, Claude Fable 5.1, Claude Opus 5.5, Claude Sonnet 5.5, Claude Haiku 5.5 |
Opus 4.7 and the later models differ by one token on two text types (Chinese and URLs with emoji); the 4.7 request overhead also differs. Details are in data.json.
The files
| File and hash | What it holds | Size |
|---|---|---|
data.jsonsha256 5cd258429057 | All results: tokenizer generations, ratios with intervals, effective price per 1,000 words, method and limits | 73 KB |
measurements.jsonsha256 b1086c4b2bdb | Every individual measurement (1,425 rows) | 252 KB |
ratios.csvsha256 ed76b44df69d | Every transition by text type | 15 KB |
METHOD.mdsha256 066cfe21c55a | The method, committed and hashed before any measurement | 4.9 KB |
METHOD.sha256sha256 5a999f1874fb | The SHA-256 of the method and of the corpus manifest, recorded before measuring | 163 B |
README.mdsha256 f60b2a31edba | Plain-language summary and limits | 3.1 KB |
corpus/MANIFEST.jsonsha256 0bd02f52f2b4 | The 33 fixed texts with a SHA-256 for each | 7.8 KB |
EVIDENCE-MANIFEST.jsonsha256 a1c5c25ffaec | URL, read time and SHA-256 of every archived raw response | 2.9 KB |
share-card.svgsha256 1f1f1b15d66a | A 1200 by 630 share image generated from the data | 1.2 KB |
How to cite it
Couey, V. W. (2026). AI Tokenizer Inflation Ledger 2026: the same text, counted by every model version we can reach. Nesyona. https://nesyona.com/research/ai-tokenizer-inflation-2026/ (edition 28 October 2026). https://doi.org/10.5281/zenodo.23730116. To cite every version: https://doi.org/10.5281/zenodo.23730115.
Cite the edition date and the file hash for a specific figure.
How it is built, and what it does not cover
The method was written and hashed before any measurement (METHOD.md). The corpus is 11 text types in 3 lengths, all written for this study or generated from a seeded generator; the three non-Latin types are assembled from 16 distinct sentences each.
- Token counts are not bills. Billing reflects the provider's own accounting; the counts are the providers' estimates and may differ slightly.
- Tokenizer behaviour depends on content. A different corpus gives different ratios; we report every text type separately and never only the mean.
- The Anthropic count includes a small fixed request overhead that differs by model generation; we subtract a one-character baseline, which understates every count by one token (the baseline character). The effect is at most 1.2 percent on the smallest item and at most 0.16 percent on the longest items we measured, and it moves a ratio between two models by less than that because it affects numerator and denominator alike.
- The interval is a bootstrap over 11 text types we chose, not over all possible text; it describes our corpus, not the world.
- Gemini model ordering follows the version numbers in the model names because the models list carries no release dates. gemini-2.5-pro was not measured: the provider returned HTTP 404 ("no longer available to new users").
- Cohere was measured for one pair with one repeat each to respect the trial key limit of 1,000 calls a month.
- OpenAI counts are offline tiktoken counts of the text alone. Only models that tiktoken itself maps to an encoding are placed in it; 33 model ids from the models list are unverified and appear in no ratio.
- Counting tokens says nothing about quality or about how many tokens a model needs to answer.
No provider was asked to comment before publication. A provider or reader who finds a wrong figure can tell us; we correct it and date the correction.
FAQ
What is tokenizer inflation?
A tokenizer splits text into the units a model is billed and limited by. When a provider ships a new tokenizer, the same text can become more tokens, so the same words cost more at an unchanged price per token. This ledger measures that with the providers' own token counters instead of repeating claims about it.
How much more do newer Claude models count?
Between Claude Opus 4.6 and 4.7 the same English prose needed 33.3% more tokens and technical English 41.0% more. JavaScript needed 23.7% more, Python 30.0%, JSON 9.9%, and Russian, Chinese and Arabic about 0%. Across all 11 text types the mean was +16.0% (95% interval +8.2% to +24.6%). The models from Opus 4.8 onward in this ledger count the same as Opus 4.7 except for one token on two text types.
Does this match what Anthropic says?
Anthropic's documentation says the newer tokenizer produces approximately 30 percent more tokens and that the exact increase depends on content. Our mixed-corpus mean is lower than the stated 30 percent (the stated value is above our interval); by text type the stated figure is matched or exceeded for prose-en, technical-en, code-py and not matched for code-js, json, md-table, text-ru, text-zh, text-ar, numbers-math, urls-emoji. The statement is therefore right for English and code-heavy text and does not describe non-Latin scripts.
Is this the same as a price increase?
Not by itself. List prices per token did not change between Opus 4.6 and 4.7, so the cost of the same English text rose by the token increase. Some steps changed both: Sonnet 4.6 to Sonnet 5 cut the list input price -33.3% while English text needed 33.3% more tokens, which is an effective -11.2% for English input. Token counts are not bills; billing is the provider's own accounting.
Why do some models count identically?
Models that share a tokenizer return identical counts on every text. The ledger groups them as one tokenizer generation only where all 33 counts are identical, instead of assuming it from the model names.
How do I check my own text?
Send your text to the provider's token-counting endpoint once per model you are comparing (Anthropic documents its count endpoint as free to use, subject to rate limits) and compare the counts. Recount against the model you plan to use rather than reusing a count from an older model; Anthropic's documentation says the same.
Can I reuse it?
Yes. It is published under CC BY 4.0: reuse is free with attribution to Vincent Wesley Couey and a link to this page.
Does it have a DOI?
Yes. Version 1 is https://doi.org/10.5281/zenodo.23730116. The DOI for all versions is https://doi.org/10.5281/zenodo.23730115.