Your language costs more tokens to say the same thing. That is a property of the tokenizer, not of your language — and it can be fixed without retraining the model.
Type below in your language. This page tokenizes it in your browser and shows what you are charged now, how much of the context window you actually receive, and what a Parity vocabulary pack gives back — with the certified bound on how far the model's behaviour may move.
Token cost of the same 24 sentences under each tokenizer, from the fertility atlas (measured on OPUS-100, not on these 24 sentences).
Measuring the imbalance is the easy part, and it is not the point. Parity repairs it: it adds language-specific tokens to an already-trained open-weight model, synthesises their embeddings from the model's own internal representations, and ships a certificate bounding how far behaviour can move.
| Component | What it does |
|---|---|
| 1. Miner | finds frequent base-token n-grams worth a vocabulary slot |
| 2. Synthesis | subspace Gauss-Newton least squares matching the residual-stream trajectory — no continued pretraining, every weight frozen |
| 3. Certificate | conformal tail bound + empirical-Bernstein + a deterministic Lipschitz bound; candidates over tolerance are not adopted |
| 4. Selection | submodular max-coverage under the embedding budget, over certified candidates only |
| 5. Serving | one model instance, many tokenizer views, shared prefix cache |
English stays exactly English. Packs only append embedding rows and a base-view request cannot see or emit a pack token, so masked-softmax over the base subset equals the original softmax — an identity, not an empirical finding.
pip install git+https://github.com/NagaYu/parity
parity build --model Qwen/Qwen2.5-0.5B-Instruct --lang ja --budget 8000On SmolLM2-135M with Japanese, 96 candidates covering 30% of the corpus's tokens were
synthesised and certified. At a 0.35-nat drift tolerance, 2 were adopted
— か？ and れる, both short and highly predictable. The rest were
refused: 83 for next-token drift, 7 for total variation, 4 for firing off-context.
That is a negative result, and it is the point of having a certificate. A method without
one would have shipped all 96 tokens and reported a 30% saving. The candidates with the
largest raw saving are single high-frequency particles (た, は),
and they are the worst possible merges — maximal contextual variability, so no single
embedding reproduces them everywhere. Full numbers, including the solver ablation that
cut median drift 5x, are in the
repository README.
Applies to open-weight models and the providers serving them — Parity needs write access to the embedding matrix. Does not apply to closed APIs: you cannot attach a pack to a model you can only reach through someone else's endpoint. If you are a user of such an API, this page can measure what you are being charged; only the provider can fix it.
A high ratio above is a property of a tokenizer — an artefact fitted to a corpus that under-represented most of the world's writing systems. It is not a property of any language, and nothing here should be read as saying otherwise. See docs/framing.md. To contribute a pack for your language, see docs/contributing-a-pack.md.