10.3Tokenization

What the model actually sees. A language model never reads letters or words. Before it sees anything, your text is cut into tokens from a fixed vocabulary and each token is replaced by a number. Everything odd about how models handle spelling, arithmetic and other languages starts here.

Runs in your browser · your text and tokenizer are saved in the link

1Type
Type or paste anything. The text is split by a real tokenizer running in your browser, the open source gpt-tokenizer port of OpenAI's vocabularies. Each cell is one token; the number under it is the token ID, which is all the model receives.
Tokenizer
Used by GPT-4o, GPT-4.1, o1, o3 and GPT-5
Text to tokenizeTokenizers split text into pieces. How many r's are in strawberry? The year 2026 has 365 days.
Show token IDs
Loading the tokenizer's vocabulary…
2Check
The same word, three ways. Written with a space before it, as it usually appears mid-sentence, it can be a single token: one number with no letters in it at all.
Loading the tokenizer's vocabulary…
Note
Asked how many r's are in strawberry, the model is not looking at letters. It gets token IDs, and the spelling of each one has to have been learned separately from text that happened to spell it out. Counting letters is easy for you and indirect for it.
3Count
Long numbers are cut into chunks of digits. Turn the dial to make the number longer and compare the tokenizer's chunks with the way you group digits.
Digits: 7
Loading the tokenizer's vocabulary…
Note
The tokenizer chunks digits from the left, while place value runs from the right, so the chunks rarely line up with thousands and millions. Add one digit and every chunk keeps its digits but changes what it is worth: the leading token “123” is 1,230,000 in 1234567 and 12,300,000 in 12345678. A model doing arithmetic has to work place value out from the IDs.
4Compare
To the model, a space and a capital letter are not small changes. Each version below is a different token, or several, with nothing in the IDs to say they are the same word. Try your own.
A word
Loading the tokenizer's vocabulary…

5Measure

The same sentence in five languages: Article 1 of the Universal Declaration of Human Rights, translated by people. Each count below was made by the real tokenizer when this page was built, and the figure after it is the multiple of the English count.

Tokens for Article 1 of the Universal Declaration of Human Rights, by language and tokenizer
Language	Characters	o200k_base tokens	cl100k_base tokens
English	170	33 1.0×	33 1.0×
Nepali	163	47 1.4×	169 5.1×
Hindi	189	54 1.6×	182 5.5×
Spanish	171	38 1.2×	44 1.3×
Chinese (simplified)	43	35 1.1×	50 1.5×

Show the five texts

English: All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood.
Nepali: सबै व्यक्ति हरू जन्मजात स्वतन्त्र हुन ती सबैको समान अधिकार र महत्व छ। निजहरूमा विचार शक्ति र सद्धिचार भएकोले निजहरूले आपसमा भातृत्वको भावना बाट व्यवहार गर्नु पर्छ।
Hindi: सभी मनुष्यों को गौरव और अधिकारों के मामले में जन्मजात स्वतन्त्रता और समानता प्राप्त है । उन्हें बुद्धि और अन्तरात्मा की देन प्राप्त है और परस्पर उन्हें भाईचारे के भाव से बर्ताव करना चाहिए ।
Spanish: Todos los seres humanos nacen libres e iguales en dignidad y derechos y, dotados como están de razón y conciencia, deben comportarse fraternalmente los unos con los otros.
Chinese (simplified): 人人生而自由，在尊严和权利上一律平等。他们赋有理性和良心，并应以兄弟关系的精神相对待。

Each model family has its own tokenizer, so other vendors' counts differ. One article is a small sample.

Note

On cl100k_base, the tokenizer behind GPT-4, Article 1 takes 33 tokens in English and 169 in Nepali: 5.1× as many for the same meaning. On o200k_base, its successor, the gap shrinks to 1.4× (47 against 33).

Tokens are what you pay for and what fills the context window, so a Nepali speaker on the older vocabulary paid several times more to say the same thing, and could fit several times less into one request. Petrov and colleagues found gaps of up to 15 times between languages across tokenizers in 2023. Newer vocabularies are fairer, not fair.

6Train
Byte-pair encoding, as Rico Sennrich, Barry Haddow and Alexandra Birch used it for translation in 2016. Each word starts as single letters with an end marker (·). Every merge finds the pair of neighbours that occurs most often and joins it everywhere. Step through the merges and watch the vocabulary grow and the text shrink.
Training text
low low low low low lower lower newest newest newest newest newest newest widest widest widest
Merges: 0 of 15
Vocabulary
11
Tokens in the text
95
Last merge
none
1. l
2. o
3. w
4. ·
5. l
6. o
7. w
8. e
9. r
10. ·
11. n
12. e
13. w
14. e
15. s
16. t
17. ·
18. w
19. i
20. d
21. e
22. s
23. t
24. ·
Note
This is the whole idea behind the vocabulary in step 1: start from single characters (bytes, in the real thing), and keep gluing together the pair that appears most often. Frequent words end up as one token; rare words, and languages that were rare in the training text, stay in small pieces. That is where step 5's gap comes from.
7Go
Tokens are the unit of cost and of context. Read on.

10.3Tokenization

10.3.1Sources