AI · Python

What an AI actually reads: tokens, explained with code

On this page
  1. A question a six-year-old can answer
  2. Why not letters, and why not words
  3. BPE: the algorithm that builds tokens
  4. A real tokenizer
  5. Four consequences you can check
  6. 1. Counting letters is hard
  7. 2. Numbers aren't seen as numbers
  8. 3. Not every language costs the same
  9. 4. Two texts identical to you, different to the model
  10. What we've covered

A question a six-year-old can answer#

Ask a child how many r's there are in strawberry and they'll count them on their fingers: three. For a while, that same question was one of the easiest ways to trip up a language model, the kind of AI behind chat assistants. Models that could write code or summarise a contract would answer "two" with complete confidence.

It wasn't a lack of intelligence. It was a problem of sight: the model never saw the letters of strawberry. It saw this:

output
['st', 'raw', 'berry']

Three pieces, and none of them is a letter. Those pieces are called tokens, and they are the only thing a language model reads and the only thing it writes. Everything else (what it understands, what it costs to use, how much text fits in it, some of its strangest failures) starts here.

This is the first part of a series on understanding AI from the inside, step by step and without magic: each piece is explained with small code you can run yourself. We start with the model's input: how a text turns into numbers.

In one sentence

A language model doesn't read letters or words: it reads tokens, variable-length pieces of text, each with an ID number. The text is split up first, and from then on the model works only with those numbers.

Why not letters, and why not words#

A language model needs a vocabulary: a closed list of pieces, each with its number. There are two obvious options, and both are bad.

Use letters. The vocabulary would be tiny (a few dozen symbols) and any text could be written. But every sentence would become a very long sequence, and length is exactly what costs a model the most: it has to relate each piece to all the others, so the work grows very quickly with every extra symbol. On top of that, a single letter means nothing; the model would spend much of its effort rebuilding words.

Use words. Sequences would be short and each piece would carry meaning. But the vocabulary would be huge: cat, cats, catty, catlike... and in Spanish, each verb has dozens of forms. And there would always be a word that isn't on the list (a proper name, a typo, a new technical term) with no way for the model to represent it.

LettersWordsTokens (subwords)
Vocabulary sizeDozensMillionsTens or hundreds of thousands
Length of a textVery longShortIn between
Unknown wordsNeverConstantlyNever
Each piece means somethingNoYesAlmost always

The solution almost every model uses today sits in the middle: subwords. Frequent words are a single token; rare ones are split into smaller pieces; and in the worst case, anything can be written with minimal pieces. You get the best of both options. The question is how to decide which pieces make it into the vocabulary.

BPE: the algorithm that builds tokens#

The most widely used method is called BPE (Byte Pair Encoding). It was born in 1994 as a data compression algorithm, published by Philip Gage, and in 2016 a group at the University of Edinburgh adapted it to split words for machine translation. The idea fits in one line:

Start with single letters and, over and over, merge the pair of neighbouring symbols that repeats the most.

Nobody decides by hand that gat or ing should be tokens: they emerge from the statistics of the training text. Whatever repeats most ends up as a piece.

This program implements it from scratch, with no libraries, and trains it on a three-sentence toy corpus. The corpus is in Spanish (gato is cat, perro is dog, come is eats), which makes it easy to see roots and endings appear:

bpe.py
# python3 bpe.py
from collections import Counter

def entrenar(texto: str, fusiones: int):
    # Cada palabra empieza partida en letras sueltas
    palabras = Counter(tuple(p) for p in texto.split())
    reglas = []
    for _ in range(fusiones):
        # Contar qué pareja de símbolos vecinos aparece más
        parejas = Counter()
        for palabra, n in palabras.items():
            for a, b in zip(palabra, palabra[1:]):
                parejas[a, b] += n
        if not parejas:
            break
        (a, b), n = parejas.most_common(1)[0]
        reglas.append((a, b))
        print(f"{len(reglas):>2}. {a!r} + {b!r} -> {a + b!r}  ({n} veces)")
        # Sustituir esa pareja por un símbolo nuevo en todo el corpus
        palabras = Counter({fusionar(p, a, b): n
                            for p, n in palabras.items()})
    return reglas

def fusionar(palabra, a, b):
    salida, i = [], 0
    while i < len(palabra):
        if palabra[i:i + 2] == (a, b):
            salida.append(a + b)
            i += 2
        else:
            salida.append(palabra[i])
            i += 1
    return tuple(salida)

def tokenizar(palabra: str, reglas):
    simbolos = tuple(palabra)
    for a, b in reglas:          # mismas fusiones, mismo orden
        simbolos = fusionar(simbolos, a, b)
    return list(simbolos)

corpus = """el gato come pescado el gato duerme la gata come
los gatos comen pescado las gatas duermen el perro come
los perros comen carne el perro duerme mucho"""

reglas = entrenar(corpus, 10)
for p in ["gatos", "perras", "comemos", "garaje"]:
    print(f"{p:>8} -> {tokenizar(p, reglas)}")
output
$ python3 bpe.py
 1. 'm' + 'e' -> 'me'  (8 veces)
 2. 'e' + 'r' -> 'er'  (6 veces)
 3. 'g' + 'a' -> 'ga'  (5 veces)
 4. 'ga' + 't' -> 'gat'  (5 veces)
 5. 'c' + 'o' -> 'co'  (5 veces)
 6. 'co' + 'me' -> 'come'  (5 veces)
 7. 'e' + 'l' -> 'el'  (4 veces)
 8. 'o' + 's' -> 'os'  (4 veces)
 9. 'c' + 'a' -> 'ca'  (3 veces)
10. 'd' + 'u' -> 'du'  (3 veces)
   gatos -> ['gat', 'os']
  perras -> ['p', 'er', 'r', 'a', 's']
 comemos -> ['come', 'm', 'os']
  garaje -> ['ga', 'r', 'a', 'j', 'e']

The output is worth reading slowly, because everything is in it:

  • The vocabulary builds itself. With ten merges, the algorithm has found the root gat, the verb come and the plural ending os. Nobody taught it grammar; those are simply the sequences that repeat the most.
  • Merges chain together. Rule 4 (ga + t) only exists because rule 3 created ga first. Repeating the process produces longer and longer pieces.
  • There are no unknown words. Perras (female dogs) and comemos (we eat) weren't in the corpus, and they still get tokenised: known pieces (come, os, er) are reused and the rest stays as single letters.
  • Rare costs more. Garaje (garage) looks like nothing in the corpus and ends up as five tokens. Frequent is cheap and rare is expensive; you'll see this rule in every section of this post.
How a real tokenizer differs

The tokenizers of real models don't start from letters but from bytes: the 256 possible values. Since any text (in any language, with emojis or odd characters) can be written as bytes, there is no text that can't be tokenised. They are also trained on huge amounts of text and perform hundreds of thousands of merges, not ten. But the mechanism is exactly this one.

A real tokenizer#

To look at a real one we'll use tiktoken, the open library OpenAI uses to publish its models' tokenizers. It ships, among others, o200k_base, a vocabulary of about 200,000 tokens which, according to the library itself, is used by GPT-4o, GPT-4.1, GPT-5 and the o-series models. The first time you use it, tiktoken downloads the vocabulary; everything in this post was tested with tiktoken 0.14.0.

Not every vendor publishes theirs. Anthropic, for example, doesn't offer its tokenizer for download, but an API service that counts the tokens of a text for a given model. The details vary from vendor to vendor, but the underlying idea is the same.

tokens.py
# pip install tiktoken   ·   python3 tokens.py
import tiktoken

enc = tiktoken.get_encoding("o200k_base")

def ver(texto: str):
    ids = enc.encode(texto)
    trozos = [enc.decode([i]) for i in ids]
    print(f"{texto!r:>12}  n={len(ids)}  {ids}  {trozos}")

for t in ["strawberry", "fresa", "hola", " hola", "Hola", "HOLA"]:
    ver(t)
output
$ python3 tokens.py
'strawberry'  n=3  [302, 1618, 19772]  ['st', 'raw', 'berry']
     'fresa'  n=2  [69, 17351]  ['f', 'resa']
      'hola'  n=1  [122845]  ['hola']
     ' hola'  n=1  [171847]  [' hola']
      'Hola'  n=1  [49864]  ['Hola']
      'HOLA'  n=2  [39, 87254]  ['H', 'OLA']

The column of numbers is what the model actually receives. It doesn't receive "hola": it receives 122845.

And look at the last four lines. To a person, hola (hello), Hola, HOLA and hola with a leading space are the same word. To the model they are different numbers with no relation to each other: 122845 is no more similar to 49864 than to any other number. The space is part of the token ( hola is the form that appears in the middle of a sentence) and capital letters change the whole piece.

So how does a model know they are all the same word? It learns it during training, by seeing them appear in the same contexts millions of times. How a model whose only material is numbers stores that similarity is the subject of the next part of the series.

Four consequences you can check#

1. Counting letters is hard#

Back to strawberry. The model receives [302, 1618, 19772]. To count the r's it would have to know by heart which letters make up token 1618 (raw) and token 19772 (berry). That information isn't in the number: it could only have learned it indirectly, from texts where someone spells those words out.

That's why questions about the shape of words (counting letters, reversing a word, rhymes or anagrams) are harder for a model than for a person, even when questions about their meaning are easy for it. Current models fail much less, especially those that can "think out loud" before answering: they tend to spell the word letter by letter, and each letter becomes its own token. That doesn't change how they see text; they have learned to work around the problem.

2. Numbers aren't seen as numbers#

numeros.py
import tiktoken

enc = tiktoken.get_encoding("o200k_base")
for t in ["2+2=4", "12345+67890", "1000000", "1.000.000"]:
    print(f"{t:>12} -> {[enc.decode([i]) for i in enc.encode(t)]}")
output
       2+2=4 -> ['2', '+', '2', '=', '4']
 12345+67890 -> ['123', '45', '+', '678', '90']
     1000000 -> ['100', '000', '0']
   1.000.000 -> ['1', '.', '000', '.', '000']

This tokenizer groups digits in threes, starting from the left. To the model, a million written without separators is "one hundred, zero zero zero, zero". And in 12345 + 67890 the digits are misaligned: the units of one number fall into a two-digit token (45) and those of the other into another (90). Adding like that is like doing a sum on paper with the numbers in the wrong columns.

That's why, for exact calculations, the right approach is for the model not to compute "in its head" but to use a tool (a calculator or a piece of code) and read the result. We'll see how that works in the part of the series about agents.

3. Not every language costs the same#

Tokens are the unit everything in a language model is measured in: APIs charge per token, the maximum text a model accepts (its context window) is measured in tokens, and so is response speed. So it matters how many tokens your text takes up.

To measure it I took the first two paragraphs of another post on this blog, From Ring 3 to Ring 0, in Spanish and in its English version, and counted the tokens with two OpenAI tokenizers: cl100k_base, used by GPT-4 and GPT-3.5, and o200k_base, used by its current models. They are the paragraphs that open that post in each language, copied as is (without the Markdown formatting) into two text files.

idiomas.py
import tiktoken

es = open("parrafo_es.txt", encoding="utf-8").read()
en = open("parrafo_en.txt", encoding="utf-8").read()

for nombre in ["cl100k_base", "o200k_base"]:
    enc = tiktoken.get_encoding(nombre)
    n_es, n_en = len(enc.encode(es)), len(enc.encode(en))
    print(f"{nombre:<12} vocabulario={enc.n_vocab:>7,}  "
          f"ES={n_es}  EN={n_en}  ES/EN={n_es / n_en:.2f}")
output
cl100k_base  vocabulario=100,277  ES=129  EN=102  ES/EN=1.26
o200k_base   vocabulario=200,019  ES=111  EN=103  ES/EN=1.08

With the older tokenizer, the same content in Spanish took 26% more tokens than in English: it was 26% more expensive and slower to process, and 21% less text fit in the same context window. The reason is BPE: if the training text has much more English, the merges favour English words and Spanish ends up more fragmented. With a vocabulary twice the size there's room for more Spanish pieces, and the gap for these paragraphs drops to 8%.

It's two paragraphs, not a study. But the trend is real and has been measured. A 2023 paper from the University of Oxford tokenised the same translation in around two hundred languages. With the GPT-4 tokenizer (cl100k_base), Spanish needed 55% more tokens than English, and even the cheapest languages, such as Portuguese or German, were around 50% more. At the other extreme, Shan, a language of Myanmar, needed up to 15 times more. My paragraphs come in below that 55% because they are technical text, full of words that are already English: CPU, kernel, Ctrl+S.

And a tokenizer isn't forever: it changes between model generations. Anthropic notes in its documentation that, starting with Claude Opus 4.7, the same text produces about 30% more tokens than on its earlier models, and recommends recounting prompts with the model you plan to use instead of reusing old figures. At the same price per token, that changes what you pay and how much text fits. Note that this compares tokenizers from the same company: it says nothing about whether one vendor is more expensive than another.

4. Two texts identical to you, different to the model#

This is the side that matters for security. Look at these three words:

disfraz.py
import tiktoken

enc = tiktoken.get_encoding("o200k_base")
variantes = {
    "normal": "contraseña",
    "espacio invisible": "contra\u200bseña",   # U+200B
    "'o' cirílica": "c\u043entraseña",          # U+043E
}
for nombre, t in variantes.items():
    trozos = [enc.decode([i]) for i in enc.encode(t)]
    print(f"{nombre:<18} {t}  ->  {trozos}")
output
normal             contraseña  ->  ['contr', 'aseña']
espacio invisible  contra​seña  ->  ['contra', '\u200b', 'se', 'ña']
'o' cirílica       cоntraseña  ->  ['c', 'о', 'n', 'tr', 'aseña']

On screen, all three say contraseña (password). But the second one hides an invisible character (U+200B, a zero-width space) and the third swaps the Latin o for a Cyrillic о, which is drawn the same. To the tokenizer they are three completely different sequences.

The security consequences are direct:

  • A banned-word filter that compares text sees neither variant. If a system blocks the word password by searching for it as is, the invisible character or the Cyrillic letter slips past it, and the text is still readable to a person.
  • The model, on the other hand, often does understand the variant. It has seen enough text with typos and odd characters to rebuild the meaning. The result is an awkward situation: the filter detects nothing and the model may understand the message anyway.
  • It isn't a new trick. It's the same one fake domains use, with look-alike letters, to pose as a bank, and the Unicode Consortium itself documents how to detect it.

The first defence almost everyone thinks of is normalising the text: Unicode defines standard forms (NFC, NFKC) so that equivalent characters are written the same way. But it isn't enough. This program checks it:

defensa.py
import unicodedata

def limpiar(texto: str) -> str:
    texto = unicodedata.normalize("NFKC", texto)
    # Quitar caracteres de formato invisibles (categoría Cf)
    return "".join(c for c in texto
                   if unicodedata.category(c) != "Cf")

def alfabetos(texto: str) -> set[str]:
    # Primera palabra del nombre Unicode: LATIN, CYRILLIC...
    return {unicodedata.name(c).split()[0]
            for c in texto if c.isalpha()}

for t in ["contraseña", "contra​seña", "cоntraseña"]:
    nfkc = unicodedata.normalize("NFKC", t)
    print(f"{t!r:<20} NFKC={nfkc == 'contraseña'!s:<5} "
          f"limpio={limpiar(t) == 'contraseña'!s:<5} "
          f"{sorted(alfabetos(t))}")
output
'contraseña'         NFKC=True  limpio=True  ['LATIN']
'contra\u200bseña'   NFKC=False limpio=True  ['LATIN']
'cоntraseña'         NFKC=False limpio=False ['CYRILLIC', 'LATIN']

Normalisation fixes neither variant: the Cyrillic о is a different letter, not an alternative form of the Latin one, and the invisible space survives. Two more measures are needed:

  • Stripping invisible format characters (Unicode category Cf) fixes the first case. Carefully: some of those characters are legitimate, such as the ones that join compound emojis or the ones scripts like Arabic or Hindi need.
  • Detecting words that mix scripts gives away the second case. A word with both Latin and Cyrillic letters is almost never legitimate. Unicode's security standard (UTS #39) goes further: it includes a table of characters that can be confused with each other, to compare texts by how they look.
Don't trust what you see on screen

If you build something on top of a language model, remember that the model processes tokens, not what shows on screen. Any control that depends on comparing text (banned words, lists of allowed commands, sensitive-data detection) has to clean the text first: normalise it, strip invisible characters and watch for mixed scripts. Even then, a word filter is a weak defence: treat it as one more layer, never the only one.

What we've covered#

  • A language model doesn't read letters or words: it reads tokens, pieces of text with an assigned number. Its input and its output are, literally, lists of numbers.
  • Tokens are built with BPE: start with minimal pieces and keep merging the most frequent pair. Whatever repeats most in the training text ends up as a single token.
  • Frequent is cheap and rare is expensive. That explains why models struggle to count letters and work with numbers, and why some languages take more tokens than others.
  • Two texts identical on screen can be different to the tokenizer, and that has security consequences.

One question remains open: if to the model hola is 122845 and Hola is 49864, how does it come to know they are the same word, or that cat is closer to dog than to contract? The answer is embeddings: the way each number becomes a point in a space where similar words end up close together. That's the next part of the series.

References

  1. Neural Machine Translation of Rare Words with Subword Units (Sennrich, Haddow y Birch, 2016)arXiv · arxiv.org
  2. Language Models are Unsupervised Multitask Learners (Radford et al., 2019)OpenAI · cdn.openai.com
  3. tiktoken: a fast BPE tokeniser for use with OpenAI's modelsGitHub · github.com
  4. Language Model Tokenizers Introduce Unfairness Between Languages (Petrov et al., 2023)arXiv · arxiv.org
  5. tiktoken/model.py: qué codificación usa cada modeloGitHub · github.com
  6. Token counting (Claude API)Anthropic · platform.claude.com
  7. A New Algorithm for Data Compression (Gage, 1994)The C Users Journal · jacobfilipp.com
  8. Unicode Technical Standard #39: Unicode Security MechanismsUnicode Consortium · unicode.org
  9. Unicode Standard Annex #15: Unicode Normalization FormsUnicode Consortium · unicode.org

Share

Keep reading

Rust

Rate limiting an API by plan in Rust: GCRA, Redis and tower-rate-tier

How to limit requests by each user's plan with tower-rate-tier: why GCRA and not a fixed window, requests that cost more, limits shared between instances with Redis, and safe defaults. With an example service and real outputs.

· 11 min

Rust

Using C++ from Rust: how a safe wrapper turns a use-after-free into a compile error

A C++ library used from Rust: what unsafe really is, how to design the C boundary, why a C++ bug stops compiling with the right wrapper, and what happens to exceptions and panics. With real code.

· 17 min

Want to train your team in AI?

I design hands-on AI training for teams and managers, tailored to your sector and the tools you already use, with materials and follow-up afterwards.

Let's talk→

Comments

Commenting requires a GitHub account. Comments live in GitHub Discussions (SoftDryzz/blog-comments) and are moderated: no promotional links.