// series · ongoing
AI from the inside: from tokens to agents
How a language model works, piece by piece and with code you can run: tokens, embeddings, attention, text generation and agents.
Parts of the series
AI · Part 1 · 13 min
What an AI actually reads: tokens, explained with code
A language model doesn't read letters or words: it reads tokens. What they are, how BPE builds them (coded from scratch in Python), and why they explain its trouble with letters and numbers, what each language costs and a trick to slip past filters.
coming soon
Part 2
Embeddings: how a number comes to mean something
coming soon
Part 3
Attention: how the model decides what to look at
coming soon
Part 4
Generating text: probabilities, temperature and hallucinations
coming soon
Part 5