AI · Part 1 · 13 min
What an AI actually reads: tokens, explained with code
A language model doesn't read letters or words: it reads tokens. What they are, how BPE builds them (coded from scratch in Python), and why they explain its trouble with letters and numbers, what each language costs and a trick to slip past filters.