// section · 1 article

AI

Applied AI: models, tools and automation.

Articles

AI · Part 1 · 13 min

What an AI actually reads: tokens, explained with code

A language model doesn't read letters or words: it reads tokens. What they are, how BPE builds them (coded from scratch in Python), and why they explain its trouble with letters and numbers, what each language costs and a trick to slip past filters.