How a word becomes a number
Language models do everything with numbers. Before a model can do anything with a word, the word has to become a list of numbers — a vector. Vectors, matrices and embeddings sound mathematical, but the core idea is simple. I'll follow one word the whole way through: cat.
Everything orange tracks cat. Hover, drag and toggle the diagrams — they're live. The numbers are illustrative.
1 · Computers can't read words
A model does maths. Words aren't maths. So step one is turning every word into numbers — the right numbers.
2 · A vector is a point in space
A vector is just an ordered list of numbers. Two numbers make a point on a flat map; the arrow from the origin is the word's location.
Real models don't use 2 numbers. GPT‑2 small uses 768. We can't draw 768 axes, so picture this 2D map as a flattened shadow of a 768‑dimensional space.
3 · Stack vectors into a grid
Stack many vectors and you get a matrix — a grid of numbers. This grid is the embedding table: one row for every word the model knows.
GPT‑2 small's embedding table is about 50,257 rows × 768 columns — one row per token it knows.
4 · It's a lookup, not a calculation
This is the single most‑missed idea. The word's ID is just a row number. Getting its vector isn't a calculation — it's a lookup. Go to that row, grab the 768 numbers. That's the embedding.
0.94 0.27 0.63
0.08 0.55 0.41 …
5 · Meaning lives in geometry
Here's the magic: similar meanings end up near each other. cat and kitten are neighbours; cat and helicopter are across town. Distance is difference in meaning. Hover any word.
6 · Arithmetic on meaning
It's not just clusters — directions mean things too. The step from man to woman is the same step from king to queen. Drag the orange arrow onto king.
Drag the orange arrow onto king →
This is the famous example, and the effect is real — but cleaner in the textbook than in practice. The tidy answer partly leans on excluding the three input words themselves, and it shows up more sharply in classic word-vector models than in a raw GPT-2 table.
7 · Nobody programmed this
No human labelled cat as an animal. The positions are learned — they emerge automatically as the model reads huge amounts of text and predicts the next word.
8 · The door into the model
That vector is the word's entry point. Everything the network does next — attention, reasoning, prediction — operates on these number‑coordinates, never on the letters c‑a‑t.
0.94 0.27 …
A note for the curious: these per‑token embeddings aren't the same object as the sentence or document embeddings used for search and RAG — same idea, different scale.