How a word becomes a number

Scroll to follow one word — cat — from letters a model can’t read to the coordinates it actually thinks in. Vectors, matrices, and embeddings, no maths required.

A computer can’t read words. Only numbers.

Models do math. Words aren’t math. So step one is turning every word into numbers — the right numbers.

A vector is a list of numbers — a point in space.

A vector is just an ordered list of numbers. Two numbers = a point on a flat map. The arrow is the word’s location.

Real models don’t use 2 numbers. GPT‑2 uses 768. We can’t draw 768 axes — so picture this 2D map as a flattened shadow of a 768‑dimensional space.

Stack the vectors and you get a grid.

Stack many vectors and you get a matrix — a grid of numbers. This grid is the embedding table: one row for every word the model knows.

GPT‑2’s embedding table is about 50,257 rows × 768 columns — one row per token it knows.

Finding a word’s vector is a lookup.

The word’s ID is just a row number. Getting its vector isn’t a calculation — it’s a lookup. Go to that row, grab the 768 numbers. That’s the embedding.

Similar meanings land near each other.

cat and kitten are neighbours; cat and helicopter are across town. Distance = difference in meaning. Hover any word to see its coordinates.

You can do arithmetic on meaning.

Drag the orange arrow onto king →

The step from man to woman is the same step from king to queen. Directions encode relationships.

These positions are learned, not labelled.

No human labelled cat as an animal. The clusters emerge automatically as the model reads huge amounts of text and predicts the next word.

The vector is the word’s door into the model.

Everything the network does next — attention, reasoning, prediction — operates on these number‑coordinates, never on the letters c‑a‑t.

A note for the curious: these per‑token embeddings aren’t the same object as the sentence or document embeddings used for search and RAG — same idea, different scale.