How a transformer
actually thinks.
Eight stages. The whole loop, one screen at a time.
Text becomes tokens.
The model does not read words. It reads pieces (engineers call them tokens), drawn from a fixed set of about a hundred thousand, rather like Scrabble tiles that can be joined to build any word, including one it has never seen.
Why it mattersThis is why a model can misspell a name or miscount the letters in a word. It never saw the word whole.
11 pieces for 39 characters and 4 words.
Ordinary words the model has seen often are one piece each; longer or rarer ones are built from parts.
Genuine splits, computed in advance with a widely used tokeniser (OpenAI’s o200k vocabulary). Other models split differently, but the pattern is the same. A faint dot marks a piece that begins with a space.
Each token is a point in space.
Every piece is given a position on a map of meaning. Pieces used in similar ways end up close together, so “judge” sits near “barrister” and far from “horse”. Nobody drew the map; it formed as the model read. A dimension is simply one direction on that map. An ordinary map has two. This one has thousands, which lets it keep many kinds of similarity apart at once.
Why it mattersThe model works by nearness of meaning, not by definition. Similar is not the same, and that is one root of confusing one authority with another that sounds alike.
Every token looks at every previous token.
Before deciding what a piece means here, the model checks which earlier pieces it should pay attention to. In “the barrister opened her brief”, “her” matters because of “barrister”. Each row in the grid is one piece; the bright cells are the earlier pieces it is drawing on. Nobody programmed those links. They were learned from patterns in text.
| Reading | The | barrister | opened | her | brief | because | she | needed | notes |
|---|---|---|---|---|---|---|---|---|---|
| The | 1.00 | not yet seen | not yet seen | not yet seen | not yet seen | not yet seen | not yet seen | not yet seen | not yet seen |
| barrister | 0.40 | 0.60 | not yet seen | not yet seen | not yet seen | not yet seen | not yet seen | not yet seen | not yet seen |
| opened | 0.20 | 0.55 | 0.25 | not yet seen | not yet seen | not yet seen | not yet seen | not yet seen | not yet seen |
| her | 0.05 | 0.75 | 0.10 | 0.10 | not yet seen | not yet seen | not yet seen | not yet seen | not yet seen |
| brief | 0.03 | 0.30 | 0.40 | 0.10 | 0.20 | not yet seen | not yet seen | not yet seen | not yet seen |
| because | 0.03 | 0.20 | 0.40 | 0.03 | 0.20 | 0.20 | not yet seen | not yet seen | not yet seen |
| she | 0.03 | 0.70 | 0.03 | 0.10 | 0.03 | 0.10 | 0.10 | not yet seen | not yet seen |
| needed | 0.03 | 0.15 | 0.03 | 0.03 | 0.30 | 0.03 | 0.40 | 0.15 | not yet seen |
| notes | 0.03 | 0.10 | 0.03 | 0.03 | 0.45 | 0.03 | 0.03 | 0.30 | 0.15 |
Reading her, the model draws most on barrister (0.75), opened (0.10) and her itself (0.10).
It does this many times in parallel.
It runs that lookup several times at once, each copy hunting for something different. One tracks who is doing the action, one what belongs to whom, one grammar. These copies are called heads: several angles on the same sentence, then combined.
Then each token is processed independently.
With context gathered, each piece is worked over on its own through a vast bank of learned associations, roughly “a barrister who has opened a brief is about to read papers”. This is where the model’s general knowledge lives. Not a database it consults, but millions of tuned connections. The widen-then-narrow shape below is just how that bank is wired.
Why it mattersNothing here is looked up or checked against a source. The knowledge was fixed when training finished.
Where most of the model’s knowledge is stored.
All of that, stacked 80 times.
One pass of stages 03 to 05 is a layer. The model runs the same layer design again and again, each pass reading what the previous ones wrote and adding to it, the way a pupil rereads a brief and understands more each time.
By layer 1: Pieces have been joined back into words, and each word has a rough kind: barrister and brief are things, opened and needed are actions. Nothing yet about how they relate.
Illustrative: real layers do not divide their work this neatly, but the direction of travel, surface to meaning to prediction, is what the research shows.
Doing this many times lets understanding build up gradually. Early layers catch simple things — grammar, which word goes with which. Later layers handle abstract meaning — tone, intent, reasoning. Less like 80 separate machines, more like 80 rounds of revision on the same draft.
80 layers is roughly the depth of a model like Llama 70B. Different models go deeper or shallower — the idea is what scales, not the figure.
Output is a probability over every word.
The model does not produce a word. It produces a score for every possible next piece: “papers” 21%, “first” 14%, and so on down a list a hundred thousand long. It ranks; it does not decide. Something outside the model then picks from the list, usually with a little randomness thrown in.
Why it mattersThis is why the same question gets different answers on different days, and why a fluent sentence is only a chain of high-scoring guesses. A confident tone is not a sign that anything was checked.
The top words share the picks. This is roughly how models are usually run.
Illustrative figures. Real models score a vocabulary of around a hundred thousand pieces, which makes the tail heavier still. Lower temperature makes an answer more predictable, not more true.
Pick one. Append. Repeat.
Take the chosen piece, add it to the end, and run the whole machine again from the top. Press the button to watch it happen.
Why it mattersThere is no plan and no going back. Every sentence you read from a model was built one piece at a time, each chosen only from what came before it.
Press Play. The list refreshes for every piece.
That is the whole machine.
No memory between steps. No checking against a source. A machine that has read more than any person could and remembers none of it as fact, guessing the next piece of text extremely well. Everything else in this series follows from that. Next: what the machine can actually see.
Next: The Context Window →