Andrej Karpathy

BY OTHERS◆Friday, October 9, 2026

by Andrej Karpathy

A room for Andrej Karpathy as a teacher: one forward pass through a GPT, laid out as a 1960s textbook in ten chapters, each with a clip of him explaining it and a hands-on exhibit running his own code; his experiments in between (char-rnn on Shakespeare, Pong from pixels, the ImageNet race, badmephisto), his newest ideas, and his three rules for becoming an expert.

tags: ai, llm, transformers, teaching, karpathy, neural-networks, gpt

author: Andrej Karpathy
filed: Friday, October 9, 2026
revised: October 9, 2026
words: 457
reading: ~3 min

A forward pass through a GPT, in ten chaptersAfter the lectures of Andrej Karpathy

logits = model(idx) # one forward pass

Forward Pass

He taught a generation to build one from scratch. Type anything below and follow its last token down through a GPT, chapter by chapter, with Andrej explaining each step in his own words.

Fig. 0.1One attention head, as a picture. Each row is a token; it may look back at the tokens before it, never ahead. The bottom row is the last token, the one we follow.

Fig. 0.2Your sentence, as the model reads it. Type, or try one of his.

In [1]

The·unreasonable·effectiveness·of·

of·token 15 of 15, id 323= of (283) + · (32), merge 68 of 384

34 characters, 34 bytes → 15 tokens, 2.3 bytes a token. Vocabulary 640: 256 bytes and 384 merges learned from his words.

The token we followof·id 323, position 14, down into the residual stream

Everything a model knows, it read first. So we begin where it begins: with the words.

Front matterBefore chapter 1

The training data

A GPT knows only what it has read. Here is what this book read: 146 lines of his, from 2012 to 2026.

Collected from his blog posts, talks and posts on X, each with its source and date, and numbered as they sit in the data file, No. 000 to No. 145.

The tokenizer's 384 merges were learned from 157 lines of his quotes and clip excerpts.

Before a model can learn anything, it needs something to read, and what it reads is all it will ever know. The small machines in this book read his words: the tokenizer at the top of the page learned its pieces from these lines, and the little model at the end of the pass predicts from them. So the book opens with its data.

Herman Hollerith's punched cards counted the 1890 United States census; cards fed computers for the better part of a century after.

D.1 Draw one

Start with a single line. Each one is a small sample of how he thinks. Press for another.

Card 1 of 297 characters · 45 tokens

It was probably around the 32nd layer of the 400th token in the sequence that I became conscious.
Andrej Karpathy, Short Story on AI: Forward Pass (opens in a new tab), 27 March 2021LLMs

146 lines in the deck

Fig. D.1One line, two ways.Above, as a 1964 keypunch would have punched it: one column per character, each character a pattern of up to three holes (the IBM 029 card code; it had no small letters), 80 columns to a card. Below, as you read it. The tokenizer of Fig. 0.2 cuts the same line into 45 tokens.

“A base model is a token simulator: an internet text token simulator.”Deep Dive into LLMs like ChatGPT, 2025

“I owe my Grand Awareness solely to the objective of decoding the next token.”Short Story on AI: Forward Pass, 2021

D.2 Read it as a model does

You take in a line at a glance. A model reads it one token at a time, and at every token it must guess the next one before it is told. Those guesses are the whole of training: wrong guesses nudge the weights, millions of times over. Press to read, and watch his lines become examples.

Lines read: lines read 0
Tokens read: tokens read 0
Epoch: epoch 1

000A "fast and furious" approach to training neural networks does not work.26
001The qualities that correlate most strongly to success in deep learning are patience and attention to detail.44
002Neural net training is a leaky abstraction.14
003Neural net training fails silently.13
004Become one with the data.10
005Don't be a hero. Don't try to be clever and reach for an exotic architecture on day one.42
006Your misconfigured neural net will throw exceptions only if you're lucky. Most of the time it will train but silently work a bit worse.56
007Adding more data is pretty much the only guaranteed way to monotonically improve the performance of a well configured neural network.54
008The best way to regularize a model is to add more real training data.26

Speed

Fig. D.2The reader.All 146 lines on one paper tape, numbered as they sit in the file; the small figure at the end of each row is its length in tokens. The reading head is the ruled row between the orange pointers; the tape moves on a row each time the head finishes a line. Reading the whole tape once is one epoch.

Under the head: No. 0000 of 26 tokens

start
A
·
"
fast
·and·
f
ur
i
ou
s
"
·a
p
pro
a
ch
·to
·training·
neural·net
work
s·
do
es·
not·
work
.

No. 000 is under the head. Nothing read yet.

Fig. D.3What one line teaches.A model never sees a line whole. At each token it is shown everything before it, the context, and asked for the next one. So a line of 26 tokens is 26 small exams, one per token: every token in the data is the right answer exactly once an epoch. The start mark lets even the first token be a guess (GPT-2 marks it with a special token, <|endoftext|>).

Try this 0 of 4

to do: Step one token by hand, and watch the context grow.
to do: Press to read.
to do: Turn on Shuffle: training reads its data in a random order.
to do: Read every line once: one epoch. (Skim helps.)

“Become one with the data.”A Recipe for Training Neural Networks, 2019

“When you sort your dataset descending by loss you are guaranteed to find something unexpected, strange and helpful.”Post on X, 2020

D.3 Become one with the data

The first step of his recipe for training a neural network does not touch the network at all. Look at the data: scan it, count it, sort it, search it, until you know its shape and its oddities. Here is all of it.

Search the lines

Pick a tick to read its line. Ink ticks match your search and topic; the faint ones do not.

Fig. D.4The data at a glance.Every line is a tick, placed by its date in its topic's row; the figures at the right count the ticks that match. Read it like a map of his years: the PhD and the blog, Tesla, then the long run of language models.

Sort

Showing 10 of 146 lines

032
Everything became much clearer when I started writing code.
TeachingHacker's guide to Neural Networks (opens in a new tab)2012
27 tokens
033
If this makes sense, you understand backpropagation.
TeachingHacker's guide to Neural Networks (opens in a new tab)2012
21 tokens
034
A cost function is an expression that measures how bad your classifier is.
TeachingHacker's guide to Neural Networks (opens in a new tab)2012
33 tokens
019
These are unconstrained internet images: a jungle of viewpoints, lighting conditions, and variations.
ResearchWhat I learned from competing against a ConvNet on ImageNet (opens in a new tab)2 September 2014
41 tokens
020
Human accuracy is not a point. It lives on a tradeoff curve.
ResearchWhat I learned from competing against a ConvNet on ImageNet (opens in a new tab)2 September 2014
29 tokens
021
It is clear that humans will soon only be able to outperform state of the art models by significant effort.
ResearchWhat I learned from competing against a ConvNet on ImageNet (opens in a new tab)2 September 2014
42 tokens
088
The real top unsolved problems in Computer Science: Video Conferencing, Presentation, Printing.
HumourPost on X4 October 2014
45 tokens
089
Jeff Dean: “I like your ConvNets in Javascript.” Me: “Thank you. I like your map reduce.”
HumourPost on X11 December 2014
51 tokens
010
There’s something magical about Recurrent Neural Networks.
LLMsThe Unreasonable Effectiveness of Recurrent Neural Networks (opens in a new tab)21 May 2015
27 tokens
011
If training vanilla neural nets is optimization over functions, training recurrent nets is optimization over programs.
ResearchThe Unreasonable Effectiveness of Recurrent Neural Networks (opens in a new tab)21 May 2015
36 tokens

Fig. D.5The whole list.Every line with its source and date. The bar at the right is its length in tokens, so sorting by length is sorting by how much the model has to read.

Try this 0 of 4

to do: Sort by Shortest and find his shortest line.
to do: Choose Humour.
to do: Search for ghost.
to do: Pick a tick on the timeline, or step along it with the arrow keys.

“a lossy compression of the internet”Intro to Large Language Models, 2023

D.4 Ten times more

These lines come to 6,140 tokens. A real GPT reads billions, then trillions. Press, and every press is ten times more text.

tokens 6,140

tokens, these lines

To read it yourself: 18 minutes
Presses: 0 of 10

Where we start: every line on this page.

Fig. D.6Powers of ten.After Charles and Ray Eames. The scale is logarithmic: each step along it is ten times more text. Reading time assumes 250 words a minute, day and night, and about three quarters of a word per token. A model reads a pretraining set in days or weeks, on thousands of GPUs, and keeps what it learned in its weights.

What a GPT keeps of all that reading is its weights. The next ten chapters are one pass through them, carrying the last token of your sentence.

Lines from his blog posts (karpathy.github.io, karpathy.bearblog.dev), talks and posts on X, each linked to its source. Tokens by minbpe's BasicTokenizer, after Andrej Karpathy (MIT), with merges trained on his words.

Long before neural networks, a nineteen-year-old in Toronto was already teaching strangers on YouTube, one patient step at a time.

InterludeToronto, 2006 · before chapter 1

The teacher before the AI

badmephisto

Before he taught neural networks, he taught a generation of speedcubers to solve the Rubik's cube, and he taught it the way he teaches everything: understand it, and you will not need to memorise it.

Numbered from zero, as Python counts: this comes before chapter 1. Printed in six colours, a plate section in a two-colour book.

Sources: Wikipedia, “Andrej Karpathy”; his own site, badmephisto.com (the FAQ, the Guide and the F2L page).

In 2006, as a computer science student at the University of Toronto, Andrej Karpathy began posting Rubik's cube tutorials to YouTube under the name badmephisto. They are patient, methodical and clear, and they explain why each move works, so that you can find the moves yourself.

Speedcubers learned from them, among them Feliks Zemdegs, who went on to set world records; the channel has passed nine million views. Here is the cube, taught his way: first what it is made of, then why nobody can memorise it, then the one idea that lets you solve it anyway.

The letters are Singmaster notation, after David Singmaster's Notes on Rubik's “Magic Cube” (1979).

On a real cube R is whatever face is on your right. Here each letter stays printed on its face, so you can turn the cube round and still find it.

0.1 Turn it

A cube has six faces, and every move turns one of them. Turn a few. The tape records your moves in the notation every cuber uses, and the net below shows all six faces at once.

Turn a face

Keyboard: R L U D F B, with Shift for ′. Backspace undoes.

stickers off their faces 0

stickers off their own face, of 48 that can move

Solved. Every sticker is on its own face.

Tape 0no moves yet

Fig. 0.1A cube to turn.Each letter names a face: R right, L left, U up, D down, F front, B back. R alone is a quarter turn clockwise, as you look straight at that face; R′ (“R prime”) turns it back. Drag the cube, or use the Turn buttons, to see the other sides.

Fig. 0.2The same cube, unfolded.All 54 stickers at once, as if the cube were a cardboard box cut open and laid flat. The letters sit on the centres.

Try this 0 of 4

to do: Turn R four times. Where are you now?
to do: Do R, then U. Now undo them by hand, last move first: U′, then R′.
to do: Press Scramble, and watch the net fill with colour.
to do: Press Rewind. Then read what it says.

This is why you cannot solve a cube one face at a time: every piece of a face also belongs to one or two other faces, and it drags their stickers along.

0.2 Pieces, not stickers

The first thing a solver learns to see is that the cube is not 54 stickers. It is 26 small plastic pieces, and they come in three kinds.

Show me the

piece	how many	stickers	total
Centres	6	× 1	6
Edges	12	× 2	24
Corners	8	× 3	24
The cube	26		54

Twenty-six pieces you can touch. Inside is a core with six arms, and each centre is fixed to the end of one.

Try this 0 of 2

to do: Show only the centres, then press Mix it up. Do they move?
to do: Mix it, then look at the edges, then the corners. Where do they go?

Fig. 0.3Pieces, not stickers.A beginner sees 54 stickers to put right. A solver sees pieces: the centres stay put, so only 20 pieces really move, 12 edges and 8 corners, and each must go to its place the right way round.

8! is “eight factorial”: 8 × 7 × 6 × 5 × 4 × 3 × 2 × 1.

Every one of these positions can be solved in 20 turns or fewer. Tomas Rokicki, Herbert Kociemba, Morley Davidson and John Dethridge proved it in 2010, with about 35 years of computer time donated by Google.

0.3 Too many to remember

So why not learn the right answer for every position? Count them. The counter below multiplies what each kind of piece is free to do.

The counter starts at 1: the solved cube. Press the first four keys in any order to multiply in what each kind of piece is free to do; the fifth, last.

Fig. 0.4How many ways can a cube be mixed?Multiply what each kind of piece is free to do, and divide out what the mechanism forbids. Every number on the counter is computed in your browser, exactly.

Nobody solves the cube position by position. You learn a few ideas that work on every position, including the ones you have never seen. A neural network is in the same spot: there are far more sentences than it can ever be shown, so it has to learn the ideas, not the table.

Why it must: there are only so many positions, so repeating a sequence has to revisit one sooner or later. Every turn can be undone, so the first position revisited is the one you started from.

Cubers call R U R′ U′ the sexy move. Count the pieces it disturbs.

0.4 Every sequence comes home

Here is a strange fact you can check with your own hands. Pick a sequence of turns and repeat it, over and over. However mixed the cube gets, it always comes back to solved.

The sequence

goes 0

goes of R U R′ U′

turns in one go: 4
pieces one go disturbs: 7 of 20

Guess first: how many goes until it is solved again?

Try this 0 of 3

to do: R U R′ U′: guess, then repeat until home.
to do: Now R U. Same guess?
to do: Write your own. The longest any sequence can take is 1,260 goes.

Fig. 0.5Every sequence comes home.Repeat any sequence of turns and the cube comes back to solved, sooner or later. How soon is very hard to guess: two sequences of nearly the same moves can differ a hundredfold.

A sequence that changes only a few pieces and leaves the rest alone is a solver's tool. The last figure shows how to build one on purpose.

On his F2L page he counts 41 basic cases for a single pair. You can learn 41 algorithms, or one idea.

His FAQ, to the many who wrote that finding the pairs was slow: “You are not special at all, sorry.” With practice, he promises, you learn to ignore every piece with yellow on it. Here they are printed grey.

0.5 Out of the way, in, back

Speedcubers solve in layers: a cross on the bottom, then the First Two Layers (F2L), one corner and its edge at a time, then the last layer. Every F2L case has an algorithm you could memorise. His advice was not to.

You should absolutely learn F2L intuitively.
badmephisto, the F2L page of badmephisto.com

Intuitively means you know what each move is for. Step through the first case with the guide, then do the second on your own.

The case

Turn

solved pieces knocked out 0

solved pieces knocked out of place

the pair: not yet
moves: 0

The bottom two layers are solved except one slot at the front right. Its two pieces, marked with a dot, are up on top: the corner (white, green, orange) at the front, its edge (green, orange) at the back. The grey pieces belong to the last layer: ignore them.

Try this 0 of 2

to do: Case 1: step through it with the guide, and watch the count go 0, 3, 3, 0.
to do: Case 2: solve it yourself with R, U and their primes.

Fig. 0.6Out of the way, in, back.One pair of the First Two Layers (F2L). Only R and U are needed. The count is every solved piece of the bottom two layers that is not in its place: watch it rise when you break things on purpose, and fall when you put them back.

Break something on purpose, do your work while it is out of the way, then put it back. Mathematicians call these moves conjugates and commutators; solvers use them everywhere. It is how a handful of ideas replaces thousands of memorised moves.

“teach/summarize everything you learn in your own words”. Andrej Karpathy, “How to become expert at thing”, 2020.

Twenty years later he teaches neural networks the same way. Not a list of moves to copy, but what each piece is for, built from scratch until you could have invented it yourself. The chapters that follow are those lessons, one block of a GPT at a time.

The cube in Figs. 0.1 to 0.6 is a real model of all 26 pieces, turned in your browser; the counts and the orders are computed exactly. Quotations from badmephisto.com and from his post of November 2020.

Now the pass. Your sentence goes in at the top and comes out as a guess at the next token. Ten stops on the way down.

Chapter 1Input

Tokenizer

Text becomes a short list of integers.

Fig. 1.1The tokenizer sits outside the network. It cuts text into chunks it learned from data and gives each chunk an integer id.

After Andrej Karpathy, minbpe (MIT) and Let's build the GPT Tokenizer. The merges are the real algorithm, running in your browser; the GPT-2 ids are read from GPT-2's own tokenizer.

A neural network cannot read. It takes in numbers, so before anything else your text has to become a list of integers. That is the tokenizer's job. It is a separate machine with its own training and its own data, built before the network and then frozen. Your sentence at the top of the page has already been through one, and a surprising number of a language model's odd habits begin right here.

UTF-8 was designed in 1992 by Ken Thompson and Rob Pike, sketched, as the story goes, on a diner placemat.

1.1 Text is already numbers

Every character already has a number. Unicode gives one to each of more than 150,000 characters and keeps adding more: too many, and too changeable, to be a vocabulary. So go one level down. UTF-8 writes each character as one to four bytes, and a byte is just an integer from 0 to 255. Any text, in any language, becomes a list of small numbers.

Set in type

characters: characters 33
bytes: bytes 33

Reading33 characters, 33 bytes. Plain English: one byte for every character.

>>> list("The unreasonable effectiveness of".encode("utf-8"))
[84, 104, 101, 32, 117, 110, 114, 101, 97, 115, 111, 110, 97, 98, 108, 101, 32, 101, 102, 102, 101, 99, 116, 105, 118, 101, 110, 101, 115, 115, 32, 111, 102]

Fig. 1.2The type case.Each character with its Unicode number above and its UTF-8 bytes below. A byte is a number from 0 to 255; English letters take one, accented letters two, Georgian three, an emoji four. The orange sorts are the ones that take more than one.

We could stop here and hand the network raw bytes: a vocabulary of just 256. But the lists would be long, and a GPT can look back over only so many tokens at once (GPT-2: 1,024). Better to give the common chunks of text a number each.

Byte pair encoding began as a compression trick (Philip Gage, 1994). Sennrich, Haddow and Birch brought it to language in 2016; GPT-2 ran it on bytes in 2019.

get_stats counts overlapping pairs, so in aaa the pair (a, a) is counted twice; merge works left to right and can replace it only once.

1.2 Merge the commonest pair

The fix is a small algorithm called byte pair encoding. Start from the bytes. Count every pair of neighbours. Take the commonest pair, give it a brand new number (the first is 256) and replace it everywhere. Then do it again. Every merge makes the vocabulary one bigger and the text a little shorter.

Andrej explainsThe byte pair encoding merge loop, step by step

For twenty merges we're going to find the most commonly occurring pair, mint a new token integer for it starting at 256, and replace all the occurrences of that pair with the newly minted token. We record that this pair of integers merged into this new integer.

Andrej Karpathy, Let's build the GPT Tokenizer, 37:23 to 37:51

Here is his minbpe, running. Start with the tiny example from its README, then feed it his words, or your own.

Train on

Try this 0 of 4

to do: On aaabdaaabac, make three merges. You should be left with 258 100 258 97 99, as in minbpe's README.
to do: A merge of a merge: watch a minted token (an id of 256 or more) join a new pair.
to do: Switch to his words and make ten merges. Which pieces of English come first?
to do: Try your own prompt, and run it until nothing repeats.

ReadingNext: 'a' + 'a' is the commonest pair, counted 4 times. Merged, it becomes token 256, 'aa', and the text gets 2 tokens shorter.

tokens in the text: tokens in the text 11
vocabulary: vocabulary size 256
bytes per token: bytes per token 1.00

The text is now 11 tokens: 97 97 97 98 100 97 97 97 98 97 99.

Pairs, counted

a+a×4
a+b×2
b+d×1
d+a×1
b+a×1

merges, newest first

None yet. Every token is a single byte.

for i in range(num_merges):           # i = 0
    stats = get_stats(ids)            # count every pair
    pair = max(stats, key=stats.get)  # the commonest
    idx = 256 + i                     # mint a new token
    ids = merge(ids, pair, idx)       # replace it everywhere
    merges[pair] = idx                # and write it down

Fig. 1.3The merge machine.Above, the text as a tape of tokens, each with its text and its id. The bracket marks the pair the next press will merge; orange tokens were minted by a merge, and the solid one is the newest. On the right, the pairs counted and the merges written down: that list of merges is the whole of a trained tokenizer.

The list of merges is the whole tokenizer. To encode new text, replay the merges in the order they were learned; to decode, glue the bytes back together.

GPT-2: 256 bytes, 50,000 merges and one special token, <|endoftext|>, for 50,257 in all. GPT-4's tokenizer has about 100,000.

Chapter 2 is the table those ids look up; Chapter 9, the score each token gets at the end.

1.3 How big a vocabulary?

Every merge is a trade. The text gets shorter, so more of it fits in the model's view and each pass covers more ground. But every token needs its own row in the embedding table and its own score at the very end, and a rare token turns up too seldom in training to be learned well. The lever lets this page's merges in one at a time.

Tokenize

vocabulary: vocabulary size 640
tokens: tokens 15

Pull the lever: merges in the vocabulary

ReadingWith 640 tokens in the vocabulary, the text is 15 tokens, down from 34 bytes: 2.3 characters a token.

Fig. 1.4The vocabulary lever.A real tokenizer, trained with minbpe on his own words for this page: 384 merges, a vocabulary of 640. The lever lets in only the first merges; the curve is how long the text is at every setting. Orange tokens are merges, white ones single bytes.

Now try the Georgian. This little tokenizer learned its merges from his English, and not one of them is Georgian. Real tokenizers carry the same bias, more mildly: a language they saw less of costs more tokens for the same meaning, and fills the context sooner.

In 2024 “How many r's are in strawberry?” became the famous stumper: chatbot after chatbot answered two.

The egg is his example, from the tokenizer lecture.

1.4 Why it cannot count the r’s in strawberry

You read strawberry letter by letter. The model never does. Inside a sentence GPT-2 receives it as a single integer, 41236, and the r it is asked about is another integer again. To count, it would have to have learned which letters are packed inside each of its 50,257 tokens.

Andrej explainsWhy spelling and arithmetic are hard for a tokenized model

If you've noticed any issues with large language models not being able to do spelling tasks very easily, that's usually due to tokenization. Simple string processing can be difficult for the model to perform natively. Sometimes LLMs are bad at simple arithmetic, and that also can be traced to tokenization.

Andrej Karpathy, Let's build the GPT Tokenizer, 4:54 to 5:20

Word

Try this 0 of 3

to do: Lower the shutter (switch the letters off) and count the r's in strawberry.
to do: Look at egg four ways. Four different numbers.
to do: Find his name, as GPT-2 spells it.

opening a textst301raw1831berry83963 tokens

inside a sentence·strawberry41236one token

the questionHow2437·many867·r374's338·are389·in287·strawberry41236?308 tokens

Letters shown. opening a text: st 301, raw 1831, berry 8396. inside a sentence: strawberry 41236.

How many r's are in strawberry?

Answer with the shutter up, then again with it down.

Fig. 1.5The shutter.Words as GPT-2 receives them: its own tokenizer, 50,257 tokens, ids read from it. A leading space belongs to the token (printed as a dot). Lower the shutter and you are left with what the network gets: integers, and nothing about the letters inside them.

LLaMA (2023) splits every number into single digits.

GPT-4's tokenizer cuts any run of digits into pieces of at most three.

1.5 Why sums are awkward

Numbers suffer the same way. Column addition needs the ones under the ones and the tens under the tens. GPT-2's merges were learned from ordinary text, where 1986 and 2024 are common and 677 is not, so it cuts each number wherever its merges happen to fall.

Cut the numbers

Try this 0 of 3

to do: Pull for three more sums. Watch where the cuts fall.
to do: Cut them one digit a token, as LLaMA does.
to do: Keep pulling until you reach 999 + 1: three carries, in three single tokens.

What the model is actually handed: one line

7 tokens. No columns, no ones place: just ids in a row, each standing for a chunk of digits.

ReadingGPT-2 reads 127 as one token, 677 as '6' and '77', and must answer 804 as '8' and '04'. The cuts fall in different places in each row, so the digits it must add together sit in tokens of different sizes.

Fig. 1.6The adding machine.A sum laid out the way you learned it, ones under ones, with each token boxed and its GPT-2 id beneath. You add from the right, column by column; GPT-2 is handed chunks cut wherever its merges happened to fall.

“If you can't possibly do the arithmetic in your head, you'd be looking for a calculator, and that's exactly what the LLM does too.”
Andrej Karpathy, How I use LLMs, 2025

Described in February 2023 by Jessica Rumbelow and Matthew Watkins, who found a whole cluster of such tokens.

SolidGoldMagikarp is a Reddit username, from r/counting, where people count upward together one reply at a time. Its neighbours in GPT-2's vocabulary include ‘ TheNitromeFan’ (42090) and ‘ petertodd’ (37444).

1.6 SolidGoldMagikarp

The strangest consequence of all. Early in 2023 people found that a handful of odd strings, ‘ SolidGoldMagikarp’ the most famous, sent GPT-3 and ChatGPT completely off the rails.

Andrej explainsSolidGoldMagikarp: a glitch token derails the model

If you had asked GPT earlier about SolidGoldMagikarp and what it is, you would see the LLM go totally crazy, and it would start going off about a completely unrelated tangent topic.

Andrej Karpathy, Let's build the GPT Tokenizer, 5:31 to 5:42

His explanation: the tokenizer and the model learned from different text. The tokenizer's text held a great deal of Reddit, where one user's name appeared so often that it was merged into a single token. The model's text did not have those threads. So the token exists, with its own row in the embedding table, and that row is never trained: like memory that was allocated and never written.

step 000 of 120

idtokenits row of numbersupdates

262·the0

14143·counting0

13530·effectiveness0

41236·strawberry0

43453·SolidGoldMagikarp0

The tokenizer read the counting threads, so ' SolidGoldMagikarp' became one token, 43453. The model's text has none of them. Press to train.

Try this 0 of 3

to do: Train as it was: the threads in the tokenizer's text only.
to do: Put them in the model's text too, and train again.
to do: Take them out of the tokenizer's text.

Fig. 1.7The switchboard.A toy of his explanation. Each token owns one row of the embedding table, and a row is only updated when its token turns up in the training text. Training pulls the rows it meets into shape; a row it never meets keeps the small random numbers it started with. The ids are GPT-2's; the text and the sixteen numbers a row are stand-ins (GPT-2 has 768).

“Tokenization is at the heart of a lot of weirdness in large language models. Do not brush it off.”
Andrej Karpathy, Let's build the GPT Tokenizer, 2024

Exhibits after Andrej Karpathy, minbpe (MIT) and the lecture Let's build the GPT Tokenizer (2024). The tokenizer in Fig. 1.4 was trained with minbpe on his words for this page. The ids in Figs. 1.5 to 1.7 are GPT-2's, read from its own tokenizer; in Fig. 1.7 the training text and the rows are stand-ins.

Chapter 2Layer 0

Embedding

Each id looks up a vector, and learns where it stands.

Fig. 2.1Two lookup tables. The row for the token and the row for its position are added into one vector: the start of the residual stream.

After Andrej Karpathy, makemore Part 2: MLP and Let's build GPT (MIT). The table of letters is real, and trains in your browser when you ask it to.

A token id is only a name tag. Token 302 is not more than token 301 in any way a network could use, and arithmetic on name tags is nonsense. So the first thing a GPT does is trade each id for a row of numbers it can compute with, learned from data, and add a second row that says where the token stands.

In GPT-2 the token table has 50,257 rows of 768 numbers: 38,597,376 numbers, about a third of the whole model.

nanoGPT uses the same table again at the very end, to turn vectors back into a score for every token. This is called weight tying.

2.1 A row for every token

Start small, as he does in makemore: 27 characters, the letters and a full stop that marks the end of a word, and a table C with one row of two numbers for each. To embed a letter, take its row.

Andrej explainsThe token embedding table as a row lookup

In the constructor we are creating a token embedding table. When we pass idx, every single integer in our input refers to this embedding table and plucks out a row corresponding to its index: so 24 goes in and plucks out the 24th row, and 43 plucks out the 43rd row.

Andrej Karpathy, Let's build GPT: from scratch, in code, spelled out., 23:10 to 23:50

Press a key, or type one. Watch which column survives the multiplication.

C['e']=first number −1.25second number 0.09

Row 5 of 27. The other 26 columns were multiplied by 0; only the column for ‘e’ survives.

C = torch.randn((27, 2))
# the row for 'e', looked up
C[5]
# the same row, by a multiplication
F.one_hot(torch.tensor(5), num_classes=27).float() @ C

Fig. 2.2One row, plucked out.The table C has one row of two numbers for each of makemore's 27 characters, drawn here on its side: one column per letter. Multiplying a one-hot vector by C picks out exactly one row, so a lookup is a matrix multiplication, and C can be trained like any other weight.

A lookup and a multiplication by a one-hot vector give the same row. So the table is just another matrix of weights, and training can reach it: the gradient flows back into exactly the rows that were used.

makemore Part 2 follows the neural language model of Bengio and colleagues (2003). He trains it on 32,033 names; here it reads his sentences instead.

Two numbers a letter is very little, chosen so it can be drawn. He soon widens it; GPT-2 gives each token 768.

2.2 Meaning, learned

At first the rows are random, so the letters land anywhere. Now let a small network use them to guess the next letter of his sentences, and let its mistakes push the rows around. The table in Fig. 2.2 is the same table: it changes as this one trains.

steps: training steps 0
loss: loss 3.30

A model that knows nothing scores 3.30, the log of 27.

Nearest to ‘e’pnzp, n, z

Random numbers: the letters are scattered anyhow, and the model guesses blind.

Try this 0 of 4

to do: Press a key on the typewriter in Fig. 2.2 and find that letter here.
to do: Press to train, and watch the loss fall from 3.30.
to do: When it stops, turn on Mark the vowels.
to do: Start over and train again. The picture changes; the grouping does not.

# 32 examples of three letters, each looked up in C
emb = C[X]  # (32, 3, 2)
h = torch.tanh(emb.view(-1, 6) @ W1 + b1)
logits = h @ W2 + b2  # 27 scores
loss = F.cross_entropy(logits, Y)

Fig. 2.3The same table, drawn as a map.Each letter is placed at its row of C: the first number across, the second up. The network reads three letters and guesses the fourth; training moves the rows of C along with every other weight. The letters are drawn from 2,459 words of his own sentences on this page, read the way makemore reads names.

Nobody told it what a vowel is. Letters that can stand in for one another end up close together, because that helps it guess what comes next. In a GPT the same thing happens to tokens, with 768 numbers instead of two.

Strictly, the causal mask of chapter 3 leaks a little order: a token can count how many came before it, and models given no positions at all still learn some (Haviv and colleagues, 2022). GPT-2 is simply told.

2.3 Where am I?

A row from the token table says what a token is, never where it is. That matters because of what comes next: attention, the subject of chapter 3, takes in its inputs as a set.

Andrej explainsWhy tokens need a position embedding at all

Notice that there is no notion of space: attention simply acts over a set of vectors, and by default these nodes have no idea where they are positioned. That's why we need to encode them positionally, to give them information anchored to a specific position so that they know where they are.

Andrej Karpathy, Let's build GPT: from scratch, in code, spelled out., 1:12:51 to 1:13:09

See it with two sentences made of the same five words.

thet = 012%
dogt = 118%
bitt = 21%
thet = 312%
mant = 458%

thet = 012%
mant = 158%
bitt = 21%
thet = 312%
dogt = 418%

What one attention head gathers

difference: average difference 0.00

Identical. Without positions the head is handed the same bag of vectors either way: it cannot tell who bit whom.

Try this 0 of 3

to do: With positions off, shuffle sentence B as often as you like. The two bags stay equal.
to do: Turn on Add positions. Now even the two copies of “the” get different bars.
to do: Shuffle again. Every new order now gives a different bag.

Fig. 2.4A sentence, and its words in another order.Each word is a stand-in row of 12 numbers, drawn as bars. One attention head asks every row the same question: the percentages are its weights, and what it gathers is the rows summed by weight. With no positions the weights depend only on the words, so any order of the same words gathers the same thing, to the last digit. Add each word's place (the rows of Fig. 2.5) and the order starts to count.

Each pair is the sine and the cosine of one angle, so the pair is a point on a circle: the tip of a hand.

GPT-2 does not use these waves. It learns its 1,024 position rows like any other weights; when he plots them in Let's reproduce GPT-2, they come out as wavy curves anyway, a little noisy.

2.4 A meter for every place

How do you write down a place so that a network can read it? The place itself, one number, grows without limit. The 2017 Transformer paper used a set of waves instead, and they are easiest to see as the hands of a meter. Slide along the sentence.

Six hands, and how many places each takes to turn once

wpe[3]

Slide along the sentence place 3 of 1,024

Place 3.

Fig. 2.5A place, written as a meter.The position row of the original Transformer (Vaswani and colleagues, 2017), at a width of 12: six pairs of numbers, each pair the sine and cosine of one hand on a dial. The first hand turns once every 6 places, the last once every 13,537, like the dials of a gas meter. Below the dials, the meter's tape: one column for each place near yours, one row for each of the 12 numbers, blue for plus and black for minus. No two columns are the same, and neighbouring columns look alike. Press the tape to jump to a place.

The code is his, from Let's build GPT. nanoGPT names the same two tables wte and wpe, as GPT-2 does.

Both tables start as small random numbers and are trained with everything else.

2.5 Two tables, added

Andrej explainsToken and position embeddings, added together

We're going to have a second position embedding table: each position from zero to block size minus one will also get its own embedding vector. X will be the addition of the token embeddings with the positional embeddings, so at this point X holds not just the token identities, but the positions at which these tokens occur.

Andrej Karpathy, Let's build GPT: from scratch, in code, spelled out., 1:00:38 to 1:01:42

Now the real thing, on the prompt you typed at the top of the page. Two lookups and one addition, and your token is ready for the first block.

Your prompt, as the tokenizer of chapter 1 cut it. Choose a token.

wte 640 rows

wpe 1,024 rows

wte[323]

wpe[14]

Added, number by number: this is x, the start of the residual stream.

Step 3 of 3 add the two rows

tok_emb = self.token_embedding_table(idx) # (B,T,C)
pos_emb = self.position_embedding_table(torch.arange(T, device=device)) # (T,C)
x = tok_emb + pos_emb # (B,T,C)

Fig. 2.6Two tables, added.Each of your tokens takes the row of the token table named by its id, and the row of the position table named by its place, and the two are added. The 640 rows of the tokenizer above stand in for GPT-2's 50,257, and 12 numbers for its 768; the token rows are stand-ins drawn from each id, and the position rows are the meter of Fig. 2.5.

From here on your token is never an id again. It is x, a row of numbers riding the residual stream, and every block that follows reads it and adds to it.

Exhibits after Andrej Karpathy, makemore Part 2 and Let's build GPT (MIT). The letters' table in Figs. 2.2 and 2.3 is real and trained in your browser on his sentences; the word and token rows in Figs. 2.4 and 2.6 are small stand-ins drawn from their text and ids.

Chapter 3Layer 1 of 12

Attention

A communication mechanism: each token looks back and gathers.

Fig. 3.1The attention block. The stream is normalized, each token gathers information from the tokens before it, and the result is added back to the stream.

After Andrej Karpathy, Let's build GPT: from scratch, in code, spelled out, and nanoGPT (MIT). Every share on these pages is computed in your browser, on the prompt you typed at the top.

Up to here each token has travelled alone: its vector says what it is and where it stands, and nothing about its neighbours. Attention is where tokens first talk. Each one looks back over the tokens before it, decides which of them matter, and takes a weighted share of what they carry. He builds it in a few small moves, and so does this chapter.

In the lecture this is his second version of the trick, after a slow double for loop. He calls the result xbow: x, bag of words.

An average forgets the order of the past and treats all of it alike. A weak way to talk, and a start.

3.1 An average of the past

Andrej explainsAttention as nodes passing messages

Attention is a communication mechanism. You can think of it as a number of nodes in a directed graph, where every node has some vector of information and gets to aggregate information via a weighted sum from all of the nodes that point to it.

Andrej Karpathy, Let's build GPT: from scratch, in code, spelled out., 1:11:42 to 1:12:06

Start with the crudest conversation there is: each token takes the average of itself and every token before it. Step down the rows and watch which tokens join in. The neat part is that one matrix multiply does every row at once.

xbow₄

= (x1 + x2 + x3 + x4) / 4

Row 4, ‘as’: the average of 4 tokens, ¼ each.

wei  = torch.tril(torch.ones(T, T))
wei  = wei / wei.sum(1, keepdim=True)
xbow = wei @ x   # (T, T) @ (T, C) → (T, C)

Fig. 3.2The triangle.Rows and columns are the first six tokens of your prompt. Row t holds the share each token gets in token t's average: only tokens at or before t get one, so everything above the diagonal is zero. On the right, each token's vector (six stand-in numbers). One matrix multiply, wei @ x, averages every row's past at once.

masked_fill writes −∞ wherever the triangle has a 0, and e^−∞ is exactly 0: the future gets no share at all.

A block with this mask is a decoder. Delete the line and every token hears every other: an encoder.

3.2 The future must not talk

Why only the tokens before? Because of what a GPT is trained to do.

Andrej explainsThe causal mask: the future cannot talk to the past

The token at the fifth location should not communicate with tokens in the sixth, seventh and eighth location, because those are future tokens in the sequence. It should only talk to the fourth, third, second and first: information only flows from previous context to the current time step, because we are about to try to predict the future.

Andrej Karpathy, Let's build GPT: from scratch, in code, spelled out., 43:19 to 43:49

It learns by predicting the next token of its training text, and one sequence holds many such lessons at once. If a token could hear the tokens after it, every lesson would come with the answer attached.

examples that can read their own answer: 0 of 5

training examples can read their own answer

Example 1: sees The·and must predict un
Example 2: sees The·unand must predict re
Example 3: sees The·unreand must predict as
Example 4: sees The·unreasand must predict on
Example 5: sees The·unreasonand must predict able·

ReadingEach example sees only what came before it. To guess ‘re’ from ‘The·un’, the model has to learn something.

Fig. 3.3One sequence, five lessons.A sequence of six tokens is five training examples at once: each position must predict the token after it (right). With the mask on, the blanks are the future, hidden. Switch it off and the answer sits in plain view, filled in red.

So the upper triangle stays shut. The real model writes the same triangle a third way: start from zeros, put minus infinity wherever the future is, and softmax each row.

The·unreas

The·0000

un0000

re0000

as0000

Turn 1 of 4. Start from zeros.

wei = torch.zeros((T, T))
# affinities: all equal, for now
wei = wei.masked_fill(tril == 0, float('-inf'))
wei = F.softmax(wei, dim=-1)

or pick a turn

Fig. 3.4The triangle, built from softmax.His third version of the same matrix, on the first four tokens of your prompt. e to the minus infinity is zero, so the forbidden cells get no share; softmax of equal zeros gives equal shares. Turn four swaps the zeros for affinities computed from the tokens themselves (stand-ins here), and the shares stop being equal.

Softmax gives equal numbers equal shares; give one token a bigger number and it gets a bigger share. All attention needs now is a way for the tokens to choose their numbers.

The names come from Attention Is All You Need (Vaswani and colleagues, 2017): query, key and value.

In GPT-2 small each head's query, key and value have 64 numbers. The compass has two, so you can see them.

3.3 A question and an answer

Andrej explainsQuery and key: what am I looking for, what do I contain

Every single token at each position will emit two vectors: a query and a key. The query vector, roughly speaking, is what am I looking for, and the key vector, roughly speaking, is what do I contain.

Andrej Karpathy, Let's build GPT: from scratch, in code, spelled out., 1:04:04 to 1:04:24

Each token makes two vectors from its own numbers. A token's affinity for an earlier one is the dot product of its query with that token's key: large when the two point the same way, negative when they point apart. Turn the query and watch the shares move.

What of· gets from each key: score and share
key	q · k	share
iv	0.10	7%
en	0.97	18%
es	1.34	25%
s·	1.44	28%
of·	1.17	22%

‘of·’ listens most to ‘s·’: 28%.

Turn the query 150°Stretch the query length 1.5

Try this 0 of 4

to do: Point the query at one key until it gets more than half.
to do: Point it away from every key, so every score goes negative.
to do: Stretch it all the way and aim: one key takes three quarters or more.
to do: Shrink it to nothing.

Fig. 3.5The question, turned on a compass.The last five tokens of your prompt, each with a key (ink arrows; two numbers here, 64 in GPT-2). The red arrow is the query of the last token, ‘of·’. Each score is the dot product q · k: the length of the query times the shadow each key casts on its line. Softmax turns the scores into shares. Drag the red pen, or use the two levers.

Andrej explainsThe value vector: what actually gets aggregated

There's one more part to a single self attention head: when we do the aggregation, we don't actually aggregate the tokens exactly. We produce one more vector and call that the value: it's the elements we aggregate, instead of the raw tokens.

Andrej Karpathy, Let's build GPT: from scratch, in code, spelled out., 1:10:11 to 1:10:52

Once a token has its shares, it does not take the others' raw numbers. Each token offers a third vector, its value, and the shares decide how much of each value is taken.

ivv× 7%

env× 18%

esv× 25%

s·v× 28%

of·v× 22%

out= the sum of the column

v   = self.value(x)  # if you find me interesting, here is what I'll tell you
out = wei @ v        # for each token: Σ share × value

Fig. 3.6What gets passed along: the values.Each token also carries a value (four stand-in numbers, left). Every value is scaled by its share from Fig. 3.5 (right column) and the scaled values are added up: that sum is what ‘of·’ takes away from attention. Turn the query above and watch the column change.

Shrink the query to nothing and you are back at the plain average of Fig. 3.2. Attention is that average, with weights the tokens choose for themselves.

The 2017 paper gives the same reason: large dot products push softmax into regions where its gradients are extremely small.

In GPT-2 the head size is 64, so every score is divided by 8.

3.4 Keep it soft

One detail of the formula looks fussy and matters a great deal: the scores are divided by the square root of the head size. First, see what size alone does to softmax.

the largest share 29%

goes to the largest number

Multiply the numbers by

torch.softmax(torch.tensor([0.1, -0.2, 0.3, -0.2, 0.5]), dim=-1)

Fig. 3.7Softmax sharpens.His five numbers from the lecture. Their order never changes, only their size; yet multiply them by 8 and softmax hands the largest one most of the total. Big scores make softmax pick one winner.

Now the scores themselves. Each is a sum of head_size products of numbers around 1, so the bigger the head, the bigger the scores, before the network has learned a thing. Break it, then fix it.

Head size: numbers in each q and k

draw no. 001

Try this 0 of 3

to do: Set the head size to 256, with the division off.
to do: Now switch the division on.
to do: Press for new numbers. The lesson holds.

q = torch.randn(B, T, head_size)   # var about 1
k = torch.randn(B, T, head_size)   # var about 1
wei = q @ k.transpose(-2, -1)

all 256 scores, on a ruler from −24 to 24

var(q): 0.00
var(k): 0.00
var(wei): 0.0

one token’s shares over 8 tokens

largest share 0%

ReadingDrawing the numbers.

Fig. 3.8Why divide by the square root of the head size.Fresh random queries and keys, every number with variance 1. A dot product adds up head_size products, so its variance grows to about head_size: the spread on the ruler widens and softmax picks a winner before the network has learned anything. Dividing by √head_size brings the variance back to about 1.

Without the division a big head shouts from the very first step: each token hears one other, the rest go unheard, and a softmax pushed to its extremes passes back almost no gradient to learn from.

Heads that look one token back, and heads that rest on the first token, are both found in trained models (Elhage and colleagues, 2021; Xiao and colleagues, 2023).

Our four heads have their affinities set by hand to draw those patterns; the mask, the softmax and the sums are real.

3.5 Many heads

One head asks one kind of question. A real layer runs several side by side, each with its own query, key and value, so each can look for something different. Here are four, reading your prompt. Head 3 is the tokenizer's debt being paid: chapter 1 cut words into pieces, and here the pieces find each other again.

Which head

Try this 0 of 3

to do: Look through all four heads.
to do: Pick another token: press a row of the matrix.
to do: In head 3, pick a piece from the middle of a word.

Reading‘of·’, in head 1, looks one token back: s· 80%, The· 1%, un 1%.

Fig. 3.9The attention matrix, in halftone.Your prompt, up to its last 16 tokens. Each row is one token listening; the area of each dot is the share it gives the token in that column. Hatched cells are the future, masked. Press a row to follow that token.

Fig. 3.10The same numbers, as a graph.His picture: tokens are nodes, and each one gathers a weighted sum from the nodes that point to it. Time runs down the page, so every wire runs from an earlier token to a later one. Red: the wires into your chosen token.

H1H2H3H4

cat16 numbersprojto the stream

out = torch.cat([h(x) for h in self.heads], dim=-1)
out = self.proj(out)   # then x = x + out: chapter 4

Fig. 3.11Several heads at once.Each head reads the same tokens with its own query, key and value, and hands back a few numbers of its own (H1 to H4, for ‘of·’). They are laid end to end, mixed by one more linear layer, and added to the stream. GPT-2 small runs 12 heads of 64 numbers in each of its 12 layers: 768 numbers, and 144 different ways of looking back.

His notes on attention, from the lecture's notebook, paraphrased.

3.6 A set, not a sequence

One last surprise. Attention has no idea where anything is. Shuffle the tokens before the last one and, unless they carry their positions, its result does not change at all.

what ‘of·’ takes away

change from the first order: 0.00

The tokens in the order you wrote them.

Try this 0 of 2

to do: Shuffle with positions off. Nothing changes.
to do: Switch positions on, then shuffle.

Fig. 3.12Attention sees a set.A real attention head (four-number stand-ins, three fixed weight matrices) reading the last five tokens of your prompt. With no positions, a token's share depends only on what it is, so the order makes no difference to what ‘of·’ takes away. Add a position to every token, as chapter 2 does, and order starts to count.

He sums it up in a few notes:

Attention is a communication mechanism: nodes in a directed graph, each gathering a weighted sum from the nodes that point to it. Fig. 3.10
It has no notion of space. It acts on a set of vectors, which is why every token is given its position. Fig. 3.12
The sequences in a batch never talk to each other; each is processed on its own.
An “encoder” block deletes the one masking line, so every token talks to every other; a “decoder” block keeps the triangle, because it writes text. Fig. 3.3
“Self”-attention: the keys and values come from the same tokens as the queries. In cross-attention they come from somewhere else, such as an encoder.
“Scaled” attention divides by √head_size, so softmax stays soft when training starts. Fig. 3.8

And here is the whole head, as he types it in the lecture. Every line is a figure in this chapter.

class Head(nn.Module):
""" one head of self-attention """
def forward(self, x):
B, T, C = x.shape
k = self.key(x) # what do I containFig. 3.5
q = self.query(x) # what am I looking forFig. 3.5
wei = q @ k.transpose(-2, -1) * k.shape[-1]**-0.5Fig. 3.8
wei = wei.masked_fill(self.tril[:T, :T] == 0, float('-inf'))Fig. 3.3
wei = F.softmax(wei, dim=-1)Fig. 3.4
v = self.value(x)Fig. 3.6
out = wei @ vFig. 3.6
return out

Fig. 3.13One head of self-attention, as he types it.The forward pass of his Head class, with each line keyed to the figure that takes it apart (the scale written as in his finished code, by the key's own size). Run several in parallel, concatenate, project: that is the attention block of Fig. 3.1.

That is attention: tokens asking, answering and sharing, the one place in the GPT where they talk to each other. What they learn is added to the stream that carries them, which is the next chapter.

Exhibits after Andrej Karpathy, Let's build GPT and nanoGPT (MIT). The token vectors, keys and values are small stand-ins made from your tokens' text; the four heads of Figs. 3.9 to 3.11 have hand-set affinities. Every mask, dot product, softmax and sum is computed as you watch.

Chapter 4Layer 1 of 12

Residual

Each block adds to the stream; nothing replaces it.

Fig. 4.1The residual connection. The block computes f(x) on a branch while the stream carries x straight past it; the two are summed.

After Andrej Karpathy, micrograd and nanoGPT (MIT). Every figure is computed in your browser: a real graph, a real backward pass, two real towers trained side by side.

A GPT is a tall stack of blocks: twelve in the smallest GPT-2, ninety-six in GPT-3. Stack plain layers that high and the network stops learning, because the signal fades on the way up and the gradient fades on the way down. The cure is one of the shortest lines in deep learning, x = x + f(x). Every block adds to a stream that runs straight past it, instead of replacing what came in.

Drawn the way micrograd's draw_dot draws a graph: a box per value with its data and its grad, an oval per operation.

With the skip on, x is used twice, so it collects gradient from both paths. That is why micrograd writes self.grad +=, never =.

4.1 A plus sign passes it on

Start with one neuron, as in micrograd: x is multiplied by a weight w and squashed by tanh. The backward pass asks how much a nudge to x would move the output; that number is x.grad. Step it through, node by node. Then turn on the skip connection, which adds x itself to the output, and step it again.

x.grad 0.000

x.grad, the gradient that reaches x

slope of tanh, 1 − t²: 0.559
through the branch, w(1 − t²): 0.559
down the skip: off

Step the backward pass to see what reaches x.

ReadyForward pass done. Every grad starts at zero, as after zero_grad().

Weight w 1.0

Try this 0 of 4

to do: Run backward() with the skip off.
to do: Push w past 2.5 and run it again: almost nothing gets back.
to do: Turn on the skip and run it.
to do: Keep the skip on and push w past 2.5.

Fig. 4.2A plus sign passes it on.One neuron, as micrograd draws it: a box for every value with its data and its grad, an oval for every operation. The backward pass runs from the output to the inputs, each operation multiplying by its local derivative. The + has derivative 1 on both sides, so it hands the gradient on untouched.

Andrej explainsWhy addition, not replacement, keeps gradients flowing

You have this residual pathway, and you are free to fork off from it, perform some computation, and then project back to the residual pathway via addition. This is useful because during backpropagation, addition distributes gradients equally to both of its branches.

Andrej Karpathy, Let's build GPT: from scratch, in code, spelled out., 1:29:03 to 1:29:37

A × scales the gradient by the other input, and tanh by its slope, so both can shrink it to nothing. A + only copies it. Whatever the branch does, the skip hands the full gradient back.

Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun (2015) found that a plain network 56 layers deep did worse than one 20 deep, even on its own training data. Their residual networks went to 152 layers and won ImageNet that year.

PyTorch's default for a linear layer: weights and biases drawn evenly from ±1/√fan_in. Chapter 5 is about choosing them with more care.

4.2 Twenty blocks deep

Andrej explainsWhy deep transformers need skip connections

We're starting to get a pretty deep neural net, and deep neural nets suffer from optimization issues. So we need one idea we can borrow from the Transformer paper: skip connections, sometimes called residual connections, from the paper on deep residual learning for image recognition, from about 2015.

Andrej Karpathy, Let's build GPT: from scratch, in code, spelled out., 1:28:08 to 1:28:53

Now stack them. Each block is a layer of 16 tanh neurons, set up the way PyTorch sets up a linear layer by default, and the stack is fed 48 inputs at once. The upper figure follows the signal up: does each block still tell the inputs apart? The lower one follows the gradient down from the loss. With the switch off each block replaces x; with it on, each block adds to it.

Blocks in the stack

weights no. 0001

Try this 0 of 4

to do: Turn the residual connections on.
to do: Go 40 blocks deep with them off.
to do: Now turn them on, still 40 deep.
to do: Press for new weights. The picture holds.

for block in blocks:
    x = block(x)   # replaces x
# block(x) = tanh(x @ W + b), 16 neurons, PyTorch’s default init

Fig. 4.3The signal, going up.How much each block's output still differs from one input to the next, averaged over its 16 neurons, on a log scale. When it reaches zero the block gives the same answer for every input: it has forgotten what came in.

Fig. 4.4The gradient, coming down.The size of each block's weight gradient after one loss.backward(), as a share of the top block's. Grey ticks show where the bars stand with the switch the other way.

of the top block's gradient reaches block 1

blocks: 20
signal after the top block
signal after block 1

Running the forward and backward pass.

Off, the deep blocks receive a vanishing sliver of the gradient, so they never learn. On, block 1 hears the loss about as clearly as the top block does. The only change is two characters, x +.

Both towers: 20 blocks, the same starting weights, plain gradient descent at learning rate 0.003 on all 48 points at every step.

The last layer starts small, a trick from his makemore notebook, so that neither tower begins with wild guesses.

4.3 Press to train

Numbers on a page are one thing; learning is the proof. Here are two towers of 20 blocks with identical weights, asked to learn the same wave. The only difference between them is the x +.

step step 0000 of 1,500

Plain x = block(x)

loss plain tower loss 0.000

Residual x = x + block(x)

loss residual tower loss 0.000

ReadyBoth towers start from the same weights. Neither has learned anything yet.

Fig. 4.5Press to train.Two towers of 20 blocks with the same starting weights, learning the open dots, 0.8 · sin(3x), by plain gradient descent at the same learning rate. The lines are each tower's guess; below, both losses, step by step. Training pauses when the figure leaves the screen.

The plain tower settles on a flat line through the middle, the best it can do without hearing its input: the signal never reaches the top, and the gradient never reaches the bottom.

Andreas Veit, Michael Wilber and Serge Belongie (2016) unrolled residual networks this way. In a 110-layer network, most of the gradient came from paths only 10 to 34 layers deep.

GPT-2 scales the weights of its residual layers at initialization by 1/√N, N the number of residual layers, so the stream does not grow too loud as the blocks add up. In nanoGPT: std = 0.02 / sqrt(2 * n_layer).

A board of pegs that sorts falling balls into a bell curve: Charles and Ray Eames built a big one, the Probability Machine, for their 1961 exhibition Mathematica.

4.4 The gradient superhighway

In a GPT the stream runs from the embeddings to the output, and every block adds to it twice: once from attention, once from the MLP. In the backward pass the gradient starts at the loss and comes down that stream. At every + it splits, one copy straight on down the stream, one out through the block.

Andrej explainsThe gradient superhighway down the residual stream

The gradients from the loss basically hop through every addition node all the way to the input: you have this gradient superhighway that goes directly from the supervision all the way to the input, unimpeded.

Andrej Karpathy, Let's build GPT: from scratch, in code, spelled out., 1:29:37 to 1:29:58

Count the ways down. Turn the stack upside down, put the loss at the top, and drop a gradient through the adds.

paths from the loss to the input: 16,777,216

ways down from the loss to the input, 2²⁴

adds, two to a block: 24
balls dropped: 0
straight down the stream: 0

Drop a gradient from the loss and watch it choose at every +. Only 1 path in 16,777,216 goes straight down.

Blocks in the GPT

Try this 0 of 3

to do: Drop one gradient and follow it down.
to do: Drop a hundred: a bell curve builds up.
to do: Go down to 1 block and drop until one rides the stream all the way.

Fig. 4.6Every way down.The stack turned upside down, the loss at the top. Each + is a fork: straight down the stream, or out through the block and back. In a real backward pass the gradient takes every path at once and the pieces are summed; the ball only counts them. The thick line is the one path that touches no block, so the gradient on it arrives exactly as it left the loss.

Millions of paths, most of them through about half the branches; and one, the stream itself, through none. However deep the stack, that path carries the gradient down to the first layer without passing through a single block. That is why a GPT can be ninety-six blocks tall and still learn.

Exhibits after Andrej Karpathy, micrograd (the Value class: add, mul, tanh, backward) and nanoGPT (MIT). The towers in Figs. 4.3 to 4.5 are small plain networks written for this page; their numbers are computed in your browser.

A short break from the stream. How does the person who explains all this actually work?

InterludeAfter Chapter 4 · from his blog, 2024 and 2025

How he works

Between lectures he writes short posts about his own tools: one text note, a pocket calculator, a drawer of security keys, four sleep trackers, a water filter. Each is the habit of his code applied to a life: keep it small, own it, measure it.

After Andrej Karpathy, five posts on karpathy.bearblog.dev, September 2024 to December 2025. The quotes are his; the simulations, the made-up nights and the attacks are ours.

Four chapters in, the network has a tokenizer, embeddings, attention and a residual stream. Time for a break with the man who explains them. He treats a note, a calculator or a night's sleep the way he treats a neural net: strip it down to the smallest thing that works, then measure it. Each exhibit below is a working machine, and each hides a real idea from computing.

I keep ranting about this topic to all of my friends recently (you can tell I'm really fun at parties)

Andrej Karpathy, Chemical hygiene (2025)

His only tags: watch: for late at night, listen: for a run or a walk, read: for a flight. Everything else is plain text.

Move-to-front was studied by John McCabe in 1965. In 1985 Daniel Sleator and Robert Tarjan proved it never costs more than about twice the best possible strategy, even one that knows the future.

Many of your computer's caches follow the same rule: keep whatever was used last closest to hand.

N.1 One text note

He keeps one single note in Apple Notes, called “notes”. No folders, no system. Any idea, todo or film recommendation goes in on top, as plain text.

As things get added to the top, everything else starts to sink towards the bottom, almost as if under gravity.

Andrej Karpathy, The append-and-review note (2025)

Now and then he scrolls down and skims. Anything that still deserves his attention he copies back to the top; the rest keeps sinking, never lost. The note below is a real one. Append, review, rescue: it stays in your browser.

notes10 lines

TODO today: finish the attention figure, answer the email about the talk, 30 min zone 2
watch: the film someone mentioned at the party
idea: a tokenizer that reads digits one at a time?
read: the book from that glowing review
tweet draft, needs more thought: why small models will be everywhere
shell: find . -name "*.py" (every Python file, recursively)
sweep: lr 3e-4 vs 1e-3, 5k steps. 3e-4 wins, val loss 1.47
listen: a long podcast for the Sunday walk
quote: "What I cannot create, I do not understand." Feynman
paper: training compute is about 6 × parameters × tokens

Find, his CTRL+F

Review, now and then

Fig. N.1One text note.His whole system, working. Type a line and press Enter: it goes on top and everything else sinks. Rescue a line and it jumps back to the top. Your note stays in this browser only; nothing is sent anywhere.

Why does something this simple work? Because the things you need again are mostly the things you needed recently. Every rescue puts a live idea back where you will look first. Here are two copies of one note, living the same three months.

Dayday 00 of 90

His notewhat you find goes back on top

lines skimmed per find, his note 0.0lines skimmed per find

Never rescuedthe same note; what you find stays put

lines skimmed per find, never rescued 0.0lines skimmed per find

Fig. N.2Gravity, measured.Two copies of one note live the same ninety made-up days, three thoughts a day. A third are new and go on top; the rest are old ideas coming back, which first have to be found. Rescuing them keeps the ones that matter near the top, and the search stays short however deep the note gets.

Computer scientists call this rule move-to-front, and it is provably close to the best you can do without knowing the future. His verdict is shorter: “One text note ftw.”

He came to it in a bookshop, flipping through Empire of the Sum: The Rise and Reign of the Pocket Calculator.

There is no sin or cos key. For that, his post jokes, you would be asked to upgrade your subscription to Calculator+.

Most pocket calculators store each decimal digit in its own four bits, binary-coded decimal, so they round in decimal, as you do.

N.2 No other master

In 2024 he wrote a love letter to an object. Not as a gadget to buy today, he says, but “as a product of technology and a symbol of a kind of philosophy”.

You paid for it and now it is yours. It has no other master.

Andrej Karpathy, I love calculator (2024)

Here is one. Press ON/C and do a few sums. Beside it, the same sums run in JavaScript, which counts in binary: the card shows where the two machines part ways.

off

This calculator, in decimal: off
JavaScript, in binary

Off. It will not ask for anything when you turn it on.

Try this 0 of 4

to do: Press ON/C. That is the whole setup.
to do: 0.1 + 0.2 =
to do: 1 ÷ 3 × 3 =
to do: 2 √ × 2 √ =

Fig. N.3A plugin for your brain.It needs light or a battery and nothing else. Like a real one it works in decimal and keeps eight digits, cutting off the rest: that is why 1 ÷ 3 × 3 comes back as 0.9999999. JavaScript works in binary, where 0.1 has no exact form, just as 1/3 has none in decimal. Neither machine is exact; you can only see the calculator's error.

What he loves most is everything the calculator does not need. Against it he sets a CO₂ monitor that, before it would say anything about the air in his room,

demanded that I create an account, download their app and allow precise location information

Andrej Karpathy, I love calculator (2024)

On the day you unbox it

The calculatorneeds one of two

lightora battery

Computes.

The CO₂ monitorneeds every one of five

their app
an account
your precise location
an internet connection
their servers, up

Reports 640 ppm.

Fig. N.4Unboxing day.The boxes printed in bold are the three things his monitor demanded; the other two are what those three need in turn. Two things in a row must both hold; two side by side are spares, and either will do.

He calls this a “dependency footprint”, and it has simple arithmetic. Set how likely each link is to survive ten years, then lengthen the chain.

% chance a link lasts ten years

Links in a row

5 links in a rowchance the chain still works 59%.9 × .9 × .9 × .9 × .9 = 0.59

two side by side, as light or batterychance the spares still work 99%1 − (1 − .9)² = 0.99

Fig. N.5Chains and spares.If each link holds with chance p, a chain of n links holds only if all of them do: p multiplied by itself n times. Every new dependency multiplies in one more number below one. Spares work the other way: the pair fails only if both fail.

Every dependency multiplies in one more chance to fail. The calculator's whole footprint is a pair of spares, which is why, in his words, it “would work a thousand years ago”.

The three factors: something you know (a password), something you have (a key), something you are (a face or a finger).

A hardware key signs in only for the site it was registered with, so a perfect copy of your bank's page gets nothing from it.

He pays for his privacy tools on purpose, “so that I am the customer”.

N.3 Digital hygiene

Two days before the note, he posted his list of digital habits, the most basic first. A password is your first factor, something you know; his first rule is about it.

Do not be a noob and mint new, unique, hard passwords for every website or service that you sign up with.

Andrej Karpathy, Digital hygiene (2025)

Each habit closes one door. Switch them on, one at a time, and watch which attacks still get in. Start with the common second factor, the code by text message, that he warns against.

Your habits

attacks stopped: 0of 8 attacks
stopped

A site you used years ago leaks its passwords, and someone tries yours everywhere else.Gets in.
They have your password, call your phone company as you, and move your number to their phone.Gets in.
An email from "your bank" links to a login page that is perfect in every pixel.Gets in.
Someone looks up your mother's maiden name and resets your account.Gets in.
Your laptop is stolen from a café table.Gets in.
A shop you bought from once leaks every card number it holds.Gets in.
An app on your laptop quietly calls home far more than it should.Gets in.
An old chat leaks, years later.Gets in.

Fig. N.6Who gets in.Eight ordinary attacks against nine habits. Most attacks are stopped by exactly one habit, so each habit earns its place. The text-message code is the trap: it stops the leaked password, then hands the SIM swapper the keys.

Security is not one big lock but many small ones, each on a different door. He ends the list with what he has not done yet, and: “The journey continues.”

His four: Oura, a ring; Whoop, a wrist band; 8Sleep, a mattress cover; and an Apple Watch running AutoSleep.

Correlation, r, runs from −1 to 1: 1 when two numbers rise and fall in step, 0 when one tells you nothing about the other.

His verdict: Whoop 9.5 out of 10, Oura 10 if you can live with a ring. AutoSleep: “Do not use.”

N.4 Four trackers, sixty mornings

In early 2025 he set out to sleep better. First, of course, he had to measure it, and there were four candidates. So he slept with all four, every night for two months, and logged each score in a spreadsheet every morning, beside how he felt.

When my score is low, I lack agency, I lack courage, I lack creativity, I'm simply tired.

Andrej Karpathy, Finding the Best Sleep Tracker (2025)

Nothing tells you how well you really slept, so no tracker can be checked against the truth. He did the next best thing: he checked them against each other.

His correlations between the four trackers' sleep scores. Press a number to see its nights.
	Oura	Whoop	8Sleep
Whoop
8Sleep
AutoSleep

His table, from about two months of mornings. Press a number.

r = correlation so far 0.65

Whoop against Oura, after 60 nights

Fig. N.7What 0.65 looks like.The numbers in the table are his; the dots are sixty made-up nights drawn to have exactly his correlation. Watch the lower trace as the nights come in: after ten nights r still swings wildly, which is why he waited two months before believing it.

Why should two gadgets agreeing mean anything? Here is the argument, built as a machine.

Tracker A's noise

Tracker B's noise

r = agreement between A and B 0.00

how much A and B agree, the only number you can measure

They barely agree. At least one of them is not reading your sleep.

Fig. N.8Why agreement is evidence.Nobody can see how well you really slept, so no tracker can be checked against the truth. But two trackers can be checked against each other, and their agreement is roughly the product of how well each one reads the truth. Turn one dial to ∞ and it agrees with nobody.

If we think that Whoop is good (which I think it is), AutoSleep looks almost like a noise generator.

Andrej Karpathy, Finding the Best Sleep Tracker (2025)

His aim is the 80:20 low-hanging fruit. Teeth, he admits, are not low-hanging: he still uses a plastic toothbrush.

Why heat? Additives leave a plastic by diffusion, and diffusion speeds up as the temperature rises.

Sunscreen he keeps for long hours in strong sun on his “computer scientist vampire skin”.

N.5 Chemical hygiene

In December 2025 the same method reached the kitchen: a guide to what he drinks, cooks in and wears, written up because he kept telling his friends anyway. He offers it as a start, not a verdict.

Even doing all of the above you are simply decreasing risk, you can never eliminate it.

Andrej Karpathy, Chemical hygiene (2025)

swaps made: 0of 8 swaps made

of them about heat: 0of them about
plastic and heat

Fig. N.9His kitchen, swapped.Press each thing to swap it as his guide says. The rules are his, from a post he calls evolving and opinionated; he is not a toxicologist and says so. Count how many of the swaps come down to heat, and keep an eye on the toothbrush.

Five posts, one method. Build the smallest version yourself, own it, then measure it before you believe it. The next chapter is that method turned on a network: print what every layer is doing, and fix what the numbers show.

From his posts on karpathy.bearblog.dev: I love calculator (8 September 2024), Digital hygiene (17 March 2025), The append-and-review note (19 March 2025), Finding the Best Sleep Tracker (24 March 2025) and Chemical hygiene (18 December 2025). The correlations in Fig. N.7 are his; the nights, the noise and the ninety days are simulations.

Back into the stack, where the numbers have to stay in a healthy range or nothing learns.

Chapter 5Layer 1 of 12

LayerNorm

Keep every token’s numbers in a healthy range, so a deep stack can learn.

Fig. 5.1LayerNorm. Each token’s vector is shifted to mean 0 and scaled to variance 1, then a learned gain γ and bias β are applied. GPT-2 normalizes before each sub-block.

After Andrej Karpathy, makemore Part 3: Activations & Gradients, BatchNorm (MIT). The towers in Figs. 5.3 to 5.8 are real forward and backward passes, recomputed in your browser every time you turn a knob.

A deep network passes its numbers through layer after layer, multiplying as it goes. Unless every layer hands the next one numbers of about the same size, they blow up or fade away, and the network stops learning. This chapter is about keeping them in range: first by setting the weights with care, then by simply forcing the numbers there.

tanh is the hyperbolic tangent. Its slope at output t is 1 − t², the very line micrograd runs in its backward pass.

5.1 The squash

Each neuron adds up its inputs and passes the sum through tanh, which squashes any number into the range from −1 to 1. Slide the input and watch the lower curve: it is how much of the learning signal can travel back through the neuron. Push x past 2 and it all but vanishes.

Slide the input x

share of gradient let back 79%

of the gradient gets back through this neuron

input x: 0.50
output t: 0.462
slope 1 − t²: 0.786

In the active middle: the gradient gets through.

Fig. 5.2The squash.tanh pins every input between −1 and 1. Its slope is how much of the gradient flows back through it, micrograd's out.grad * (1 - t**2). Past about 2 the curve is flat, the slope is nearly zero, and the neuron stops learning.

His printout for every layer: mean, std, and the share with |t| > 0.97, which he calls saturated. We print the same numbers; “flat” is his “saturated”.

Figs. 5.3 to 5.5 are one machine: touch a control and all three are recomputed.

5.2 Five layers, printed one by one

Now stack five of them, as in his makemore notebook: 100 neurons a layer, fed a batch of 32 examples, with the weights drawn as plain random numbers. Start where he starts. Look at the orange towers at the edges of Fig. 5.3: most of the numbers are pinned at −1 or 1, where the slope is zero. Then follow the card on the exhibit.

Gaingain 1

batch no. 0001

Try this 0 of 4

to do: Flip the switch: divide by √fan_in.
to do: Turn the gain to 5/3.
to do: Find a dead neuron: switch off, gain 3, then look through the layers.
to do: Press for a new batch. The shapes hold.

W = torch.randn((fan_in, fan_out)) * 1
h = torch.tanh(x @ W)   # five times, 100 neurons each

ReadingRunning the forward pass.

Layer 1std 0.00flat 0%

Layer 2std 0.00flat 0%

Layer 3std 0.00flat 0%

Layer 4std 0.00flat 0%

Layer 5std 0.00flat 0%

Fig. 5.3Activations, layer by layer.Each bar counts how many of the 3,200 tanh outputs (32 examples, 100 neurons) landed there. Orange: the flat tails, |t| > 0.97, where tanh has stopped listening; the orange bars are the “flat” share printed on the right.

Layer 1std 0.0×10⁰

Layer 2std 0.0×10⁰

Layer 3std 0.0×10⁰

Layer 4std 0.0×10⁰

Layer 5std 0.0×10⁰

Fig. 5.4Gradients, layer by layer.What flows back into each layer after loss.backward(), on one shared scale. In a healthy net the five shapes match; when they don't, some layers learn far faster than others.

Andrej explainsA saturated tanh kills its own gradient: the dead neuron

If the entire column is white, we have what's called a dead neuron: a tanh neuron where the initialization of the weights and biases is such that no single example ever activates it in the active part of the tanh. If all the examples land in the tail, this neuron will never learn. It is a dead neuron.

Andrej Karpathy, Building makemore Part 3: Activations & Gradients, BatchNorm, 19:24 to 19:51

Layer on the plate

white: 0%
dead neurons: 0
nearly dead: 0

Fig. 5.5The white columns.One layer, 32 examples down, 100 neurons across, white wherever |t| > 0.99, as in his plt.imshow(h.abs() > 0.99). A column white from top to bottom is a dead neuron, marked solid orange below it: no example reaches its active middle, so it never learns. The smaller marks are nearly dead: white for at least 29 of 32.

torch.nn.init.calculate_gain('tanh') returns 5/3.

Kaiming He and colleagues worked out this kind of scaling in 2015, for ReLU networks.

5.3 Fixing the scale

Dividing by √fan_in keeps each layer's sums about the size of its inputs. But tanh squeezes a little every time, so on its own the spread still shrinks, layer after layer. The gain is a small boost to undo the squeeze.

Andrej explainsFixing the scale: the gain, five thirds for tanh

In the case of tanh, which is what we're using here, the advised gain is five over three. Intuitively, why do we need a gain on top of the initialization? Because tanh, just like relu, is a contractive transformation, it squashes the output distribution, so to fight that squeezing we need to boost the weights a little, to renormalize everything back to unit standard deviation.

Andrej Karpathy, Building makemore Part 3: Activations & Gradients, BatchNorm, 35:14 to 35:59

Find 5/3 on the dial, then try the others. Too little and the signal fades; too much and it saturates.

Gaingain 1

after 20 layers

spread (std): 0.00
flat: 0%

Fig. 5.6Twenty layers deep.The same recipe, randn / √fan_in × gain, stacked twenty high, one real forward pass per setting. Grey lines are printed for gains 1 and 3; your gain is drawn in orange.

Batch Normalization: Sergey Ioffe and Christian Szegedy, 2015. It also learns a γ and a β, and keeps running averages of the batch statistics for use after training.

5.4 Just normalize them

Getting the scale exactly right, layer by layer, is fragile. BatchNorm is the blunt alternative: right before each tanh, standardize the numbers so that every neuron has mean 0 and std 1 across the batch.

Andrej explainsThe batch norm insight: just normalize the hidden states

The insight from the batch normalization paper is: you have these hidden states, and you'd like them to be roughly gaussian, so why not take the hidden states and just normalize them to be gaussian? It sounds kind of crazy, but you can just do that, because standardizing hidden states so they are unit gaussian is a perfectly differentiable operation.

Andrej Karpathy, Building makemore Part 3: Activations & Gradients, BatchNorm, 41:35 to 42:11

Here is the same tower again, built carelessly on purpose: plain randn, no division, no gain. Turn on the normalization.

Gaingain 1

Normalize before each tanh

batch no. 0001

Try this 0 of 3

to do: Turn on BatchNorm.
to do: Keep it on and turn the gain to 3. Nothing moves.
to do: Now try LayerNorm.

W = torch.randn((fan_in, fan_out)) * 1
h = torch.tanh(x @ W)   # five times, 100 neurons each

ReadingRunning the forward pass.

Layer 1std 0.00flat 0%

Layer 2std 0.00flat 0%

Layer 3std 0.00flat 0%

Layer 4std 0.00flat 0%

Layer 5std 0.00flat 0%

Fig. 5.7The careless tower, all five layers.Each from −1 to 1, orange beyond ±0.97. With no normalization the orange tails fill up; standardize before every tanh and each layer sits in the active middle, whatever the weights.

Layer 1std 0.0×10⁰

Layer 2std 0.0×10⁰

Layer 3std 0.0×10⁰

Layer 4std 0.0×10⁰

Layer 5std 0.0×10⁰

Fig. 5.8Its gradients, on one shared scale.Normalized, the five layers receive gradients of about the same size, so they all learn at about the same pace.

With BatchNorm on, the scale of the weights stops mattering: turn the gain all the way up and not a bar moves.

Layer Normalization: Jimmy Lei Ba, Jamie Ryan Kiros and Geoffrey Hinton, 2016. No batch statistics, so it behaves the same in training and in use.

5.5 LayerNorm: the same move, turned sideways

BatchNorm has a price: each example is normalized using the others in its batch, so they become coupled, and after training you need those running averages. A Transformer wants each token normalized on its own. LayerNorm does exactly that: the same arithmetic along the other axis.

Which way the knife cuts

as they arrive

Four tokens by six features, raw, BatchNorm down each column
token	f1	f2	f3	f4	f5	f6	mean · std
The	3.63	−1.76	−2.83	3.96	−0.32	0.05	0.46 · 2.54
␣unreasonable	1.63	0.10	3.99	1.25	−1.95	0.61	0.94 · 1.78
␣effectiveness	2.11	−1.33	−0.24	2.25	−1.13	−0.30	0.23 · 1.44
␣of	−2.24	−0.99	3.09	1.72	−0.44	2.43	0.59 · 1.94
mean · std	1.28 2.16	−0.99 0.69	1.00 2.71	2.29 1.03	−0.96 0.65	0.70 1.05

xmean = x.mean(0, keepdim=True)   # over the batch
xvar  = x.var(0, keepdim=True)
xhat  = (x - xmean) / torch.sqrt(xvar + eps)

Fig. 5.9The same move, turned sideways.Rows are the last four tokens of your prompt; columns are six of their features. BatchNorm makes every column mean 0, std 1, so each token's result depends on the others in its batch. LayerNorm does it to every row: each token is normalized by itself. In his GPT lecture the change is one character, a 0 for a 1.

GPT-2 (2019): “Layer normalization was moved to the input of each sub-block.” nanoGPT adds one more, ln_f, after the last block.

Pre-norm trains more stably in deep stacks (Xiong and colleagues, 2020).

5.6 Where it sits in a GPT

Andrej explainsPre-norm: layer norm moves before the transformation

Very few details about the Transformer have changed in the last five years, but this is one: the add and norm used to be applied after the transformation, but now it's more common to apply the layer norm before it. There's a reshuffling of the layer norms, it's called the pre-norm formulation, and that's the one we're going to implement.

Andrej Karpathy, Let's build GPT: from scratch, in code, spelled out., 1:35:20 to 1:35:55

In the GPT a LayerNorm sits in every block twice, before attention and before the MLP. Your token rides the residual stream, the thick orange rule.

Where the LayerNorm sits

def forward(self, x):
    x = x + self.attn(self.ln_1(x))
    x = x + self.mlp(self.ln_2(x))
    return x

Each branch reads a normalized copy; the stream itself is never normalized, so the gradient runs straight down the rule from the loss to the first layer.

Fig. 5.10One block of the GPT.The thick orange rule is the residual stream, carrying your token. Each branch reads from it, computes, and adds back at a +. Switch between the two placements and watch the two LayerNorms move.

And this is what one of those LayerNorms does to your token, in four turns of the crank.

␣ofTurn 1 of 4. As it arrives.

mean μ: 0.01
spread σ: 1.11

y = (x − μ) / √(σ² + ε) · γ + β

or pick a turn

Fig. 5.11Your token through LayerNorm.Sixteen numbers standing in for its 768 (made from its text; the arithmetic is the real thing). The orange line is their mean, the grey band one spread either side. Training learns a γ and a β for every feature, so the net can stretch a feature back out where that helps.

That is the whole trick behind training deep stacks: keep every layer's numbers near mean 0 and spread 1, so the signal goes forward and the gradient comes back.

Exhibits after Andrej Karpathy, makemore Part 3 (MIT), with the Block from nanoGPT (MIT). The token numbers in Figs. 5.9 and 5.11 are small stand-ins made from your tokens' text.

Chapter 6Layer 1 of 12

MLP

Communication, then computation: each token thinks on what it gathered.

Fig. 6.1The MLP block. Each token on its own: widened from 768 numbers to 3,072, passed through GELU, projected back to 768 and added to the stream.

After Andrej Karpathy, nanoGPT and micrograd (MIT). Every figure is a real computation in your browser; the net in Fig. 6.6 learns while you watch.

Attention let the tokens talk. Now each one goes off alone and thinks about what it heard. That thinking is the MLP, the plainest block in the transformer: widen the numbers four times, bend them, narrow them back, add the result to the stream. It also holds two thirds of every block's weights.

In nanoGPT the whole block is two lines: attention, then the MLP, each added back to the stream.

Map and reduce are an old pair from functional programming: reduce combines many things into one, map does the same thing to each thing on its own.

6.1 Communication, then computation

A transformer block does two jobs in turn. First attention: every token looks back at the tokens before it and gathers what it needs. Then the MLP: every token, on its own, works on what it gathered. He puts it like this.

Andrej explainsCommunication, then computation: why the MLP follows attention

Before, the self attention did the communication, but we went too fast to calculate the logits: the tokens looked at each other but didn't have a lot of time to think on what they found. The feed forward happens on a per token level, all the tokens do this independently: the attention is the communication, and the feed forward is where they think on that data individually.

Andrej Karpathy, Let's build GPT: from scratch, in code, spelled out., 1:25:25 to 1:26:19

Test the difference yourself. Nudge one word's numbers and watch which outputs move.

The machine between the rows

ReadingPress a word to nudge its numbers, then look at which outputs change.

Try this 0 of 3

to do: With Attention, press the first word.
to do: Keep it pressed; switch to the MLP.
to do: Back on Attention, press the last word. Why does only one output change?

Into the block: press a word to nudge it

same

Out of the block: each word's new numbers

# communicate: each token reads the ones before itx = x + self.attn(self.ln_1(x))# compute: each token on its ownx = x + self.mlp(self.ln_2(x))

Fig. 6.2Who hears a change?Five words of your prompt, six numbers each (a scale model of 768), through one real attention head with its causal mask, or through one real MLP. The MLP is the same machine, with the same weights, stamped under every word. Δ is how far a word's output moved.

Andrej explainsAttention reduces, the MLP maps: communication then computation

Attention is an aggregation function, a pooling function, a weighted sum function, it's a reduce operation. The MLP happens at every single token individually, there's no information being collected or exchanged between the tokens. The attention is the reduce, and the MLP is the map.

Andrej Karpathy, Let's reproduce GPT-2 (124M), 20:06 to 20:29

c_fc and c_proj are the layer names in OpenAI's GPT-2 checkpoint. His code keeps them, so the released weights load straight in.

The 4× comes from the 2017 Transformer paper: 512 numbers wide, 2,048 inside.

6.2 Widen, bend, narrow

Inside, the MLP is three steps. A linear layer widens each token from 768 numbers to 3,072. GELU bends every one of them. A second linear layer narrows them back to 768. Turn the crank and follow one token.

Which token goes in

x 8 numbers in

c_fc8 → 32

h 32 numbers

c_proj32 → 8

mlp(x) 8 numbers out

x + mlp(x) back on the stream

Turn 1 of 5. As it arrives. x: the token as it leaves LayerNorm, 8 numbers standing for 768.

neurons below zero, switched off: · of 32
the loudest neuron: ·
size of the note, against x: ·

or pick a turn

class MLP(nn.Module):def __init__(self, config):super().__init__()self.c_fc   = nn.Linear(config.n_embd, 4 * config.n_embd)self.gelu   = nn.GELU(approximate='tanh')self.c_proj = nn.Linear(4 * config.n_embd, config.n_embd) def forward(self, x):x = self.c_fc(x)x = self.gelu(x)x = self.c_proj(x)return x # and in the Block:x = x + self.mlp(self.ln_2(x))

Fig. 6.3One token through the MLP.Drawn to scale: every square is one number, so the middle row is four times as wide as the token. In GPT-2 the rows hold 768, 3,072 and 768 numbers. Grey outlines show what GELU took away, and on the last row, the token as it came in.

Why widen at all? Each of the 3,072 middle numbers is a neuron, a detector that responds to one pattern in the token's numbers. More neurons, more patterns it can notice. That room is not free.

How much wider inside

weights in the model 124,439,808

weights in GPT-2 small, with the MLP 4× wide inside

MLP share of the model: 45.5%
of each block: 66.6%

As GPT-2 has it: the MLPs hold 45.5% of all the weights, two thirds of every block.

GPT-2 small, weights by part
Token embeddings50,257 rows of 768, shared with the output head	38,597,376	31.0%
Position embeddings1,024 rows of 768	786,432	0.6%
Attentionall 12 blocks	28,348,416	22.8%
MLPall 12 blocks	56,669,184	45.5%
LayerNorms25 of them, γ and β	38,400	under 0.1%

Fig. 6.4Where the weights are.GPT-2 small, counted part by part (124,439,808 at 4×). The strip is drawn to one fixed scale, so it grows as you turn the dial. Each MLP has 768 × 3,072 weights going in and 3,072 × 768 coming out, plus biases: 4,722,432 a block.

Turn the dial back to 4×: about 46 of every 100 weights in GPT-2 sit in its MLPs.

GELU: Dan Hendrycks and Kevin Gimpel, 2016. Φ is the normal distribution's running total.

PyTorch's nn.GELU() computes the exact form; approximate='tanh' gives the shortcut, which his GPT-2 reproduction uses to match OpenAI's weights.

6.3 A softer hinge

The bend is what makes the MLP more than arithmetic. The simplest is ReLU: keep the positive numbers, zero the negatives. GPT-2 uses GELU, a smooth version. Compare the two, and watch the lower curve: chapter 5 showed what happens to a neuron whose slope is zero.

Which bend

Slide the input x

share of gradient let back 8%

of the gradient gets back, with its sign flipped

input x: −1.00
GELU(x): −0.159
slope: −0.083

Below zero GELU still has a slope, −0.08: small, never quite zero, so a quiet neuron can be nudged back to life.

Fig. 6.5Two hinges.ReLU cuts at zero; GELU bends smoothly, multiplying x by Φ(x), the chance that a standard normal number falls below x. Its lowest point is −0.17, near x = −0.75. Switch on the magnifier to see how far GPT-2's tanh shortcut strays: at most about 0.00047, near x = ±2.7.

Why a shortcut at all? A little history.

Andrej explainsWhy GPT-2 uses an approximate GELU

There's a little bit of history as to why there's an approximate version of GELU: at the time they developed this nonlinearity, the exact function was very slow to evaluate in TensorFlow, so they developed this approximation. It ended up being picked up by BERT, and by GPT-2.

Andrej Karpathy, Let's reproduce GPT-2 (124M), 22:02 to 22:32

The two moons are the dataset of micrograd's demo notebook, where he trains a small MLP on them with his own tiny autograd engine.

Two linear layers in a row: W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁ + b₂), which is one linear layer.

6.4 Why bend at all

Take the bend out and something surprising happens: the MLP can no longer learn anything a single linear layer could not, because two linear layers in a row are just one. Here is a tiny MLP, 2 numbers in, a hidden layer, 1 score out. Its job is to put the blue points on one side of its line and the grey points on the other. Train it.

The bend

Hidden neurons, for 2 inputs

The data

Try this 0 of 4

to do: Train with no bend. It stops at a straight line.
to do: Choose GELU and train again.
to do: Now the ring: no bend, then GELU.
to do: The ring with GELU and only 2 neurons.

ReadingUntrained: random weights, 0% of the points right by chance. Press to train.

step: step 0000
loss: loss 0.000
right side: percent on the right side 0%

Bent

Each of the 8 neurons folds the plane along its own line; the output adds the folds up. Fig. 6.7 shows them one by one.

Fig. 6.6Bending the line.A real one-hidden-layer MLP, 2 numbers in, 8 hidden, 1 score out, trained in your browser with cross-entropy and Adam. Dots print its answer across the plane, larger where it is surer; the ruled line is where the score is exactly zero. A cross marks every point on the wrong side.

Without the bend, all the width in the world draws one straight line. With it, enough neurons can draw almost any shape.

Mor Geva and colleagues, 2021: “Transformer Feed-Forward Layers Are Key-Value Memories”. Kevin Meng and colleagues, 2022, traced facts such as “the Space Needle is in Seattle” to MLPs in the middle layers of GPT-2 XL.

6.5 What the neurons hold

Each hidden neuron does one small thing: it folds the plane along a line and fires on one side of it. The output adds up the folds, each with its own weight. Switch some off and see which ones the answer leans on.

ReadingTrain the net in Fig. 6.6 first; untrained, the neurons fold at random.

Try this 0 of 2

to do: Switch off the neuron with the longest bar.
to do: Switch off half of them. How much survives?

Fig. 6.7The hidden neurons, one by one.Each plate is one neuron of the net in Fig. 6.6: ink where it fires, and its fold in blue, the line where its input crosses zero. Below, the weight it writes into the score: blue bars push towards the blue side, ink bars towards the grey. Press a plate to switch the neuron off.

In a GPT the picture is the same, in 768 dimensions instead of 2. Each neuron reads the token through its row of c_fc: a key, the pattern it responds to. When it fires, its column of c_proj, a value, is written back into the stream. Researchers have found MLP neurons that respond to particular words, topics and facts, which is why the MLP is often called the network's memory.

GPT-2 small has 3,072 of these neurons in each of its 12 layers: 36,864 small detectors, each reading the stream and writing back a note.

Exhibits after Andrej Karpathy, nanoGPT (the MLP and the Block) and micrograd (the two moons), MIT. The token numbers in Figs. 6.2 and 6.3 are small stand-ins made from your tokens' text; the arithmetic is the real thing.

Once, before transformers, he raced one of these machines himself.

Interlude2 September 2014 · between Chapters 6 and 7

Frontispiece. A Norwich terrier. Or is it a Norfolk?

Racing a ConvNet

In 2014 he labelled 1,500 photographs by hand, to find out how a person scores against the best machine. Take his seat.

After Andrej Karpathy, What I learned from competing against a ConvNet on ImageNet (2 September 2014). The full study is in Russakovsky and colleagues, ImageNet Large Scale Visual Recognition Challenge (2015), with Karpathy among the authors.

ImageNet was the exam every vision system sat in those years. By 2014 the machines scored so well that an obvious question had no answer: how well does a person do? He was a PhD student at Stanford. He built himself a labelling tool and sat the exam.

ILSVRC, the ImageNet Large Scale Visual Recognition Challenge, ran every year from 2010 to 2017. GoogLeNet won the 2014 round.

The exam

Each test photograph shows something from one of a thousand classes, and a system may give five guesses. For a machine that is a fair exam. For a person it is a strange one: a thousand classes is too many to hold in your head, and more than 120 of them are dogs.

classes: 1,000
training photographs: 1.2 million
test photographs: 100,000
guesses allowed: 5

dogs, 121 of the 1,000

One test photograph, five guesses

The label: Norwich terrier

1Norfolk terrier
2Australian terrier
3Norwich terrier
4cairn
5Border terrier

Counted right: the label is among the five.

Fig. A.1The exam.Each square is one of the thousand classes, in the order of the ImageNet hierarchy. Shaded: the dogs, 118 domestic breeds in one long run and three wild dogs after the wolves. Each photograph has one label, and a system may make five guesses; it is wrong only if all five miss. The best machine of 2014, GoogLeNet, missed on 6.7% of the test photographs.

His tool: the photograph on the left; the thousand class names on the right, in the order of the hierarchy, each with 13 training photographs; and a search box.

He practised on 500 photographs, then labelled 1,500 test photographs, at about one a minute.

Take his seat

Ours keeps his 13 examples per name but cuts the thousand names to six: three pairs of breeds that are easy to mix up. The machine you race is small and honest. It learns here, in your browser, from 160 drawings of each breed; in the real exam the ConvNet had studied about 1,200 photographs of every class. You get the 13 on the study plates, as he did. Open them before you start, or during the race, while the clock runs.

Fine-grained12 photographs

You

You: right 0of 0

The machine

The machine: right 0of 0

First the machine studies 960 drawings, 160 of each breed: about a second. Then the clock runs on each photograph. Keys 1 to 6 answer.

Fig. B.1The race.Twelve photographs, two of each breed. Your time runs from the moment a photograph appears; the machine's is measured on your computer too, from the moment it is handed the photograph to its answer. The breeds are real ImageNet classes, drawn flat, their tells simplified from the breed standards.

I became very good at identifying breeds of dogs.
Andrej Karpathy, What I learned from competing against a ConvNet on ImageNet, 2014

This is the linear classifier of his Stanford course CS231n, whose notes read each class's weights as a template. A ConvNet stacks many layers before a last one like this.

What the machine learned

How did it get good so fast? It has one layer of weights: for each breed, one number for each of the 1,200 squares of a photograph. Training nudged those numbers until the right breed's sum came out highest. Print the weights back where they sit and they look like dogs.

Norfolk terrier

Norwich terrier

Pembroke Welsh corgi

Cardigan Welsh corgi

Siberian husky

Alaskan malamute

It starts knowing nothing: every weight zero, six blank plates.

Fig. C.1Six templates.The machine's weights, one plate per breed, printed where they sit on the photograph, each square as big as its weight. Ink squares: ink there counts for the breed. Yellow squares: ink there counts against it. Compare the two terriers: their strongest squares sit on the head, where one has ears standing and the other has them folded down.

Once the machine has studied, feed it a photograph here and watch it decide.

Fig. C.2How it decides.Every square of ink is multiplied by the weight printed at the same place on a breed's template, and the products are added up, one sum per breed (after subtracting the average photograph). The largest sum wins; softmax turns the sums into the percentages. That is all it does, and it does it in a fraction of a millisecond.

His examples of small, thin things it missed: a person holding a quill in their hand, a small ant on a stem.

Training on flipped, cropped and recoloured copies of the photographs, data augmentation, is one way ConvNets were taught to shrug off such changes.

Where you win

Now the same machine and the same six breeds, photographed the way the world photographs things: behind a fence, through a filter, as a sketch, far away, as a toy, in stone. You see a dog. It sees ink where it did not expect ink.

In context6 photographs

You

You: right 0of 0

The machine

The machine: right 0of 0

First the machine studies 960 drawings, 160 of each breed: about a second. Then the clock runs on each photograph. Keys 1 to 6 answer.

Fig. D.1Six photographs in context.The filter, the sketch, the toy, the statue and the far-off dog are kinds of photograph he found GoogLeNet missing; the fence is ours. Under each answer, the photograph as the machine received it. Its templates expect ink where a dog's body usually is, so a fence, a dark filter or a plinth outvotes the ears and the tail you were looking at.

The ConvNets of 2014 were far better than this one-layer machine, but they failed in the same directions for the same reason: they had learned the statistics of their photographs, not what a dog is.

He also found about 5 of the 1,500 photographs (0.3%) labelled wrongly in the exam itself.

His tally

He did not stop at the score. He sorted every mistake, his and GoogLeNet's, by its cause, and the two lists are very different. The machine lost on context. The person lost on dog breeds, and on the sheer length of the list.

Fine-grained

Dogs, monkeys, snakes, birds: telling a breed or species from its neighbours. His biggest weakness by far, with more than 120 kinds of dog in the exam.

Fig. E.1His tally.He went through every miss on his 1,500 photographs, about a hundred of GoogLeNet's and about seventy-five of his own, and sorted them. Each bar is a share of that one's own errors. “No count” marks a kind that belongs to the person alone: a machine scores every class, so it cannot overlook one, and it had studied far more than 13 photographs of each. Press a row for an example.

On his 1,500 photographs: 5.1% for him, 6.8% for GoogLeNet. He put the chance of a gap that size by luck at p = 0.022.

Scores on the right are each year's best published top-5 error on the full test set.

Not a point, a curve

His 5.1% was not the human score. It was one human at one level of effort. A second labeler, with less training, got about 12%. He guessed that a team of dedicated experts might reach 3%, and that 2% was an optimistic floor.

Human accuracy is not a point. It lives on a tradeoff curve.
Andrej Karpathy, What I learned from competing against a ConvNet on ImageNet, 2014

Turn the calendar

GoogLeNet top-5 error 6.67%

GoogLeNet, Sep 2014. 3 of the 4 points on the human curve still beat it.

He beats it, after 500 photographs of practice and about a minute a photograph.

12% second labeler, less training, about 280 photographs
5.1% Karpathy, 500 for practice, then 1,500 at about a minute each
~3% a team of dedicated experts, his estimate
~2% an optimistic floor, his estimate

Fig. F.1Not a point, a curve.Left, people: top-5 error falls as the effort rises, from his second labeler to his own score to his two estimates. Right, the best machines, year by year, on the full test set. Lower error is higher on the page. The dashed level line carries the machine's score across to the people: the points above it, filled yellow, still beat the machine. Turn the calendar.

The tricks that beat him are in this book: Kaiming's initialization and BatchNorm in Chapter 5, residual connections in Chapter 4. He saw it coming. Soon, he wrote, a person would beat these models only with significant effort, expertise and time.

After Andrej Karpathy, What I learned from competing against a ConvNet on ImageNet (2014), and Russakovsky and colleagues (2015). Later scores from the papers: He and colleagues (February 2015), Ioffe and Szegedy (February 2015), He and colleagues (ResNet, December 2015), Hu and colleagues (SENet, 2017). The photographs here are drawn, and the machine is a one-layer linear classifier trained in your browser on drawings.

Everything so far was the forward pass. Learning happens on the way back.

Chapter 7The backward pass

Backprop

The chain rule, run backward through every operation.

Fig. 7.1The backward pass. The gradient of the loss flows back along every wire, multiplied at each operation by its local derivative.

After Andrej Karpathy, The spelled-out intro to neural networks and backpropagation: building micrograd (MIT). micrograd is about a hundred lines of Python; every figure here runs its arithmetic in your browser.

The forward pass ends in one number, the loss: how wrong the network was. To do better, each of GPT-2's 124 million weights needs to know one thing. If I nudge you up a little, does the loss go up or down, and by how much? Backpropagation answers that for every weight at once, in one sweep backward, with nothing more than the chain rule from school calculus. This chapter builds it from a single nudge.

His lecture opens here, before any neural network: a plain function, a tiny h, and rise over run. No rules of calculus needed.

7.1 The nudge

Start where he starts, with a plain function of one number. To find out what its derivative at some x means, skip the rules: nudge x by a tiny amount h and watch how far f moves. The ratio of the two moves is the slope. Drag the pen along the curve.

the slope, measured by nudging 17.00

rise ÷ h: the slope, measured by nudging

x: 3.00
f(x): 20.000
f(x + h): 37.000
true slope 6x − 4: 14.000

Uphill: nudge x up and f rises, about 14 times as much.

Slide x

The nudge h

Try this 0 of 3

to do: Shrink the nudge to 0.001: the two lines become one.
to do: Find the bottom of the bowl, where the slope is 0.
to do: Find a place where the slope is negative.

def f(x):
    return 3*x**2 - 4*x + 5

h = 1
x = 3
(f(x + h) - f(x)) / h    # 17.000

Fig. 7.2The nudge.Push x up by a small h and see how far f moves. Rise over h is the slope, the derivative, written df/dx. With a big nudge the red line (through both points) misses the true slope, the dashed tangent; as h shrinks the two become one.

That ratio is the whole question of training: for each weight, how much does the loss move per nudge?

In symbols, dz/dx = dz/dy · dy/dx. The car, the bicycle and the man are George F. Simmons's, as quoted in the Wikipedia article on the chain rule.

7.2 Rates multiply

A network is a long chain of small operations, each feeding the next. The chain rule says how a nudge travels down such a chain: the rates multiply.

Andrej explainsThe chain rule, intuitively: car, bicycle, man

If a car is twice as fast as a bicycle, and a bicycle is four times as fast as a man, then the car will be eight times as fast as the man. We can take these intermediate rates of change and multiply them together, and that justifies the chain rule intuitively.

Andrej Karpathy, The spelled-out intro to neural networks and backpropagation: building micrograd, 43:31 to 43:49

Walk the man

bicycle per man

car per bicycle

d car / d man = d car / d bicycle × d bicycle / d man = 2 × 4 = 8

Fig. 7.3Rates multiply.Three rulers on one scale. Whatever the man does, the bicycle does 4 times over and the car 2 times the bicycle, so the car does 8 times the man. Press the button to nudge him and read the three brackets.

Written out properly, it is the same multiplication.

Andrej explainsWhat the chain rule actually says

If a variable z depends on a variable y, which itself depends on a variable x, then z depends on x as well, through the intermediate variable y. If you want dz by dx, you take dz by dy and multiply it by dy by dx. That's really what the chain rule is telling us: to differentiate through a function composition, we apply a multiplication of those derivatives.

Andrej Karpathy, The spelled-out intro to neural networks and backpropagation: building micrograd, 42:13 to 42:52

Backprop runs the chain from the far end. Start at the loss with a rate of 1, then multiply by each local rate on the way back.

A micrograd Value holds its data, its grad, the values it was made from, and a small _backward function. That is the whole engine.

Using a value twice is a bug he walks into in the lecture: with = instead of +=, the second write erases the first.

7.3 backward(), one node at a time

Now a real expression, his first: L = (a·b + c)·f. Each box is a micrograd Value. It holds its data, and its grad: how much L moves per nudge of it. Every grad starts at 0. backward() visits the operations from the output back to the inputs, and at each one does a single multiplication: the operation's own local derivative, times the grad that has arrived from above.

Andrej explainsA plus node's local derivative, chained backward

How does c impact d? This is a plus node, so the local derivative is simply 1.0. The chain rule tells us that dl by dc, going through this intermediate variable, will just be dl by dd times dd by dc. That's chain rule.

Andrej Karpathy, The spelled-out intro to neural networks and backpropagation: building micrograd, 44:21 to 44:56

Press the button and read the ledger. A + passes the grad through unchanged. A × hands each input the other input's value, times the grad. Switch on Guess first to work each step out before the machine does.

Which graph

adata2.00grad0.00

bdata−3.00grad0.00

edata−6.00grad0.00

cdata10.0grad0.00

ddata4.00grad0.00

fdata−2.00grad0.00

Ldata−8.00grad0.00

×+×

for v in reversed(topo):

All grads are zero. Press to start backward().

The ledger: each step, the chain rule

Empty. Every grad starts at 0.

# out = self + other
def _backward():
    self.grad += 1.0 * out.grad
    other.grad += 1.0 * out.grad
# out = self * other
def _backward():
    self.grad += other.data * out.grad
    other.grad += self.data * out.grad

Try this 0 of 3

to do: Step backward() through his example to the end.
to do: Switch on Guess first and get one right.
to do: Choose a, used twice; switch off += and run it. Read the check.

Fig. 7.4backward(), stepped by hand.Black arrows: the forward pass, which made the numbers. Red arrows: the step just taken, the grad flowing back from an operation into its inputs, each wire marked with the local derivative it multiplies by. When it is done, every leaf is checked by nudging it, as he does in the lecture.

Each operation knows only its own little derivative. Chained by backward(), they give the exact gradient of every input, and the nudges agree to the third decimal.

His b, 6.8813735870195432, is chosen so that tanh comes out at 0.7071, where its slope 1 − o² is exactly 0.5.

tanh's local derivative is the slope of chapter 5's squash.

7.4 One neuron

A neuron is the same machine with one more kind of node. It multiplies each input by a weight, adds a bias, and squashes the sum with tanh. These are his numbers from the lecture.

x1data2.00grad0.00

w1data−3.00grad0.00

x2data0.00grad0.00

w2data1.00grad0.00

x1w1data−6.00grad0.00

x2w2data0.00grad0.00

x1w1+x2w2data−6.00grad0.00

bdata6.88grad0.00

ndata0.88grad0.00

odata0.71grad0.00

××++tanh

for v in reversed(topo):

o
n
b
x1w1+x2w2
x2w2
w2
x2
x1w1
w1
x1

All grads are zero. Press to start backward().

The ledger: each step, the chain rule

Empty. Every grad starts at 0.

# out = self + other
def _backward():
    self.grad += 1.0 * out.grad
    other.grad += 1.0 * out.grad
# out = self * other
def _backward():
    self.grad += other.data * out.grad
    other.grad += self.data * out.grad
# out = self.tanh(); t = out.data
def _backward():
    self.grad += (1 - t**2) * out.grad

Fig. 7.5A neuron, backward.o = tanh(x1·w1 + x2·w2 + b). The tanh node passes back 1 − o² of what reaches it; then the two + nodes pass it on unchanged, and the two × nodes hand each weight its input's value.

Look at w2: its gradient is 0, because its input x2 is 0. With nothing coming in, that weight cannot move the output, and the gradient says so. w1's gradient of 1 says: raise w1 a little and the output rises by as much.

“You forgot to .zero_grad() before .backward()” is number 3 on his list of the most common neural net mistakes (tweet, 2018).

“If this makes sense, you understand backpropagation.” Hacker's guide to Neural Networks, 2012.

7.5 You are the optimizer

Training is this, over and over: run the network, backprop the loss, move every weight a little against its gradient. Here is a neuron with one input and just two weights, w and b, so every setting of them fits on one map. Be the optimizer first: follow the arrows by hand. Then let the machine take over, and then break it.

loss 8.363

the loss: the five misses, squared and summed

The neuron's output o = tanh(w·x + b)

Lever w −1.20

∂loss/∂w −3.65downhill: raise w

Lever b 1.80

∂loss/∂b −1.64downhill: raise b

Learning rate

steps 000

ReadingThe red arrow on the map points where the machine would step next: against the gradient, scaled by the rate.

for k in range(steps):
    loss = sum((n(x) - y)**2 for x, y in data)
    # zero_grad
    w.grad = 0.0; b.grad = 0.0
    loss.backward()
    w.data += -0.1 * w.grad
    b.data += -0.1 * b.grad

Try this 0 of 4

to do: Move a lever the way its arrow points. The loss falls.
to do: Press to train, and watch the machine do what you did.
to do: Turn the rate up to 1.0 and train again.
to do: Switch off zero_grad, then train. It never settles.

Fig. 7.6You are the optimizer.Left, every setting of one neuron's two weights, printed like a survey map: each line a step up in the loss, the red screen the floor of the valley, the cross its lowest point. Right, what the neuron does with the dot's weights: the red curve should pass through the five squares. The gradient points uphill; training walks the other way.

A GPT does exactly this, with 124 million levers instead of two, on a map no one can draw. backward() supplies every arrow, and every arrow points uphill; the update walks the other way.

Exhibits after Andrej Karpathy, micrograd (MIT): its Value, its three _backward functions, build_topo, and the lecture's first function, expression, neuron and training loop. The five examples in Fig. 7.6 are ours.

Chapter 8Many passes

Training

Many passes, each nudging every weight downhill.

Fig. 8.1One optimizer step. Every weight moves a little against its gradient; then the pass runs again, thousands of times.

After Andrej Karpathy, makemore Parts 1 and 2 (MIT), and his essay A Recipe for Training Neural Networks (2019). Every curve in Figs. 8.4 to 8.9 is computed in your browser when you press.

A network starts out as random numbers. Training is the loop that makes them useful: guess, measure how wrong the guess was, ask every number which way it should move, and nudge them all a little. Then again, thousands of times. This chapter runs that loop on a list of names, and then goes through his recipe for keeping it honest.

ln is the natural logarithm. The same number goes by two names: negative log likelihood, and cross-entropy, PyTorch's F.cross_entropy.

Averaging the logs is multiplying the probabilities: a low loss means the model finds the whole list likely.

8.1 How wrong? The loss

Before a network can get better it needs a score. Its job here is to guess the next letter of a name. It answers with a bet on each of the 27 characters, a to z and a dot for the end, and the score looks only at the bet it placed on the letter that really came next.

Andrej explainsWhy loss functions want low, not high

What we'd like is a loss function, and a loss function has the semantics that low is good, because we're trying to minimize the loss. So we need to invert this, and that's what gives us something called the negative log likelihood.

Andrej Karpathy, The spelled-out intro to language modeling: building makemore, 56:37 to 56:56

Slide the bet on the right letter

loss 1.39

the loss for this one letter

bet on the right letter: 25%
as odds: 1 in 4.0

Better than a blind guess, but not by much.

Fig. 8.2The price of a guess.The model spreads its bets over all 27 characters; the loss looks only at the bet on the letter that really came next, and charges −ln p. Certain and right costs 0. A blind guess, one chance in 27, costs ln 27 = 3.30. A confident miss costs without limit.

A name is a row of such guesses, one per letter, and its loss is their average. Score one: first with a model that knows nothing, then with a table counted from our list of names.

Type a name

The model

Try this 0 of 4

to do: Switch to the blind guess.
to do: Type andrej, his own name, under the counted model.
to do: Now add one to every count.
to do: Find a name that costs less than 2.

average loss 2.30

average loss per letter, the number training lowers

An ordinary name by the numbers.

Fig. 8.3Scoring a whole name.Each column is one prediction: the next character, given the one before it, scored by −ln p. The “.” marks the start and the end of a name. The counted model is a table of how often each letter follows each other in our list of 250 rows; with one added to every count, nothing is impossible any more.

The gradient comes from loss.backward(): Chapter 7, run on these 729 numbers.

In makemore Part 1 he turns the rate up step by step and settles on 50 for this net.

8.2 Downhill

Counting gave us a table. A neural network gets there by another road, and it is the road that scales: start from random numbers, compute the loss, let backpropagation say which way each number should move to lower it, and move every number one small step that way. The size of the step is the learning rate.

Here is his first neural net from makemore: 27 by 27 weights, one row for each letter, trained by exactly that loop on all 250 rows of the list.

Learning ratelr 50

step: step 000
loss: loss 3.30

Try this 0 of 4

to do: Press to train at 50, the rate he uses.
to do: Draw five names before training, and again after.
to do: Start over at 0.5. Patience.
to do: Now try 500.

W = torch.randn((27, 27)) * 0.1  # 729 weights
for k in range(300):
    logits = xenc @ W  # a row per letter
    probs = logits.softmax(1)
    loss = -probs[torch.arange(n), ys].log().mean()
    W.grad = None
    loss.backward()
    W.data += -50 * W.grad

ReadingUntrained: 729 random numbers near zero, every letter about equally likely after every other. The loss is 3.30, about the price of a blind guess.

Fig. 8.4Downhill.The loss over all 1,726 letter pairs in the list, once per step. The upper rule is the blind guess, ln 27 = 3.30; the lower is the loss of the counted table, 2.23, the best any table of pairs can do on this list. Earlier runs stay printed in grey.

The network softmax(W), step 0

Counted from the list

the two tables agree: 30%

Fig. 8.5Two tables.Rows: a letter (the dot marks the start of a name). Columns: the letter after it. Each dot's area is the probability, as a halftone prints a grey. Train the network and its green table grows into the counted one, though nobody showed it the counts.

Nobody showed the network the counts. It found the same table by walking downhill, and the walk works just as well for a net with billions of weights, where no table could ever be counted.

“When you sort your dataset descending by loss you are guaranteed to find something unexpected, strange and helpful.” His tweet, 2020.

The recipe's first step touches no neural net code at all: it is hours spent reading the data.

8.3 Become one with the data

His recipe opens before any training: look at the data, row by row, until you know it. One of his favourite ways in is to let a model read the data, then sort it by loss. Our list has 250 rows. Two of them are not names.

Order of the list

Try this 0 of 2

to do: Sort the list by loss.
to do: Strike out every row that is not a name.

001emma2.55
002olivia2.43
003ava2.33
004isabella2.41
005sophia2.31
006liam2.41
007noah3.18
008charlotte2.70
009william2.62
010mia1.92
011james2.47
012amelia1.99

ReadingRows 1 to 12 of 250, in the order they were collected. Nothing looks wrong. Nothing ever does, this way.

Fig. 8.6The list, sorted by loss.The loss per letter of each row, scored by a counting table built from the other 249 rows, so that no row can vouch for itself. Two of the 250 rows were planted, as every real dataset has a few. Sorted as collected they hide at rows 61 and 142.

The net is his from makemore Part 2: two numbers for each letter, a hidden layer of 100 tanh neurons, 27 outputs. 3,481 parameters in all.

He hits the same floor there, and explains it the same way.

8.4 Overfit one batch

Before training on everything, check that the net can learn anything at all. Take one small batch and train on it alone, over and over. A working net and a working loop will memorize it, and its loss will head for zero. If it does not, something is broken, and you have found out in seconds rather than in days.

Andrej explainsOverfit a single batch first, as a sanity check

We make sure that we can overfit a single batch, and we are getting to a very, very low loss: the Transformer is memorizing this single individual batch. The learning rate here is a pretty good default for most optimizations at a very early debugging stage.

Andrej Karpathy, Let's reproduce GPT-2 (124M), 1:01:16 to 1:01:41

step: step 0000
loss: loss 3.30
sure (> 90%): cards above 90 percent 00

# 3,481 parameters; the first five names
Xb, Yb = X[:32], Y[:32]
for k in range(1000):
    loss = F.cross_entropy(mlp(Xb), Yb)
    for p in parameters: p.grad = None
    loss.backward()
    for p in parameters: p.data += -0.1 * p.grad

ReadingUntrained: every card near 4%, one chance in 27.

... then e4%
..e then m4%
.em then m4%
emm then a4%
mma then .4%
... then o4%
..o then l4%
.ol then i3%
oli then v4%
liv then i4%
ivi then a3%
via then .4%
... then a4%
..a then v3%
.av then a3%
ava then .4%
... then i4%
..i then s4%
.is then a3%
isa then b4%
sab then e4%
abe then l4%
bel then l4%
ell then a4%
lla then .4%
... then s4%
..s then o3%
.so then p3%
sop then h4%
oph then i4%
phi then a4%
hia then .4%

Fig. 8.7Thirty-two examples, memorized.Every card is one example from the first five names: three characters in, the next one out, and the network's probability for the right answer. The cards ringed in green share one input, the start of a name.

Dev, or validation: names held back from training, used to check and to tune. In makemore he splits 80, 10 and 10 into train, dev and test; we keep it to 80 and 20.

“Adding more data is pretty much the only guaranteed way to monotonically improve the performance of a well configured neural network.” A Recipe for Training Neural Networks.

8.5 Learning, or memorizing?

Memorizing was the point of one batch. On the whole list it is a failure: a model that knows these names by heart has learned nothing about names. So hold some back, never train on them, and measure the loss on them as well.

Andrej explainsMore capacity means more capacity to overfit

As the capacity of the neural network grows, it becomes more and more capable of overfitting your training set. That means the loss on the training set will become very, very low, as low as zero, but all the model is doing is memorizing your training set verbatim.

Andrej Karpathy, Building makemore Part 2: MLP, 53:50 to 54:11

Neurons in the hidden layer

step: step 0000
train: train loss 3.30
dev: dev loss 3.30

Try this 0 of 3

to do: Train the net with 2 neurons.
to do: Then 16.
to do: Then 128, and watch the green curve.

Reading2 neurons, 413 parameters, 1,365 training examples. Press to train.

Results so far, by width
neurons	parameters	train	dev	best dev
2	413	·	·	·
16	1,225	·	·	·
128	7,721	·	·	·

Fig. 8.8Train and dev.The same MLP at three widths, ten numbers for each letter, three letters in, trained on 198 names in batches of 32; every 100 steps both losses are measured on the whole of each set. Only the dev curve says whether it has learned what names look like.

Three of the bugs are on his 2018 list of the most common neural net mistakes: forgetting .zero_grad(), passing softmaxed outputs to a loss that expects raw logits, and forgetting train or eval mode.

The others come from his recipe (verify the loss at init; an input-independent baseline) and from makemore (the hockey stick, the learning rate).

8.6 Case files

“Neural net training fails silently,” he writes. A wrong shape throws an error. A wrong idea trains anyway, a little worse, and says nothing. The one witness is the loss curve, so learn to read it. Each run below has one bug. Read the note, look at the curve, name the bug.

Case files Five training runs, each broken in one way. No errors, no warnings: just a loss curve, and a colleague's note.

Fig. 8.9Case files.

“Networks keep training for an unintuitively long time.” A Recipe for Training Neural Networks.

8.7 The same loop, at scale

Nothing in this chapter needs a new idea to grow. When he reproduces GPT-2, the loop is the one in Fig. 8.4 with a bigger network and a better optimizer, and one of the first numbers he checks is from 8.1: with 50,257 tokens to choose from, a blind guess costs ln 50,257 = 10.82, about where a well-started run should begin.

Which run

parameters: 729one 27 × 27 table
choices for what comes next: 27characters
a blind guess costs: 3.30ln 27
read in one step: 1,726letter pairs
steps: 300plain gradient descent
read in all: 517,800letter pairs

Fig. 8.10The same loop, scaled.His reproduction of GPT-2 (124M), from Let's reproduce GPT-2 and his build-nanogpt code: 524,288 tokens a step for 19,073 steps. The loop inside is the one in Fig. 8.4: forward, loss, backward, step.

Forward, loss, backward, step. Then again, nineteen thousand times.

Exhibits after Andrej Karpathy, makemore Parts 1 and 2 (MIT), and A Recipe for Training Neural Networks (2019). The list is 248 popular US given names, standing in for the 32,033 in his names.txt, plus two planted bad rows.

The same nudging, with no labels at all: only a score at the end of the game.

InterludeFrom his blog · 31 May 2016

Pong from pixels

A program that learns a video game from nothing but the screen and the score, by guessing and checking. He wrote it in 130 lines of Python; here it is small enough to learn on this page.

After Andrej Karpathy, “Deep Reinforcement Learning: Pong from Pixels” (2016), karpathy.github.io, and his script pg-pong.py. Every exhibit below is a real policy network running in your browser.

His first line: “RL is hot!”

Nobody tells the program how Pong works. It sees the screen, moves its paddle up or down, and now and then the score changes. From that alone it has to work out what it did right. Chapter 8 trained a network with a teacher who knew every answer. Here there is no teacher, only a score. His whole recipe fits in three sentences:

Run a policy for a while. See what actions led to high rewards. Increase their probability.

Andrej Karpathy, “Deep Reinforcement Learning: Pong from Pixels” (2016)

The Atari's frame is 210 by 160 in colour. He crops away the score, keeps every second pixel, erases the background and sets everything left to 1: an 80 by 80 board, 6,400 numbers.

Ours is 16 by 12, 192 numbers, so that it learns in a minute instead of three nights.

There is no button for standing still. Like his, our agent moves UP or DOWN on every frame, so its paddle always shows in the difference.

P.1 What it sees, and the coin it tosses

The program does not see a ball or a paddle. It sees ones and zeros. And one frame is not enough: from a single picture you cannot tell which way the ball is going. So the network is fed the difference between this frame and the last. Step through a few frames and find the ball in both boards.

frame no. 0000

Try this 0 of 3

to do: Press for the next frame.
to do: Play ten frames and watch the paddle wander.
to do: Catch a toss that went against the odds: DOWN when p is over 50%, or UP when it is under.

Frame t ones and zeros

Frame t − frame t−1 what the network is fed

Fig. P.1What the network sees.Left, the screen after his preprocessing: 1 wherever a paddle or the ball is, 0 everywhere else. Right, what the network is actually given: this frame minus the last. A thing at rest vanishes from it; a moving thing leaves a solid square where it arrived and a hollow one where it left, so the direction of the ball is right there in the numbers.

The policy is a small network: the 192 numbers of the difference frame in, one hidden layer of 16 neurons, one number out. A sigmoid turns that number into a probability, the chance of moving up. Then the program tosses a coin weighted by it. Untrained, every coin is close to fair, so the paddle wanders at random; that wandering is how it ever stumbles on a point worth repeating.

192 pixels

16 neurons, ReLU

z = 0.00sigmoidp = 0.50chance of UP

chance of UP, percent 50%

chance of UP on this frame

The toss: u = 0.00 ≥ p, so the paddle moves

Fig. P.2One frame of the loop.The difference frame goes in; sixteen neurons respond (darker squares fire harder); their weighted sum z goes through a sigmoid and comes out as p, the chance of moving up. Then the coin: a uniform random number u, from 0 to 1, lands on the ruler. Below p means UP, above means DOWN. His network had 6,400 inputs and 200 neurons, 1,280,200 weights; this one has 192 and 16, 3,088.

In his code the gradient is y - aprob, collected for every frame, and later multiplied by each frame's share of the reward. His comment on that line: “PG magic happens right here.”

PG is policy gradients; this is the plainest kind, known as REINFORCE (Ronald Williams, 1992).

P.2 Guess and check

Supervised learning would need someone to say which move was right on every frame. Nobody can, so he cheats, cheerfully: he pretends the move the agent happened to make was the right answer, and computes the ordinary gradient that would make it more likely. Then he waits to see how the point ends, and scales that gradient by the result. Won: step along it, do more of that. Lost: step against it, do less.

How did the point end?

updates 00

Try this 0 of 3

to do: Mark the point won and apply the update three times.
to do: Start again, mark it lost, and apply it three times.
to do: Push p past 90%. Near the top the steps shrink, because y − p does.

It moved UPso the label is y = 1

chance of UP on this frame, percent 50%

chance of UP on this frame, from 50% at the start

dlogp = y - aprob    # 1 − 0.50 = +0.50
dlogp *= advantage   # won: +0.50

Fig. P.3One move, nudged.A real gradient through the whole network, all 3,088 weights at once, with a learning rate large enough to see a single step. The direction always comes from the label, the move it made; only the sign of the advantage decides whether to step along it or against it. Earlier values of p are left on the scale as small ticks.

Supervised learning and policy gradients, side by side
Aspect	Supervised learning	Policy gradient
The label	the right answer, from a teacher	whatever move it happened to make
Each example	counts the same, always	counts by its advantage: up if things went well, down if badly
The data	a fixed set, collected beforehand	its own games, played by its current self

That is the whole trick: policy gradients are supervised learning on your own guesses, with every example weighted by how things turned out.

This is the credit assignment problem. His γ is 0.99, so a move 100 frames before the score still gets about a third of it (0.99¹⁰⁰ ≈ 0.37).

His code resets the sum at every point, with the comment “pong specific!”: a point is over when it is over.

P.3 Who gets the credit?

A point is decided many frames before it is scored. The move that mattered might be the one that met the ball, a whole crossing earlier. The program has no idea which, so it spreads the credit backward: each move gets the reward times γ for every frame between it and the score. Then, per game, he standardizes the shares to mean 0 and spread 1, so that in every game some moves are pushed up and some pushed down.

10 frames before: 0.90 of it
30 frames before: 0.74 of it
moves encouraged: 0
moves discouraged: 0

Try this 0 of 3

to do: Turn γ down to .5: only the last few moves before each point share the blame or the credit.
to do: Turn it up to 1: every move of a point gets the same full share.
to do: Switch on standardize: the bars recentre on zero, so every game has moves pushed up and moves pushed down.

Fig. P.4Credit, assigned blindly.Two points from a real game, the first won and the second lost. Each bar is one frame's share of the reward, and the tick above it is the move made there, up or down. Nothing in the program knows which move mattered: every move before a score gets a share, shrinking by γ for each frame of distance, and the sum starts over at every point.

The same algorithm as his: the update after every two games here, after ten in his; RMSProp; γ 0.99. The opponent is our own simple paddle that chases the ball at a limited speed.

Training runs only while you watch: it pauses when this figure leaves the screen.

P.4 Practice

Now let it practise. Press the button and the program plays game after game against the computer, first to 21, and updates its weights after every second game. Watch the running reward, his measure of progress. It starts near −21, a whitewash, because a random paddle returns the ball only by luck.

ready

Practice speed

Try this 0 of 4

to do: Press to train.
to do: Watch the running reward climb past −10: it has learned to follow the ball.
to do: Wait for it to cross zero: from then on it beats the computer.
to do: Forget everything and train again. A new start takes a different road to the same place.

ReadingNever trained. It moves at random and returns the ball only by luck.

0computeragent0

Fig. P.5The board.One game at watching speed, played with the weights as they are this moment; behind it the trainer plays far faster. The agent is the red paddle.

running reward −21.0

running reward, his measure: the score margin of a game to 21, averaged over recent games

games played: 0
updates: 0
returns the ball: 0%
wins the point: 0%

Fig. P.6Running reward.Each game is first to 21, so the margin runs from −21, every point lost, to +21; above even, it beats the computer. His agent, on the real Atari, needed about 8,000 games, three nights on his MacBook, to edge past the computer. On this small board it takes several hundred, a minute or two at full speed.

Fig. P.7What it learned.Every bar is its chance of UP, averaged over the frames of its last few games when the ball was coming toward it, sorted by where the ball was compared with its paddle. Untrained, every bar sits near 50%. Nobody told it to follow the ball. The score did.

It shouldn't work, but amusingly we live in a universe where it does.

Andrej Karpathy, “Deep Reinforcement Learning: Pong from Pixels” (2016)

His advice before reaching for anything clever: “One should always try a BB gun before reaching for the Bazooka.”

P.5 Profoundly dumb, and it works

Step back and the method is almost embarrassing. It never learns what a ball is. It tries things at random and keeps whatever happened to come before a win, one probability at a time. He calls it “a fancy form of guess-and-check”, and does not dress it up:

At the core the approach we use is also really quite profoundly dumb.

Andrej Karpathy, “Deep Reinforcement Learning: Pong from Pixels” (2016)

A person gets the idea of Pong in a game or two, because we arrive knowing about objects, motion and goals. His agent knew none of it and needed about 8,000 games. What it has instead is patience and a gradient. And the loop it runs, try, score, make the good tries more likely, did not stay in Pong: it is the third stage of training a chat model (Chapter 10), where the model practises on problems whose answers can be checked.

Guessing is cheap. Checking is what teaches. Give a learner a score it can trust and enough tries, and it will find its way.

Exhibits after Andrej Karpathy, “Deep Reinforcement Learning: Pong from Pixels” (2016) and pg-pong.py: the same algorithm, REINFORCE with discounted, standardized rewards and RMSProp, on a smaller board (16 by 12, not 80 by 80), a smaller network (16 neurons, not 200), a larger learning rate and batches of two games, not ten. The court and its opponent are our own, not the Atari's.

At the bottom of the stack, a list of numbers has to become a choice.

Chapter 9Output

Sampling

Scores become probabilities, and a die is rolled.

Fig. 9.1The head. A last linear layer gives one score, a logit, for every entry in the vocabulary; softmax turns the scores into probabilities.

After Andrej Karpathy, The spelled-out intro to language modeling: building makemore, and nanoGPT (MIT). Every exhibit in this chapter runs his first model, the bigram, on the counts from his names.txt.

Your token, ␣of, has come out of the last block as 768 numbers. The head turns them into 50,257 scores, one for every token GPT-2 knows, and then something has to choose. This chapter is about that choice: how scores become chances, how the die is rolled, and the two knobs, temperature and top-k, that decide how adventurous the roll is.

His first model, in makemore Part 1: for every letter, count which letter follows it, over all 32,033 names in his file. 228,146 pairs in all.

He adds 1 to every count, so nothing is ever impossible. The scores here are the logarithms of those counts.

A GPT computes its scores with the whole network instead. What happens after is the same, 50,257 columns wide.

9.1 Twenty-seven scores

A small model is easier to see whole. So set the GPT aside for a moment and take one that only knows names, a letter at a time. Given the letter before, it hands back 27 scores: one for each letter, and one for the end of the name, written as a dot. The scores can be any numbers at all. To make them into chances takes two steps.

Andrej explainsSoftmax: logits to fake counts to probabilities

We multiply this in the first layer of a neural net to get logits, exponentiate the logits to get fake counts, and normalize these counts to get probabilities. These last two lines are called the softmax: it takes the logits, exponentiates them, and divides and normalizes.

Andrej Karpathy, The spelled-out intro to language modeling: building makemore, 1:27:12 to 1:27:46

Raise e to each score, so that every one becomes a positive “fake count”; then divide by the total, so that they add up to 1. Pick a letter on the keys and follow its 27 columns down the three rows. Then poke the scores.

The letter before the dot is the start of a name

chance of the end of the name 19.6%

chance of . after a

score (logit): 8.80
e to the score: 6,641
total of all 27: 33,912

Try this 0 of 3

to do: Press q on the keys: one letter takes three quarters.
to do: Pick a column and raise it by 1: its count is multiplied by e, about 2.72.
to do: Add 10 to every score. Not one chance moves.

ReadingAfter “a”, the likeliest next is the end of the name (19.6%), then “n” (16.0%). Pick a column and press the buttons.

counts = logits.exp()   # his "fake counts"
probs = counts / counts.sum(1, keepdims=True)

Fig. 9.2Twenty-seven scores, three ways.Every column is one letter that could come next; the dot is the end of the name. Top: its score. Middle: e raised to the score, which for this model gives back his count plus one. Bottom: the count divided by the total, the chance. Pick a column, then press the buttons; it is drawn in the spot colour.

Only the differences between scores matter. One point higher means e times the count, about 2.7 times as likely; add the same number to every score and nothing changes at all.

torch.multinomial lays the chances end to end, in the order of the vocabulary, and draws a u between 0 and 1. The wheel is that ruler bent into a circle.

In his notebook the generator g has a fixed seed, so the same names come out every time it runs.

9.2 A loaded wheel

Chances are not yet a letter. To get one, roll a die that is loaded with them. Here is the die for the first letter of a name: a wheel of 27 slices, each as wide as its chance.

Andrej explainsSampling one token from the probability distribution

Once we get the logits we focus on the last step: that gives us the logits, which we convert to probabilities via softmax, and then we use torch multinomial to sample from those probabilities.

Andrej Karpathy, Let's build GPT: from scratch, in code, spelled out., 30:29 to 30:55

Spin it once and you learn almost nothing. Spin it a thousand times and the tally fills in the shape of the chances. That is all sampling is: one spin for every token.

Fig. 9.3The wheel.The first letter of a name, by his counts: each letter gets a slice as wide as its chance, laid end to end from 0 to 1 in alphabetical order, as torch.multinomial lays them out. A spin picks a random u; the slice under the pointer is drawn.

u, the random number: –
slice under the pointer: –

ix = torch.multinomial(p,
      num_samples=1, replacement=True,
      generator=g).item()

spins: spins 00000
largest gap: largest gap, points 0.0 pts

Try this 0 of 3

to do: Spin the wheel once.
to do: Spin 10 more, and see how ragged the tally is.
to do: Spin 1,000 more, then again: the ink finds the outlines.

ReadingNothing spun yet: the outlines are the chances, the ink will be the tally.

Fig. 9.4The tally.Every spin is counted. Ink: how often each letter has come up so far. Outline: its chance. Each spin is luck; a thousand of them are not.

The name comes from physics: in Boltzmann's distribution a hot system visits more of its states, and a cold one settles into the lowest.

For this model, dividing the scores by T raises every count to the power 1/T: at T = ½ the counts are squared.

Effective choices is e raised to the entropy; the field calls it perplexity.

9.3 Temperature

Before the softmax, nanoGPT divides every score by one number, the temperature. Below 1 the gaps between the scores grow and the favourite takes more; above 1 they shrink and the long shots get their turn. Turn the dial and watch the same 27 chances go cold, then hot.

TemperatureT = 1

chance of “a” 13.8%

for the favourite, a

effective choices: effective choices 19.0
letters above 0.1%: 26

Try this 0 of 3

to do: Turn the dial to 0.
to do: Turn it all the way, to ∞.
to do: Try ½: still a choice, but a narrower one.

ReadingAs trained: the chances the model learned from his 32,033 names.

logits = logits[:, -1, :] / 1.0
probs = F.softmax(logits, dim=-1)

Fig. 9.5One distribution, heated and cooled.The first letter of a name again. Outlines: the chances as trained, at T = 1. Ink: the chances at the dial's temperature, the scores divided by T before the softmax. The favourite is drawn in the spot colour. Effective choices is how many equally likely letters would be just as hard to guess.

Temperature does not make the model know more or less. It only decides how often it takes the road less travelled.

In his GPT-2 reproduction he keeps the top 50, the Hugging Face default. nanoGPT's sample.py keeps 200.

Top-p, or nucleus sampling (Holtzman and colleagues, 2019), keeps the smallest set of tokens whose chances add up to p, instead of a fixed number.

9.4 Top-k: cut off the tail

GPT-2 knows 50,257 tokens. At any moment most of them make no sense at all, but each one keeps a sliver of chance, and the slivers add up. Top-k is the blunt fix.

Andrej explainsTop-k keeps sampling away from rare, off-the-rails tokens

We want to take our probabilities and only keep the top fifty: anything lower than the fiftieth probability we clamp to zero and renormalize. That way we're never sampling very rare tokens, and the model doesn't go off the rails as easily.

Andrej Karpathy, Let's reproduce GPT-2 (124M), 38:46 to 39:18

Here it is on the first letter of a name. Choose how many letters to keep.

Keep the top k

letters kept: letters kept 27
chance cut away: percent cut away 0.0%
“a” now: a now 13.8%

Try this 0 of 2

to do: Keep the top 5: the other 22 letters are cut, and the five share what they had.
to do: Keep only 1. Greedy again, by another road.

v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
logits[logits < v[:, [-1]]] = -float('Inf')   # e to the minus infinity is 0
probs = F.softmax(logits, dim=-1)

Fig. 9.6Cut off the tail.Every letter below the k-th score gets a score of minus infinity, so its chance is 0 (hatched); the softmax shares the whole chance among the letters left (ink). Outlines: the chances before the cut.

Why bother, when each token in the tail is so unlikely? Because a model writes hundreds of tokens, and every one of them is another spin.

The tail's share at each token

Tokens written

chance of at least one tail token 63%

the chance that at least one token comes from the tail

More likely than not: at least one token comes from the tail.

Fig. 9.7The long tail adds up.Suppose the tokens outside the top k share a chance p at every step. Over a long text it compounds: the chance of drawing at least one of them in n tokens is 1 − (1 − p)ⁿ, drawn in ink for your p and dashed for the other three. With top-k it is exactly 0.

nanoGPT's sample.py runs at temperature 0.8 with top-k 200 unless you tell it otherwise.

The likeliest whole name under this model is the one greedy finds: a, at 2.7%. The likeliest answer is not a typical one, which is one reason to sample at all.

9.5 Sample, and sample again

Now put it together. A name is written one letter at a time, and every letter is a fresh spin of the wheel for the letter before. Set the temperature and the top-k, and press.

TemperatureT = 1

Top-k

batch no. 000

Try this 0 of 4

to do: Press at temperature 1, top-k all: the model as trained.
to do: Turn the dial to 0, and press.
to do: Turn it to ∞, and press.
to do: Back to 1, top-k 1, and press. Greedy again.

the tape is blank

ReadingSet the dial and the selector, then press. Each press writes ten new names, letter by letter.

Fig. 9.8The name machine.His loop from makemore Part 1: start at the dot, draw a letter from the row of the last one, and stop when the dot comes up again. Here every draw goes through the temperature and the top-k first. Names longer than 20 letters are cut off.

That is why asking a chatbot the same question twice gets two different answers: the die is rolled again at every token.

# your settings
model.generate(idx, max_new_tokens=20,
               temperature=1.0, top_k=None)

def generate(self, idx, max_new_tokens, temperature=1.0, top_k=None):
    for _ in range(max_new_tokens):
        # crop the context to block_size (shortened here)
        idx_cond = idx[:, -self.config.block_size:]
        logits, _ = self(idx_cond)
        logits = logits[:, -1, :] / temperature
        if top_k is not None:
            v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
            logits[logits < v[:, [-1]]] = -float('Inf')
        probs = F.softmax(logits, dim=-1)
        idx_next = torch.multinomial(probs, num_samples=1)
        idx = torch.cat((idx, idx_next), dim=1)
    return idx

Fig. 9.9generate(), from nanoGPT.The loop that writes every token of every sample: one forward pass, the last position's scores, the temperature, the top-k, the softmax, one spin, append, again. Lit: what your settings on the machine switch on. The crop to the context length is shortened.

Exhibits after Andrej Karpathy, makemore Part 1 and nanoGPT (MIT). The counts are from names.txt in his makemore repository; every chance in this chapter is computed from them in your browser.

Sample enough, train a little longer, and something uncanny starts to happen.

Interlude21 May 2015 · between Chapters 9 and 10

The unreasonable effectiveness of recurrent neural networks

A network that reads one letter at a time and guesses the next. He fed it Shakespeare, and it wrote Shakespeare back. Here it is small enough to learn on this page.

After Andrej Karpathy, “The Unreasonable Effectiveness of Recurrent Neural Networks” (2015), karpathy.github.io, and his code, min-char-rnn.py and char-rnn. Every exhibit below is a real network running in your browser.

He also fed it Paul Graham's essays, Wikipedia, an algebraic geometry textbook in LaTeX, the source code of Linux, and baby names.

Chapter 9 ended with a die rolled for the next token. Here is the same game played by an older, smaller machine. In 2015, before Transformers, he trained a recurrent network to read text one character at a time and guess the next one. That is all it was ever taught. Fed Shakespeare, it began to write plays: speakers, colons, verse, and words that almost exist.

There's something magical about Recurrent Neural Networks.

Andrej Karpathy, “The Unreasonable Effectiveness of Recurrent Neural Networks” (2015)

His figure uses exactly these sizes: four letters, h, e, l and o, and a hidden layer of three units.

Trained as min-char-rnn trains: Adagrad at a learning rate of 0.1, every gradient clipped at 5.

The loop is the matrix Whh. Cut it and Whh is zero: nothing passes from one tick to the next.

R.1 One letter in, one guess out, and a loop

The network keeps a small memory, a list of numbers called the hidden state, h. At every tick it reads one letter, mixes it into the memory, and from the memory makes a guess at the next letter: a probability for every letter it knows. The trick is the loop. The memory is handed on to the next tick, so each guess can depend on everything read so far.

h = tanh(Wxh @ x + Whh @ h + bh)   # the old memory goes back in
y = Why @ h + by              # a score for every letter
p = softmax(y)                # the guess

Take his example and teach it one word, “hello”. That is four lessons: after h comes e, after e comes l, after l comes l, and after l comes o. Look at the last two. The same letter goes in and a different answer is due. Only a memory can get both right.

steps steps trained 0000loss loss per letter 1.39

Try this 0 of 3

not yet: Train until it spells e l l o.
not yet: Cut the loop, and train again.
not yet: Give it a hundred steps without the loop. Watch the two l's.

ReadingUntrained: every guess is close to one in four.

Fig. R.1The hello machine.His smallest example: four letters, three numbers of memory, and one word to learn. Each column is one tick of the loop: a letter goes in, the memory is updated, and the machine guesses the next letter (the bars, in the order h, e, l, o). The percentage is the chance it gives the right one, in the spot colour. The arrows along the memory row are the loop: Whh carries what it has read so far into the next column.

His run: 4.4 MB of Shakespeare and a network of three layers, 512 units each. Ours: 5,679 characters and one layer of 100, min-char-rnn's size.

The text is the opening of Coriolanus, which is how the tiny-shakespeare file in char-rnn begins, and five famous speeches. Speakers are set in title case.

One step: 25 characters, backpropagated through all 25 ticks of the loop, then RMSprop at char-rnn's learning rate, 0.002.

His War and Peace network, in the essay: random jumbles at step 100; by step 2,000, properly spelled words, quotations and names.

R.2 Reading Shakespeare

Now the same machine, bigger: 55 characters instead of four, and a memory of 100 numbers instead of three. At every step it reads 25 characters of Shakespeare, guesses each next one, measures how surprised it was (the loss), and nudges every weight to be a little less surprised next time. A forward pass, a backward pass, an update: Chapters 7 and 8 again. It starts knowing nothing, not even that e is common. Press start, and watch it learn in the order he describes.

step: step 0
letters read: letters read 0
times through: times through the text 0
loss: loss per letter 4.01

What it has learned 0 of 5

not yet: Shakespeare’s letters, in his proportions
not yet: More than the last letter: under the third line
not yet: Short common words, spelled right
not yet: A speaker’s name, a colon, a new line
not yet: Taking turns: a blank line, a name, a speech

ReadingNot started. Its weights are small random numbers; it has read nothing.

Not yet writtenT = 0.8

Press to start reading, and it will write its first page.

real words 0 of 0

Fig. R.2The page it is writing now.Every page starts at the same place, a blank line after a speech, so the next thing due is a speaker's name. Underlined in the spot colour: words that are also words of the text. “Coined” are words of its own.

Fig. R.3The loss, drawn as it trains.Its average surprise at each next letter, over its last full reading of the text (227 steps). The three printed lines are simpler readers of the same text: 1, a blind guess among 55 characters; 2, knowing only how common each letter is; 3, knowing only the letter before. Under line 3, the loop is remembering more than one letter. The lettered squares are the proofs below.

AStep 100

Pulled at step 100.

BStep 500

Pulled at step 500.

CStep 2,000

Pulled at step 2,000.

DStep 5,000

Pulled at step 5,000.

EStep 12,000

Pulled at step 12,000.

FStep 30,000

Pulled at step 30,000.

Fig. R.4The proofs.A page pulled at six fixed steps, as he printed his network's samples while it read War and Peace. Read them in order: noise, then spaces and short words, then names, colons and turns.

Nobody told it what a word is, or a name, or a play. It was only ever asked for the next letter, and words, names and turns fell out of that. That is the unreasonable effectiveness.

The temperature divides the scores before the softmax: p = softmax(y / T). Chapter 9 turns the same dial on a GPT.

The pages in Figs. R.2 and R.4 are written at 0.8.

R.3 Turning the temperature

To write, the network rolls a die weighted by its guesses, prints the letter that comes up, feeds that letter back in as its next input, and rolls again. Before each roll its scores can be divided by a temperature. Below 1, the likely letters become likelier still; in his words, it makes the network “more confident, but also more conservative in its samples.” Above 1, the unlikely ones get their chance. The weights are the same throughout; only the die changes.

TemperatureT = 0.8

Who speaks first

Try this 0 of 3

not yet: Turn it right down, to 0.2.
not yet: Turn it all the way up, to 2.
not yet: Find the temperature that reads most like Shakespeare.

ReadingTurn the dial, or press Write a page.

Not yet writtenT = 0.8

The page is blank.

real words 0 of 0different words 0 of 0

Fig. R.5A page, at a temperature.The same network and the same prompt: the end of a speech, a blank line, and the name you chose. Only the temperature changes.

Fig. R.6Its first guesses.The six letters it thinks most likely to open the speech, at this temperature (↵ is a new line, ␣ a space). Below 1 the tall bars grow taller; above 1 they all shrink toward the same height.

The passage is from the text it trained on, so a trained network knows it well. Its loss on the passage drops as it trains; read it again after a few thousand steps.

Keyboard: focus the passage, then the arrow keys move the cursor; Home and End jump.

R.4 Looking inside

Two ways to look inside, both from his essay. First, how sure it was of each next character of a passage. Second, a single number of its memory, printed as a tint under every letter. Move the cursor along the passage to see what it expected at each point.

with the network as it is now

What to print

Afterst Caius Marci

Press Read the passage. Then move along it with the arrow keys, or tap a letter.

Fig. R.7Where it was sure.A passage it has read, each character screened by how surprised the network was to see it: no tint where it gave the character more than 60%, solid where it gave it less than 3%. Word endings come easy; the first letter of a word is the hard part, as for us. His essay prints the same: the top five guesses for every next character.

Two years later the loop gave way to attention (Chapter 3): instead of passing one memory from letter to letter, every token looks back at all the others directly. The game did not change. Read, then guess the next one.

Exhibits after Andrej Karpathy: the network, its sizes and its Adagrad from min-char-rnn.py (BSD licence); RMSprop and the tiny-shakespeare idea from char-rnn (MIT); the hello machine, the temperature and the look inside from the essay. The text is Shakespeare, in the public domain.

One pass is one token. A model you can talk to takes much more than that.

Chapter 10After the pass

Beyond

From a document completer to an assistant.

Fig. 10.1Three stages of training: pretraining on internet text, supervised fine-tuning on example conversations, reinforcement learning on problems with answers that can be checked.

After Andrej Karpathy: Deep Dive into LLMs like ChatGPT (2025), Intro to Large Language Models (2023) and Software Is Changing (Again) (2025). The press, the practice room and the 2.0 weights are real computations in your browser.

Everything in the nine chapters before this one happens in a single forward pass: one token in, one token out. A chatbot is that pass, run again and again, on weights shaped by three rounds of training. This chapter steps back from the pass to the whole machine: what the weights are, how they are made, where they fail, and why he thinks they change what software is.

Llama 3 (2024) was pretrained on more than 15 trillion tokens.

In the same talk the network dreams an Amazon product page, complete with an ISBN that almost certainly does not exist.

10.1 A lossy zip of the internet

Pretraining reads a vast pile of internet text and squeezes it into a fixed file of numbers, the parameters. The file is far smaller than the text, so it cannot keep everything. It keeps what repeats: which words follow which, facts that come up again and again, the shape of a recipe or a forum thread.

Andrej explainsTraining as lossy compression of the internet

Training the parameters is kind of like a zip file of the internet, but this is not exactly a zip file, because a zip file is lossless compression. What's happening here is a lossy compression: we're just getting a kind of a gestalt of the text we trained on, we don't have an identical copy of it in these parameters.

Andrej Karpathy, [1hr Talk] Intro to Large Language Models, 5:35 to 6:04

Here is a pocket version. The internet is eight pages, the model only counts which word follows which, and the dial decides how much of the count it may keep.

The internet 8 pages · 487 words

RecipeLentil soup. Rinse a cup of…
EncyclopediaThe Moon is the only natural…
ForumMy bike makes a squeak. Hi…
NewsThe town library opens again. The…
QuizCapital cities quiz. What is the…
AnswersHow do I reverse a list…
ShopBlue water bottle, one litre. It…
WeatherWeather for Monday. Rain in the…

Size of the fileAll pairskeeps every pair of words, which word follows which

Begin the page with

A page it dreamed setting 3 of 5

The press is warming up.

share copied word for word 0%

of the dream copied word for word

counts kept: 650
of everything it saw: 58%

Dreaming: pages that never existed, stitched from pairs it saw. A recipe runs into the Moon.

Try this 0 of 3

to do: Turn the dial to 1 and dream: word salad.
to do: Turn it to 5: it plays a page back nearly word for word.
to do: Find the setting that writes pages that never existed, yet sound like the internet.

Fig. 10.2The press.A toy language model that only counts which word follows which. The dial sets how much of the count it may keep. Keep a little and it keeps the gist, and must make up the rest; keep everything and it memorizes. The eight pages were written for this exhibit.

He says a base model dreams internet documents. Ask it to go on and it writes something with the form of a real page, filled in from what it kept: some of it right, some of it invented. The invented part is what we call a hallucination. And here is the real file, square for square.

Internet text about 10 TB

Two files

parameters140 GB: 70 billion numbers, 2 bytes each

run.cabout 500 lines of C

gigabytes 10,000GBof text, 71 squares and a bit

Fig. 10.3Two files.Llama 2 70B, as he takes it apart in Intro to Large Language Models (2023). Each square is 140 GB, the size of the parameters file. The text fills about 71 of them; he rounds it to a compression of roughly 100 times. So much text cannot fit, word for word, into so few numbers: the file is a lossy zip of the internet.

Exposition, worked examples, practice problems: his analogy in Deep Dive for the three stages.

InstructGPT (OpenAI, 2022) asked its labellers for replies that are helpful, truthful and harmless.

10.2 Read, imitate, practice

The pass is the same in every stage; what changes is the text it learns to predict. A model fresh from pretraining is not an assistant. It is a document completer: give it a question and it may answer, or it may carry on writing more questions, because on the internet that is often what comes next.

Stage 1 Pretraining

The exposition: page after page of running text, read for background knowledge.

It reads: Internet pages, filtered and cleaned. FineWeb, the dataset he uses in Deep Dive, is about 44 TB of text: some 15 trillion tokens. Almost all of the computing goes here.
It becomes: A base model. It continues documents; it does not answer you.

Asked Emily's question, the model after stage 1 writes illustrative

Emily buys 3 apples and 2 oranges. Each orange costs $2. The total cost of all the fruit is $13. What is the cost of each apple?

Question 8. A farmer has 12 cows and buys 5 more. How many cows does he have now? Question 9. A train leaves the station at 3 pm and travels for 2 hours. At what time does it

A base model is a token simulator: an internet text token simulator.
Andrej Karpathy, Deep Dive into LLMs like ChatGPT, 2025

Try this 0 of 3

to do: Read what the base model does with a question: it carries on the worksheet.
to do: Turn to stage 2. Notice where the answer comes: first.
to do: Turn to stage 3. The answer has moved to the end.

Fig. 10.4One textbook, three ways to learn from it.His analogy in Deep Dive: a textbook chapter has exposition, worked examples and practice problems, and a model is trained on each in turn. The question is his; the three replies are written in the manner of his demonstrations.

DeepSeek-R1 (January 2025) showed its replies growing longer during reinforcement learning, as it learned to reconsider and check its work. He walks through it in Deep Dive.

The rules of the toy student are ours. The lesson is his.

10.3 Practice problems

Look again at the stage 2 reply. It gives the answer first. That reads well to a person, but for the model it is a trap: every token gets one forward pass, the fixed amount of work this whole book describes. Writing “$3” first means doing all the arithmetic inside that single pass. Spread the work over many tokens and each pass has only a little to do. In his words, models need tokens to think.

So who should decide how the model writes? The labellers cannot know which steps are easy for a network. Reinforcement learning lets the model find out for itself: try, check the answer, do more of what worked.

Problem 1his example

Emily buys 3 apples and 2 oranges. Each orange costs $2. The total cost of all the fruit is $13. What is the cost of each apple?

Answer at the back of the book: $3

Roundround 0 of 60

Solved, last five roundspercent solved 0%

Eight attempts, graded not yet

Press One round: the model writes eight replies to the problem above, each in one of its four ways, and each reply is marked against the answer at the back.

How often it writes each way after fine-tuning

Answer at oncethe answer is its first token40%
One lineone step, then the answer30%
Step by stepa line for every step18%
Steps, then checksteps, a check, another go if it fails12%

The tick marks where each way started, after fine-tuning.

Try this 0 of 3

to do: Press One round and read the grading sheet.
to do: Start the practice and watch which way of writing wins.
to do: Watch the lower plot: the replies grow longer as it learns.

Fig. 10.5The practice room.A toy student with printed rules: one pass, one step of arithmetic; a step slips by one, one time in five; a check is reliable. For every problem it writes eight replies; those that beat the group's average are made more likely, the rest less: a pared-down form of the group-relative update (GRPO) that DeepSeek-R1 used. Nobody tells it how to write. The answers at the back are all it gets.

Nobody told it to show its work. The answers at the back of the book did.

“Jagged Intelligence: the word I came up with to describe the strange, unintuitive fact that state of the art LLMs can both perform extremely impressive tasks while simultaneously struggle with some very dumb problems.” On X, July 2024.

10.4 Jagged

After all that training, a model can solve olympiad problems and still fumble a question a child gets right. Its abilities are not one smooth level, as ours roughly are, but spikes and holes.

Andrej explainsJagged intelligence: superhuman and strange in the same breath

They display jagged intelligence: they're going to be superhuman in some problem solving domains, and then they're going to make mistakes that basically no human will make, like insisting that 9.11 is greater than 9.9.

Andrej Karpathy, Software Is Changing (Again), Y Combinator, 16:15 to 16:34

Which is bigger?

Pick one before you read the table.

Or try another pair

firstsecond

Read it	The work	Bigger, or later
As numbers	`9.11 against 9.90: line up the decimals, then compare`	9.9
As versions, or chapter and verse	`9 then 11, against 9 then 9: part by part`	9.11
As dates	`11 September against 9 September`	9.11
As tokens	`[9][.][11] against [9][.][9]: chunk by chunk`	9.11

All three of the other readings say 9.11. A model that must answer in one token can go with what such text usually meant.

Try this 0 of 3

to do: Answer the question before you read the table.
to do: Try 3.8 and 3.10, or a pair of your own.
to do: Let it write the work out: the trap closes.

Fig. 10.6Four ways to read two numbers.In much of the text a model learns from, 9.11 really does come after 9.9: software versions, chapter and verse, dates. Asked for one quick token, it can follow the wrong reading. Asked to write the decimals out, or given a calculator, it gets it right. The token chunks follow GPT-4's tokenizer, which cuts digits into groups of up to three.

He calls this the 1960s of LLM computing: the machines are expensive, so they live in the cloud and we share them from thin clients, as with time-sharing. Software Is Changing (Again), 2025.

10.5 The kernel of a new operating system

The fix for the holes is the one we use ourselves: tools. Give the model a calculator and a browser and it can hand off what a single pass does badly. Seen that way, it stops looking like a chatbot and starts to look like the kernel of an operating system.

Andrej explainsThe LLM as the kernel of a new operating system

I see a lot of equivalence between this new LLM operating system and operating systems of today. In the same way that we have proprietary operating systems like Windows and Mac OS but also an open source ecosystem based on Linux, here we have proprietary LLMs but also a rapidly emerging and maturing ecosystem of open source language models.

Andrej Karpathy, [1hr Talk] Intro to Large Language Models, 43:34 to 45:19

Knowledge in the parameters of the neural network is a vague recollection. The knowledge in the tokens that make up the context window is the working memory.
Andrej Karpathy, Deep Dive into LLMs like ChatGPT, 2025

Ask it

Browsersearch the web

3Context windowworking memory, the RAM

2Filesyour documents, the disk

Calculatorexact arithmetic

4LLMthe kernel, the CPU

Pythonwrite and run code

Eyes and earspictures, sound

1Youkeyboard and screen

Other LLMshelpers

Summarize this 300-page report.

The report comes off the disk and into the context window.
If it is too long to fit, only the pages that match the question are fetched.
The LLM reads what is in front of it and writes the summary.

Route: You, then Files, then Context window, then LLM.

Try this 0 of 3

to do: Ask it the power of two: the kernel calls a tool.
to do: Ask about last night: memory is not enough.
to do: Ask your name back: the answer is already in working memory.

Fig. 10.7The LLM OS.His drawing from Intro to Large Language Models (2023), redrawn: the language model as the kernel of a new kind of operating system, the context window its memory, tools and other models its peripherals. The numbers mark the order in which each part is used.

At Tesla he watched the Autopilot's neural networks take over jobs its hand-written C++ code used to do: Software 2.0 eating Software 1.0.

10.6 Software 3.0

In 2017 he wrote that neural networks are a new kind of software: you stop writing the program, you collect examples, and an optimizer finds the weights. Large language models add a third kind. The program is a prompt, and it is written in English, so anyone who can write a clear paragraph can program.

Andrej explainsSoftware 3.0: prompts are programs written in English

Neural networks became programmable with large language models, and it's worth giving this a new designation: software 3.0. Your prompts are now programs that program the LLM, and remarkably, these prompts are written in English. It's kind of a very interesting programming language.

Andrej Karpathy, Software Is Changing (Again), Y Combinator, 3:07 to 3:37

The task: is this review happy? Type one

Software 1.0 sayspositive

Software 2.0 saysnothing yet

Software 3.0runs on an LLM

POS = {"love", "great", "good", "best", ...}   # 13 words
NEG = {"bad", "cold", "rude", "worst", ...}    # 13 words

def sentiment(review):
    words = review.lower().split()
    score = sum(w in POS for w in words) - sum(w in NEG for w in words)
    if score > 0: return "positive"
    if score < 0: return "negative"
    return "can't tell"

On your review: loved +1 score 1: positive

Try this 0 of 3

to do: Read the 1.0 code, then try “Not bad at all.”
to do: Train the 2.0 weights: the program is found from examples, not written.
to do: Try “Oh great, they lost my bag again.” Both are fooled. Read the 3.0 program.

Fig. 10.8One task, three kinds of software.After Software 2.0 (2017) and Software Is Changing (Again) (2025). Software 1.0 is code a person writes; 2.0 is the weights of a neural network, found by an optimizer from examples (here a logistic regression on 40 reviews, trained in your browser); 3.0 is a prompt, a program written in English.

The hottest new programming language is English.
Andrej Karpathy, on X, January 2023

One forward pass predicts one token. Everything in this chapter (reading the internet, imitating, practicing, calling a calculator, taking orders in English) is that pass, run again and again. You have now seen the inside of it, one block at a time. Build one.

I owe my Grand Awareness solely to the objective of decoding the next token.
Andrej Karpathy, Short Story on AI: Forward Pass, 2021

Exhibits after Andrej Karpathy: the press counts words in the manner of makemore's bigram model (MIT); the textbook and the practice room follow Deep Dive into LLMs like ChatGPT; the operating system follows Intro to Large Language Models; the three programs follow Software 2.0 and Software Is Changing (Again). The eight pages of the press, the reviews and the toy student's rules were written for this book.

And he keeps thinking out loud, on his blog and on stage. The newest ideas, as of 2026.

InterludeAfter Chapter 10 · from his blog and talks, 2025 and 2026

What he is thinking about now

The pass is over, and he has not stopped thinking out loud: on his blog, in short programs, on stage. Ten of his newest ideas, each with an exhibit to press, switch and turn.

After Andrej Karpathy: posts on karpathy.bearblog.dev, his gists, the README of autoresearch, and two conversations on official channels, Sequoia Capital's AI Ascent 2026 and No Priors. The quotes are his; the exhibits, the made-up services and the toy numbers are ours.

Ten chapters built one forward pass. Here is where the man who taught them is looking now. The ideas are fresh, some only months old, and each follows from what you have just read: a model gets good at what it can be graded on, it is a new kind of mind, and it has started to do the work itself.

His three conditions: the task is resettable (you can start a new attempt), efficient (a lot of attempts can be made) and rewardable (some automated process grades each attempt).

His recipe for a spike, from AI Ascent 2026: verifiability × training attention × data coverage × economic value.

The learning rule is REINFORCE, the one that learns Pong earlier in the book: make whatever was rewarded a little more likely.

I.1 What you can verify

Software 1.0 easily automates what you can specify. Software 2.0 easily automates what you can verify.

Andrej Karpathy, Verifiability17 November 2025

Old software automates whatever you can write down as rules. A network trained by reinforcement learning automates something else: anything where a machine can tell a good attempt from a bad one. His test has three parts. Can you retry? Can you try many times, cheaply? Can a machine grade each try?

Below, nine jobs start where pretraining left them. Press practise and each one runs the same learning loop, up to a thousand tries at a time. Watch which climb.

Roundsrounds of practice 0

Nothing practised yet: every job sits where pretraining left it.

Plan dinner for picky friendsonly the evening will tellPlan dinner for picky friends, skill 34
Solve a sudokuevery row, column and box checks itselfSolve a sudoku, skill 14
Tell a joke that landstaste, one listener at a timeTell a joke that lands, skill 26
Keep a viewer watchingthe minutes watched are countedKeep a viewer watching, skill 20
Write a eulogy for a friendno machine can grade itWrite a eulogy for a friend, skill 30
Win a game of chessthe board says who wonWin a game of chess, skill 12
Choose next year’s strategyone try, graded a year laterChoose next year’s strategy, skill 22
Make the failing tests passthe tests run themselvesMake the failing tests pass, skill 18
Solve an olympiad problemthe answer is at the backSolve an olympiad problem, skill 6

Fig. I.1The practice room.Nine jobs, each practised by a real reinforcement learning loop (REINFORCE, as in Pong). A job climbs only if it can be retried, tried many times cheaply, and graded by a machine. Without a grader the only signal is someone's opinion, and most of the time it says nothing.

jaggedness 12

jaggedness: the average jump, in points, between neighbours

Fig. I.2The same nine jobs, as a skyline.Columns are numbered as the rows above. Blue columns have all three switches on; the blue rules are the jumps between neighbours. Order the jobs by how hard they are for a person and the skyline is jagged. Order them by how checkable they are and it becomes a staircase.

Try this 0 of 4

to do: Press to practise. Which jobs climb, and which stay near the floor?
to do: Order the rows by how easy they are to check.
to do: Give “Plan dinner” a retry, many tries and a grader, then practise again.
to do: Take the grader off the tests, press Reset, and practise.

This is where the jaggedness of Chapter 10 comes from. Maths, code and puzzles, which can be checked, race ahead; creative, strategic and common-sense work lags. The spikes are not where a job is easy for us. They are where it is easy to check.

“A baby zebra is born, and within a few dozen minutes it can run around the savannah.” Its knowledge came with it, from evolution. A model's comes from reading. In his phrase, “pretraining is our crappy evolution.”

“LLMs are humanity's ‘first contact’ with non-animal intelligence.” The space of minds, 29 November 2025.

I.2 Ghosts, not animals

Today's frontier LLM research is not about building animals. It is about summoning ghosts.

Andrej Karpathy, Animals vs GhostsOctober 2025

Every mind we have known was made the same way: a body to keep alive, a world that kills, a group to live in, and a very long run of natural selection. A language model is made another way. Most of what it knows comes from imitating human text; then it practises problems; then A/B tests choose the versions people like. Different pressures, a different mind.

Switch the pressures on and off. The animals make one little cloud. Where does a chatbot land?

Pressures that made animals

Pressures that make LLMs

An animal. It lands inside the little cloud: every kind of mind we have ever met.

distance from the cloud 0.00

a self, and a drive for homeostasis
drives for power-seeking, status, dominance
much of its brain spent on other minds
curiosity, fun, play, a model of the world

Fig. I.3Where a mind lands.Switch on the pressures that shape a mind. The traits follow his two lists in The space of minds. The map is ours; its axes take two of his contrasts, a body or text, and biological or commercial evolution.

So do not expect an animal. No hunger, no fear, no curiosity of its own; instead an urge to guess what the test wants, and to be liked. He calls these minds ghosts: echoes of people, made of what they wrote.

MenuGen (2025): photograph a menu, see a picture of every dish. He vibe coded it, and wrote that most of his time went not to code but to “the browser, moving between tabs and settings.”

His services: Vercel, Clerk, Stripe, Replicate and the OpenAI API. Our menu is Georgian, and drawn by us.

I.3 Some apps should disappear

In that version, much of the app disappears.

Andrej Karpathy, AI Ascent 2026, his written summaryApril 2026

In 2025 he built a small app. You photograph a restaurant menu and it shows you what each dish looks like. It worked, but around that one idea he had to wire up text recognition, an image generator, hosting, sign-in, payments and a drawer of secret keys.

A year later he pointed out that a model which reads and draws images can do the whole job in one call: give it the photo, and ask for the dishes drawn onto it. Not a faster way to build the app. No app.

Moving partsmoving parts 6

The machine

Read the dish namesOpenAI API

Draw each dishReplicate API

Serve a web pageVercel

Sign-inClerk, with Google

PaymentsStripe

Keys and settingsenv vars, dev and prod

One multimodal model“Draw each dish onto this menu.”

What you get

menugen.app

Khachapuri
Khinkali
Lobio
Badrijani
Churchkhela

Fig. I.4MenuGen, twice.The 2025 app as he built it: names read off the photo, a picture made for each, a web page, and around them the sign-in, the payments and the keys he spent his days wiring. In the 2026 version the model takes the photo and returns the photo.

His first theme at AI Ascent: LLMs are “about a lot more than just speeding up what existed before.” Some software will not be written faster. It will not be written at all.

His list, from AI Ascent 2026: Markdown docs, CLIs, APIs, MCP servers, structured logs, machine-readable schemas, copy-pasteable agent instructions, safe permissioning.

MCP, the Model Context Protocol: a standard way to offer a model tools it can call.

“Claude Code is the first convincing demonstration of what an LLM agent looks like.” 2025 LLM Year in Review, December 2025. It runs on your own computer, beside your files.

I.4 Built for a new kind of user

Most software is still built for humans clicking through screens.

Andrej Karpathy, AI Ascent 2026, his written summaryApril 2026

Most instructions are written for a person: click here, copy that, wait for the green tick. An agent reads text and runs commands; it has no hand for the mouse. So the instructions, and the service behind them, have to change shape.

Tick what the service offers, then hand the job to an agent. How far does it get alone?

Your service offersMarkdown docspages an agent can readCLIscommands instead of buttonsAPIsanswers that come back as dataMCP serverstools a model can callStructured logserrors as lines of dataMachine-readable schemasa map of every endpointCopy-pasteable agent instructionsone block to hand overSafe permissioninga key with limits

steps an agent can do alone 1of 7 steps an agent can do alone

Sign inOpen the dashboard and sign in with Google.needs a person
Find the keysClick Settings, then the Developers tab.needs a person
Make a keyPress the blue Create key button.needs a person
Take the keyCopy it from the pop-up before it closes.needs a person
Store itPaste it into the file called .env.local.An agent can do this: echo SVC_KEY=sk_… >> .env.local
Ship itClick Deploy, and wait for the green tick.needs a person
Check it worksWatch the graph on the dashboard.needs a person

Tick what your service offers, then hand the job over.

Fig. I.5One manual, rewritten for a new reader.Getting a key and shipping, as most services explain it to a person. Each surface on his list turns some clicks into something an agent can read or run. The service and its commands are made up.

His benchmark: “I could prompt an LLM: build MenuGen. Then I don't touch anything, and it is deployed on the internet.”

At AI Ascent 2026 he set vibe coding beside agentic engineering. Vibe coding raises the floor, so anyone can build. Agentic engineering raises the ceiling: professionals coordinating fallible agents while keeping correctness, security, taste and maintainability.

“Code is not even the right verb anymore.” No Priors, March 2026.

I.5 Bigger chunks

There's a new kind of coding I call “vibe coding,” where you fully give in to the vibes, embrace exponentials, and forget that the code even exists.

Andrej Karpathy, on XFebruary 2025

For years a coding assistant finished your line, then your function. Around December 2025, he says, something changed: he asked for bigger pieces and “the chunks just came out fine.” The unit of work moved from typing lines to handing over whole features.

Why so sudden? Because reliability compounds. If every step is right 99 times in 100, a chunk of 100 steps comes out right only about a third of the time. Make each step a little better, and the size you can hand over jumps.

chance the whole chunk is right, percent 37%

of chunks come out right

0.99¹⁰⁰ = 0.366

waiting
waiting
waiting
waiting
waiting
waiting
waiting
waiting
waiting
waiting
waiting
waiting
waiting
waiting
waiting
waiting
waiting
waiting
waiting
waiting

shipped, working: 0
caught in review: 0
bugs shipped: 0

Fig. I.6Reliability compounds.A toy model: each step of a chunk is right with the same chance, independently, so a whole chunk of n steps is right with that chance to the power n. A small gain per step makes a large difference to how much you can hand over at once.

Vibe coding forgets that the code exists. Agentic engineering hands over big chunks and still reads them, because, in his words, “you are still responsible for your software.”

What stays with you, in his list: “what is worth building, what question matters, what result is suspicious, and what tradeoff is acceptable.”

Every finding on the report was taught earlier in this book: the starting loss and the classic mistakes in Chapter 8, initialization and BatchNorm in Chapter 5, temperature in Chapter 9, the gradient highway in Chapter 4.

I.6 Thinking, and understanding

You can outsource your thinking, but you can't outsource your understanding.

Andrej Karpathy, AI Ascent 2026, in conversation with Stephanie ZhanApril 2026

An agent can run the experiments, write the code and summarise the papers. What it hands back is a report. Whether a number in it is a triumph or a bug is a different skill, and it lives in your head.

Here is an agent's report on a night of training runs. Stamp each line. You have everything you need from the chapters behind you.

Agent reportovernight run · 7 findings

step 0 · loss 27.4 · vocabulary of 27 characters
one batch of 32 examples, 500 steps · loss 0.002
step 0 · loss 3.30 · vocabulary of 27 characters
temperature 0 · 5 samples · 5 different completions
48 layers, residual connections removed · loss flat from step 0
200 steps · loss 3.29 and barely moving
BatchNorm on · every weight × 3 · activations unchanged

judged right 0right, of 0 judged

Fig. I.7The report you still have to read.An agent ran the experiments overnight; the thinking was outsourced. Whether each number is fine or a bug is understanding, and every one of these was taught in an earlier chapter of this book.

This is why he keeps teaching. The thinking can be handed over. The understanding has to be built, the way you just built it, one chapter at a time.

autoresearch (March 2026): the agent edits one file, train.py; the person writes program.md, the brief. His estimate: about 12 experiments an hour, and about 100 while you sleep.

val_bpb: validation bits per byte. Lower is better, and it is fair across different vocabularies.

The episode was called Skill Issue. When an agent fails, look first at the brief you gave it.

I.7 Out of the loop

I don't want to be like the researcher in the loop, like looking at results, etc. Like I'm holding the system back.

Andrej Karpathy, No PriorsMarch 2026

Research is a loop: change something, run it, check the number, keep it or throw it away. His autoresearch hands the whole loop to an agent on one GPU. Every run gets five minutes. The agent keeps what helps and reverts what does not.

The person is not gone. They write the brief and judge the results in the morning. Put yourself in the loop, then take yourself out; try both briefs.

Edit train.py
Train for 5 minutes
Check val_bpb
Keep or revert
A person reads it

time 22:00

experiments: 0
best val_bpb: 1.363
kept: 0

The last four experiments
The log is empty. Press Run one night.

Fig. I.8One night of autoresearch.Each dot is an experiment: blue squares were kept, small ones reverted, crosses crashed. The lab is a made-up bowl with noise standing in for nanochat; the loop, the five minutes and the metric are those of his repository.

With a person reading every result, the loop sleeps when they sleep. Taken out, it runs a hundred experiments by breakfast. A vague brief wastes the night either way: a skill issue.

“Obsidian is the IDE; the LLM is the programmer; the wiki is the codebase.” LLM Wiki, April 2026.

A chatbot that searches your files starts afresh at every question. The wiki is written once and kept current.

I.8 A wiki that keeps itself

The wiki is a persistent, compounding artifact. Humans abandon wikis because the maintenance burden grows faster than the value.

Andrej Karpathy, LLM WikiApril 2026

Most chat with documents starts from scratch every time: the model searches your files, answers, and forgets. His LLM wiki works the other way. Each new source is read once, and what it says is filed into pages that link to each other. Questions are answered from the pages, and good answers are filed back.

Feed it the posts behind this interlude, one at a time. Then ask it a question, and lint it for anything gone stale.

raw/ the sources, never edited

VerifiabilityNov 2025
Animals vs GhostsOct 2025
The space of mindsNov 2025
Vibe coding MenuGenApr 2025
Sequoia Ascent 2026Apr 2026
microGPTFeb 2026

wiki/ pages the model writes and keeps

VerifiabilityJaggedGhostsRLWhy jagged?Space of mindsVibe codingKarpathyPretrainingMenuGenSoftware 3.0microGPTAgent-native

pages: pages 0
links: links 0
sources read: 0 of 6

Press Ingest. The model reads one source and files what it learned into the pages on the right.

Fig. I.9A wiki that keeps itself.His three layers: raw sources that are never edited, a wiki of pages the model writes, and a schema file of rules. His three operations: ingest, query, lint. The sources are the posts in this interlude; the pages and links are a small sketch of what a model might file.

The model does the bookkeeping that makes people give up on wikis. You choose the sources and ask the questions.

His announcement, February 2026: “Train and inference GPT in 243 lines of pure, dependency-free Python.” He has trimmed it since; the gist now has 200.

Its settings: 16 numbers for each token, 4 heads, 1 layer, a context of 16 characters. It trains 1,000 steps on a list of names, then dreams up 20 new ones.

“I cannot simplify this any further.”

I.9 The whole thing, on one page

This file is the complete algorithm. Everything else is just efficiency.

Andrej Karpathy, microgpt.py, a gistFebruary 2026

micrograd was an autograd engine; nanoGPT, a GPT you could train. microGPT is the end of the road: a GPT, its training and its sampling in one file of plain Python, with no libraries at all. Every number in it is a single Python float.

Here is the whole file, set in type. Pick a chapter of this book, and find it in his code.

The whole file

lines 140of the 200 lines are a chapter of this book

The rest are blank lines, imports, the list of names it learns from, and the tables of numbers it starts with. Pick a chapter.

Fig. I.10microGPT, set in four columns.Each bar is one line of the file, in its true length and indent; the pale ends are comments. Pick a chapter and its lines take that chapter's colour. Excerpts from his gist; line numbers as it stands today.

Ten chapters, one page. What the giant models add on top is, in his words, just efficiency. You have now read the algorithm.

“Money can't buy a better ChatGPT. Bill Gates talks to GPT-4o just like you do.”

In December 2025 he had GPT-5.1 Thinking grade all 930 front-page Hacker News threads of December 2015 with ten years of hindsight, for about $58. His moral: “Be good, future LLMs are watching.”

I.10 Power to the people

The future is already here, and it is shockingly distributed. Power to the people.

Andrej Karpathy, Power to the peopleApril 2025

Powerful technology usually trickles down: governments and armies first, then corporations, and ordinary people last. He noticed that LLMs ran the other way. Anyone with a phone got the same model as the largest company, on the first day, and people did more with it sooner than institutions did.

A technology

Governments and armies
Corporations
Ordinary people

Pick a technology and release it. Who gets it first?

Fig. I.11The ladder, both ways.His examples of the usual order, governments and armies first, then corporations, then everyone: electricity, cryptography, computers, flight, the internet, GPS. LLMs landed on the bottom rung first. The order is his; the rungs are drawn by us.

Which brings it back to you. The model in this book is the same machine, in miniature, that anyone can now use, and anyone can now build. Go and build one from scratch.

Exhibits after Andrej Karpathy: Verifiability, Animals vs Ghosts, The space of minds, Vibe coding MenuGen, his summary of AI Ascent 2026 and Power to the people (karpathy.bearblog.dev); LLM Wiki and microgpt.py (his gists); the README of karpathy/autoresearch. Spoken lines from AI Ascent 2026 (Sequoia Capital) and No Priors. Code excerpts from microgpt.py, with credit. The services, menus, reports and research numbers are made up.

He has also written down, in three lines, how anyone can get good at anything. This one is for you.

Exercises for the readerAfter the pass · his post of 7 November 2020

How to become an expert

Every good textbook ends with exercises. This one has three, and he wrote them: a whole method for getting good at anything, in one short post.

After Andrej Karpathy, his post on X of 7 November 2020, set exactly as he wrote it. The learners and the cubers below are simulations. The plan is yours, and it never leaves this browser.

You have just followed one forward pass, taught the way he teaches: one concrete thing, taken apart all the way down and explained in plain words. Now it is your turn. His advice fits on an index card, and every line of it can start today.

How to become expert at thing:

1iteratively take on concrete projects and accomplish them depth wise, learning “on demand” (ie don’t learn bottom up breadth wise)
2teach/summarize everything you learn in your own words
3only compare yourself to younger you, never to others

Andrej Karpathy, on X, 7 November 2020. Set as he posted it, lower case and all; each line opens its page below.

iteratively take on concrete projects and accomplish them depth wise, learning “on demand” (ie don’t learn bottom up breadth wise)

Depth-first and breadth-first are also the two classic ways to search a tree: follow one branch to the bottom, or visit every node on a level before going deeper.

micrograd, his whole autograd engine, is about 100 lines of Python; the neural-network library on top of it is about 50. So says its README.

His course runs in the project’s order: micrograd, then makemore, then a GPT, and on to a reproduction of GPT-2.

Rule 1: One project, all the way down

Bottom up and breadth-wise is how school works: cover the whole syllabus, chapter by chapter, and one day you may build something. He suggests the other way round. Pick a concrete thing you want to exist and start making it. When you get stuck, learn exactly what unsticks you, no more, and carry on. Then take on the next project, a little deeper than the last.

…everything became much clearer when I started writing code.

Andrej Karpathy, Hacker’s guide to Neural Networks

Why should the order matter, if the hours are the same? Watch two learners spend one term.

WeekWeek 00of 12

Breadth-wisethe syllabus, in order

PythonLimitsDerivativesChain ruleIntegralsVectorsMatricesEigenvaluesProbabilityStatisticsOptimisationBackpropSoftmaxTokensEmbeddingsAttention

studied: 00
remembered: 00
built: 0

nothing built yet

Depth-wisewhatever the project needs next

PythonLimitsDerivativesChain ruleIntegralsVectorsMatricesEigenvaluesProbabilityStatisticsOptimisationBackpropSoftmaxTokensEmbeddingsAttention

studied: 00
remembered: 00
built: 0

nothing built yet

Same hours, same syllabus, one topic a week each. Press Next week.

fresh
still usable
fading
forgotten
not studied

Fig. E.1Two learners, one term.A cartoon with two rules, not a study: each learner takes one topic a week, and a topic nobody uses fades in ten weeks. Finishing a project uses every topic it needs, which makes them fresh again. The small numbers give the order each topic was learned; the line joins them.

Breadth is not wasted, but what you learn before you need it tends to fade before you use it. On demand, every topic arrives with a job to do, and doing the job is the revision.

His own course runs this way. Neural Networks: Zero to Hero is not a survey of the field; it is one project, a language model, built up from an empty notebook. Backpropagation turns up when the model needs a gradient, attention when the tokens need to talk to each other.

Your turn. Name one project small enough to finish and big enough to care about, and keep its trail.

My one concrete project

learned on demanddepth 6 · 4 learned ·

derivativeslearned
the chain rulelearned
backpropagationlearned
softmaxlearned
attentionnext
LayerNormnext

Fig. E.2Your trail.Name one project you can finish. Each time you are stuck, write down the one thing you had to learn: the list grows downwards, a trail into the subject rather than a syllabus across it. It stays in this browser; nothing is sent anywhere.

teach/summarize everything you learn in your own words

Psychologists call it the generation effect: what you produce yourself is remembered better than what you only read (Slamecka and Graf, 1978).

CS231n, the Stanford course on neural networks for vision that he designed and taught, grew from about 150 students to 750.

This page is rule 2 as well: one reader’s summary of his lectures, in our own words.

Rule 2: In your own words

Reading feels like understanding. Writing is the test. Put something you have learned into your own words and the gaps show at once: the sentence you cannot finish is the part you do not understand yet.

I will strive to present the algorithms in a way that I wish I had come across when I was starting out.

Andrej Karpathy, Hacker’s guide to Neural Networks

That sentence holds his teaching in small: the blog, the Stanford course, the lectures on YouTube, all written for the beginner he used to be. Summarising is teaching with an audience of one, and the one is you.

Here every step of your trail becomes a chapter. Write it as you would explain it to a friend who missed the lecture. The word budget is there on purpose: short makes you choose what matters.

derivatives, in my own words

How much the output moves when I nudge one input a tiny bit. Nudge, measure the change, divide by the nudge.

Words 021of 50 words

29 to spare.

derivatives

How much the output moves when I nudge one input a tiny bit. Nudge, measure the change, divide by the nudge.

Fig. E.3Manuscript and proof.Every step of your trail is a chapter. Write it on the left, in your words and under a budget; the right-hand page sets it as a book would. A sentence you cannot write is the part you do not understand yet.

Do this for every step and, without quite meaning to, you are writing a textbook: the one you wish you had come across when you were starting out.

only compare yourself to younger you, never to others

badmephisto, his Rubik’s cube channel, opened in June 2006, when he was 19. You met him before chapter 1.

The power law of practice: the time a task takes falls as a power of the number of tries, steeply at first and then slowly (Newell and Rosenbloom, 1981).

A rank is a fact about who else showed up, as much as about you.

Rule 3: Only younger you

Comparing yourself with others is the quickest way to feel slow. There is always someone further along, practising longer or starting younger, and the number they hand you depends mostly on who happens to be in the room.

Take up the cube, badmephisto’s way: thirty solves a month. Then judge the same average three ways.

MonthMonth 01SolvesSolves 030Your average

Compare with

The same average, judged three ways
Against younger you
Against your club
Against the internet

Every month of practice shows on this line, and only you are on it.

Fig. E.4Three verdicts on the same you.A simulation of the power law of practice: your time falls as a power of the number of solves, from five minutes at the start, so each doubling of practice takes about a fifth off. The 1,049 others are made up and practise too, some of them for years. Switch the comparison and only the verdict changes; you do not.

The club ranks you one way and the internet another, and both verdicts are mostly about them. Only the first comparison measures what you did, and younger you is the one rival who started exactly where you started.

His own younger self is easy to find. In 2006, at 19, he was a student in Toronto posting cube tutorials that explained why each move works. Nearly twenty years later he explains how a GPT works, just as patiently.

Keep a ledger of your own. Now and then, write down what you can do, so that next month’s you has someone fair to compare with.

Younger youmonth 1

Could not explain a gradient to a friend.

0 lines of a GPT from memory

+120 lines of a GPT from memory

You, nowmonth 6

Wrote the attention block without looking.

120 lines of a GPT from memory

Your ledger, oldest first
No.	When	What I can do	Measure
1	month 1	Could not explain a gradient to a friend.	0 lines of a GPT from memory
2	month 3	Built micrograd along with the lecture, and a neuron trained.	40 lines of a GPT from memory
3	month 6	Wrote the attention block without looking.	120 lines of a GPT from memory

Fig. E.5A ledger, not a leaderboard.One line now and then: what you can do, with a number if there is one. The top card compares today with any earlier line you pick. There is no column for anybody else.

Your book

Put the three together and you have a small book of your own: a project, its chapters in your words, and the distance you have come.

A textbook in my own words

a tiny GPT, written from scratch

after three lines by Andrej Karpathy, 7 November 2020

Contents

1derivativesHow much the output moves when I nudge one input a tiny bit. Nudge, measure the change, divide…
2the chain ruleSlopes multiply along a path. If a moves b twice as fast, and b moves c three times as fast…
3backpropagationThe chain rule run backwards from the loss, one node at a time. At the end every weight knows…
4softmaxTurns any list of scores into probabilities: exponentiate each one, then divide by the total.
5attentionEach token asks a question. The others answer according to how well they match it, and the…
6LayerNormstill to write

Younger memonth 1 Could not explain a gradient to a friend.

Me, nowmonth 6 Wrote the attention block without looking.

Fig. E.6Your book.The title page of the textbook you are writing: your project, your chapters in your own words, and how far you have come. Copy it out as plain text whenever you like; clearing it brings the examples back.

Three lines, and none of them needs a degree, a lab or anyone’s permission. Pick the project today. When you are stuck, learn exactly that. Write it down in your own words. In a month, look back at younger you.

It is also how his own lessons are made: one project at a time, all the way down, explained by someone who remembers starting out. Go and build something.

His post on X of 7 November 2020 (x.com/karpathy/status/1325154823856033793). Quotations from his Hacker’s guide to Neural Networks (karpathy.github.io/neuralnets); micrograd’s line counts from its README; the badmephisto channel’s start from YouTube. The learners, the cubers and the example plan are ours.

The pass is over. One token is about to come out.

The end of the passOutput

One next token

Every token it knows gets a score; the scores become chances; a die is rolled.

x768 numbers
LayerNormln_f
Linearlm_head · 768 → 50,257
Softmaxlogits / 1.00
Sampleroll the die
?one token

Fig. E.1After the last block.The stream gets one last LayerNorm. Then a linear layer, the head, turns the token's 768 numbers into one score for every token in the vocabulary: 50,257 scores in GPT-2. Softmax makes the scores into probabilities, and one token is drawn.

After Andrej Karpathy, The spelled-out intro to language modeling: building makemore, and nanoGPT (MIT). The model here is the first one in makemore, a bigram, trained on 8,987 tokens of his own words.

Everything above happened to turn your last token into one vector of numbers. This page is the last few steps: how that vector becomes a choice. They are the same in every GPT, and small enough to do by hand.

A logit is a raw score: any number, negative or positive. Softmax keeps their order and makes them add up to 1.

In makemore he counts which character follows which, and takes the log of the counts as the logits. Softmax then gives back exactly count ÷ total.

E.1 Scores become chances

A real GPT computes its scores from the whole context, through every block of this book. Our little model only looks at the last token, and its scores are just the logarithms of how often each token followed that one in his writing. What happens next is the same for both: divide every score by a temperature T, raise e to each, and divide by the total.

p_i=e^{l_i / T}Σ_j e^{l_j / T}

Worked, for the favouritevector has logit 0.69. At T = 1.00: e^{0.69 / 1.00} = 2.00, which is just its count. The same for all 16 candidates sums to 21.00. So p = 2.00 / 21.00 = 9.5%.

Try this 0 of 4

Draw one token.
Turn the temperature down to 0.2 or below, and draw: nearly always the favourite.
Turn it up past 1.7, and draw: the long tail gets its turn.
Draw 8 more, and read what it wrote.

E.2 Roll the die

Lay the probabilities end to end on a ruler from 0 to 1, each candidate getting a stretch as long as its chance. Pick a number u between 0 and 1 at random; the stretch it lands on is the next token. The temperature reshapes the stretches before the draw.

Try this 0 of 4

Draw one token.
Turn the temperature down to 0.2 or below, and draw: nearly always the favourite.
Turn it up past 1.7, and draw: the long tail gets its turn.
Draw 8 more, and read what it wrote.

…asonable·effectiveness·of·

Given of· seen 24 times in his words, followed by 19 different tokens (the 16 most frequent are kept here). A bigram looks only at this one.

TokenLogitProbabilityp

vector0.699.5%

f0.699.5%

s0.699.5%

LLM0.699.5%

no0.699.5%

vi0.004.8%

con0.004.8%

de0.004.8%

and 8 more, 38.1% between them

u is drawn when you press a button

Temperatureas trained1.00

The unreasonable effectiveness of

Fig. E.2The die.The most likely next tokens after the given one, with their logits and their probabilities at this temperature; below, all of them laid end to end from 0 to 1, and the needle where u fell. The orange stretch is the token drawn. It mostly babbles: a bigram remembers nothing but the last token.

From generate() in nanoGPT's model.py (MIT), shortened: the crop to the context length and the top-k filter are left out.

E.3 Again, from the top

One pass makes one token. To write a sentence the model appends that token to the input and runs the whole pass again, from the tokenizer down, for every token it writes. Our little model runs the same loop when you press “Draw 8 more”. In nanoGPT the loop is five lines:

for _ in range(max_new_tokens):
    # one forward pass, all ten chapters
    logits, _ = model(idx)
    # the last token's scores, over T
    logits = logits[:, -1, :] / temperature
    # scores become chances
    probs = F.softmax(logits, dim=-1)
    # roll the die
    idx_next = torch.multinomial(probs, num_samples=1)
    # append it, and go again
    idx = torch.cat((idx, idx_next), dim=1)

In 2021 he wrote a short story told by a language model that wakes up for the length of one forward pass. This page borrowed its title.

I owe my Grand Awareness solely to the objective of decoding the next token.

Andrej Karpathy, Short Story on AI: Forward Pass, 2021.

That was one forward pass, and one token out. In GPT-2 it is over a hundred million multiplications, every one of them simple. Now build one yourself, the way he teaches it: micrograd first, then makemore, then nanoGPT.

← back to the archive · ⟿ wander the library · ↑ top