LLM_scratch #001: Large Language Models Explained Briefly — The Whole Picture in 8 Minutes
Large Language Models Explained Briefly — The Whole Picture in 8 Minutes
Highlights: A large language model is one big function. It reads a piece of text and returns a probability for every possible next word. Grant Sanderson (3Blue1Brown) packs the whole story into eight minutes. It covers next-word prediction, parameters, training by backpropagation, a compute bill of more than 100 million years, pretraining versus RLHF, and the Transformer’s attention and feedforward blocks. This is LLM_scratch #001, the first post of our new series that builds LLMs from scratch. It gives a map of what we will build later, piece by piece, in PyTorch.
- Next-word machine: a chatbot is a script (“User: … AI Assistant: ___”) that a model completes one word at a time.
- Probabilities, not answers: for “Paris is a city in ___” the model gives France 17%, and 15%, the 9%. Sampling less likely words is why the same prompt gives different answers.
- Hundreds of billions of dials: the “large” in LLM is the number of parameters. They start random and are tuned only by training.
- 100,000,000+ years: that is how long training the largest models would take at 1 billion additions and multiplications per second.
- Two training stages: pretraining on internet text, then reinforcement learning with human feedback (RLHF) to turn an autocompleter into an assistant.
- Attention + feedforward: words become vectors. Attention lets them exchange context (“river bank” vs “bank deposit”), and feedforward layers store learned patterns.
Tutorial Overview:
- A Script With a Missing Ending
- Probabilities, Not Answers
- Parameters: The Dials
- How One Training Step Works
- The Scale of Compute
- Pretraining, Then RLHF
- GPUs and the Transformer
- Inside the Transformer
- Emergent Behaviour
- Summary
1. A Script With a Missing Ending
In short
- A chatbot conversation is a movie script whose last line has been torn off.
- A “magic machine” predicts the next word; repeat it and the script finishes itself.
- This loop is exactly what happens every time you chat with an LLM.
1.1 The torn-off script
The video opens with a thought experiment. You find a script where a person asks an AI a question, but the AI’s answer is missing.
- The question: “Can you explain the history of transistors and how they’re relevant to computers?”
- The gap: the line “AI assistant:” is followed by nothing. The paper is torn.
- The tool: a machine that takes any text and gives a sensible guess for the next word.

Fig 1. A script with the AI’s answer torn off: the starting point of every chatbot reply.
1.2 One word at a time
The machine itself is simple to describe: text goes in, one word comes out.
- Classic example: “To be or not to ___” goes in, be comes out.
- The loop: append the predicted word to the script, feed the longer script back in, predict again.
- The result: “A transistor is a semiconductor device used to amplify or switch electronic signals…” is built word by word.

Fig 2. The model as a black box: text in, next word out.

Fig 3. Repeating the prediction with a growing script completes the AI’s answer.
2. Probabilities, Not Answers
In short
- An LLM does not output one word. It outputs a probability for every possible next word.
- A chatbot is a system prompt + the user’s message + repeated sampling.
- Sampling less likely words makes the text sound natural, and makes answers differ between runs.
2.1 A distribution over the vocabulary
Sanderson’s definition: a large language model is a sophisticated mathematical function that predicts what word comes next for any piece of text.
- Not certainty: the model assigns a probability to all possible next words.
- Example: for “Paris is a city in ___”, the top guesses are France 17%, and 15%, the 9%, Logan 7%.
- Long tail: Henry, northern, central, Paris, Texas… every word gets some probability.

Fig 4. The output is a probability distribution over next words, not a single word.
2.2 How a chatbot is built
A chatbot is not a different kind of model. It is the same next-word predictor, given a script to complete.

Fig 5. A chatbot = a system prompt, the user’s message, and an empty “AI Assistant:” line for the model to fill in.

Fig 6. The chatbot loop: predict, sample one word, append it, repeat.
2.3 Why the same prompt gives different answers
The model always picks from the distribution, but not always the top word. Text sounds more natural when less likely words are sometimes chosen at random.

Fig 7. Each new word is sampled from the distribution; the most likely word is not always the one chosen.

Fig 8. The function is fixed; the random sampling step is what makes each run different.
3. Parameters: The Dials
In short
- The model’s behaviour is fully set by many continuous numbers called parameters, or weights.
- Change the parameters and you change the next-word probabilities.
- Modern LLMs have hundreds of billions of them, and no human sets any of them by hand.
3.1 Tuning the dials on a big machine
Sanderson’s picture for training: a huge machine covered in dials.
- Parameters (weights): continuous values that fully determine how the model behaves.
- Effect: turning a dial changes the probabilities the model gives for the next word.
- “Large”: what puts the large in large language model is having hundreds of billions of these parameters.

Fig 9. Each dial is one parameter; together they decide the next-word probabilities.

Fig 10. Hundreds of billions of dials: this is what “large” means.
3.2 Starting from random
- No hand tuning: no human ever deliberately sets the parameters.
- Random start: at the beginning the parameters are random, so the model outputs gibberish.
- Refinement: the parameters are repeatedly adjusted using many example pieces of text.

Fig 11. Before training the parameters are random and the predictions are gibberish.
4. How One Training Step Works
In short
- Take a real text, hide the last word, ask the model to predict it.
- Backpropagation nudges every parameter so the true word becomes a bit more likely.
- Trillions of such steps make the model good on text it has never seen.
4.1 Predict the hidden last word
A training example can be a handful of words or thousands. The recipe is always the same four steps.

Fig 12. The four steps of one training example.

Fig 13. Training example: the model’s guess is compared with the true next word, “worst”.
4.2 Backpropagation
- Backpropagation: tweaks all parameters so the true word gets a bit more likely and every other word a bit less.
- In the animation: worst rises from 10% to 24% while age drops from 87% to 70%.
- Generalisation: after trillions of examples, predictions are also reasonable on text the model has never seen.

Fig 14. Backpropagation shifts probability toward the true word, one small step at a time.
5. The Scale of Compute
In short
- GPT-3’s training text would take a human over 2,600 years to read non-stop.
- Training the largest models takes more than 100 million years at 1 billion operations per second.
- Newer models train on much, much more text than GPT-3.
5.1 Three numbers

Fig 15. The scale of an LLM in three numbers.
5.2 Guess the training time
Sanderson asks the viewer to guess. At 1 billion additions and multiplications per second, is it a year? 10,000 years? The answer is well over 100 million years.

Fig 16. The thought experiment: one billion additions and multiplications per second.

Fig 17. Even at that speed, training the largest LLMs would take over 100 million years.
6. Pretraining, Then RLHF
In short
- Everything so far is pretraining: autocompleting random internet text.
- Autocompletion is not the same goal as being a good assistant.
- RLHF uses human feedback to change the parameters toward responses people prefer.
6.1 Step 1: Pretraining
Autocompleting a random passage of internet text is a very different goal from being a helpful assistant.

Fig 18. Pretraining alone produces a strong autocompleter, not yet a good assistant.
6.2 Step 2: RLHF
Reinforcement learning with human feedback, which Sanderson calls “just as important” as pretraining: workers flag unhelpful or problematic predictions, and their corrections change the parameters again.

Fig 19. Step 2 adds Reinforcement Learning with Human Feedback.

Fig 20. Many human reviewers flag bad outputs; their feedback tunes the same dials again.

Fig 21. Pretraining vs RLHF side by side.
7. GPUs and the Transformer
In short
- The compute is only possible on GPUs, chips built to run many operations in parallel.
- Before 2017 most language models read text one word at a time, which is hard to parallelise.
- Google’s Transformer (2017) reads the whole text at once, in parallel.
7.1 Parallel hardware
- GPUs: special chips optimised for running many, many operations in parallel.
- Catch: not every language model can use that parallelism easily.

Fig 22. GPUs run huge numbers of additions and multiplications in parallel.
7.2 Before 2017: one word at a time
- Sequential models: read “It was the best of times it was the worst of times” left to right.
- Bottleneck: each step waits for the previous one, so GPUs sit partly idle.

Fig 23. Before 2017, most language models processed text one word after another.
7.3 The Transformer
- Origin: a team of Google researchers, in the paper Attention Is All You Need (2017).
- Key change: Transformers do not read from start to finish. They soak in the whole text at once, in parallel.

Fig 24. The Transformer was introduced by Google researchers in 2017.

Fig 25. Previous models vs Transformers: sequential reading vs all words processed in parallel.
8. Inside the Transformer
In short
- Step 1: every word becomes a long list of numbers (a vector).
- Attention lets the vectors talk to each other and refine their meaning from context.
- Feedforward layers add capacity to store patterns; the two blocks repeat many times.
- The last vector is turned into a probability for every possible next word.
8.1 Words become vectors
The very first step, in a Transformer and in most other language models, is to turn each word into numbers.
Training only works with continuous values, so each word gets a vector that may somehow encode its meaning.

Fig 26. Each word is mapped to a long list of numbers: a vector.

Fig 27. Meaning as direction: similar meanings get similar vectors.
8.2 Attention: words talk to each other
What makes Transformers unique is a special operation called attention.
- Mechanism: all the vectors get a chance to talk to one another and refine their meaning from the surrounding context, all in parallel.
- “River bank”: the vector for bank can be changed by river to encode a river bank.
- “Deposit a check at the bank”: here deposit and check push bank toward the financial meaning.

Fig 28. Attention uses context words to decide which meaning of “bank” is meant.

Fig 29. After attention, the “bank” vector carries the more specific meaning “river bank”.
8.3 Feedforward: extra memory
The second operation is a feedforward neural network. It gives the model extra capacity to store patterns about language learned in training.

Fig 30. The feedforward network adds capacity to store language patterns.

Fig 31. Attention moves information between words; feedforward stores what was learned.
8.4 Many repetitions
The data flows through many rounds of attention → feedforward. Each round enriches every vector with whatever helps predict the next word.

Fig 32. Each layer refines the vector’s meaning with more context.

Fig 33. A Transformer stacks many attention + feedforward blocks.
8.5 The final prediction
By the end, the last vector has absorbed the whole context and everything learned in training. One final function turns it into a probability for every possible next word.

Fig 34. The last vector is turned into probabilities for the next word.

Fig 35. The full forward pass on one slide.
9. Emergent Behaviour
In short
- Researchers design the framework, but not the specific behaviour.
- Behaviour emerges from how hundreds of billions of parameters were tuned.
- That is why it is so hard to explain why a model makes a given prediction.

Fig 36. What researchers choose vs what training decides.
What we can see is the output: text that is uncannily fluent, fascinating, and even useful.

Fig 37. Back to the torn script: word-by-word prediction produces a fluent, useful answer.
9.1 What we will build from scratch
This post is a map. Each block in it becomes code in the next LLM_scratch posts.
| Concept in this video | What we will code |
|---|---|
| Words → vectors | Tokenizer (BPE) + embedding layer |
| Probabilities for every next word | Softmax output head, temperature and top-k/top-p sampling |
| Parameters + backpropagation | Cross-entropy loss and a PyTorch training loop |
| Attention | Self-attention, causal mask, multi-head attention |
| Feedforward | MLP block, residual connections, LayerNorm |
| Many repetitions | Stacked Transformer blocks: a GPT-style model |
| RLHF | Post-training: SFT, reward models, PPO/DPO/GRPO |
10. Summary

Fig 38. The whole video in one picture.
- Next-word function: an LLM maps any text to a probability for every possible next word.
- Chatbots: system prompt + user message + repeated sampling of the assistant’s next word.
- Parameters: hundreds of billions of numbers, random at first, tuned only by training.
- Training: hide the last word, predict it, backpropagate. Trillions of times.
- Compute: over 100 million years at a billion operations per second, possible only on parallel GPUs.
- Two stages: pretraining (autocomplete the internet) + RLHF (be a helpful assistant).
- Transformer: words → vectors → many rounds of attention + feedforward → next-word probabilities.
Source
Video: Large Language Models explained briefly by Grant Sanderson, 3Blue1Brown (2024). YouTube
Recommended follow-ups by the same author:
- Transformers, the tech behind LLMs · Deep Learning Chapter 5
- Attention in transformers, step-by-step · Deep Learning Chapter 6
- How might LLMs store facts · Deep Learning Chapter 7
- Vaswani et al. · Attention Is All You Need (2017)
dataHacker.rs — LLM_scratch #001: Large Language Models Explained Briefly — The Whole Picture in 8 Minutes. By Vladimir Matic.