LLM_log #003 Understanding Large Language Models – Transformers illustrative explanation

datahacker.rs Other 26.01.2026 | 0

🚀 Understanding Large Language Models: A Complete Visual Guide

🎯 What You’ll Learn

Large Language Models like GPT-4, Claude, and Llama have transformed how we interact with AI, but understanding what actually happens when you type “The cat sat on the” and the model predicts “mat” can feel like opening a black box.

In this comprehensive guide, we’ll demystify the entire process by following a single example through every step of a Transformer’s architecture. You’ll learn exactly how text becomes numbers, how those numbers flow through 12 sequential processing blocks, and how mathematical operations like attention and matrix multiplication work together to generate intelligent predictions.

We’ll use GPT-2 as our reference model (117 million parameters, 768 dimensions, 12 Transformer blocks) because it’s well-documented and represents the core architecture used in modern LLMs.

Tutorial Overview

The Three Main Components – Tokenizer, Transformer Stack, and Language Modeling Head
The Forward Pass – How text flows through the model to generate predictions
Positional Encodings – Teaching the model about word order
Inside Transformer Blocks – Self-attention and feedforward networks
The Feedforward Network – Where 48% of parameters live
Self-Attention in Action – How context influences predictions
Mathematical Deep Dive – Real numbers through attention
Multi-Head Attention – Why we split into 12 parallel heads

1. The Three Main Components

When you ask ChatGPT a question or have Claude write code for you, your text goes through three major processing stages. Understanding this high-level architecture is essential before we dive into the details.

Every Transformer LLM follows the same basic pipeline: Tokenizer → Transformer Stack → LM Head

Figure 3-4: The Three Main Components of a Transformer LLM

Input Text

“The cat sat on the”

▼

1
TOKENIZER

• Breaks text into tokens

• Converts tokens to token IDs

“The” → 464

” cat” → 3857

” sat” → 3332

” on” → 319

” the” → 262

▼

2
STACK OF TRANSFORMER BLOCKS

Transformer Block 1

↓ PROCESSES ↓

Transformer Block 2

⋮

Transformer Block N

(e.g., 32 blocks)

• Processes token representations

• Each block refines understanding

▼

3
LM HEAD

Language Modeling Head

• Converts final representation to probabilities

• Scores EVERY token in vocabulary

• Predicts most likely next token

50,000 probability scores →

One selected token

▼

Output: Next Token

“mat”

(Most probable next token)

Complete output:

“The cat sat on the mat“

💡 Key Insight

Tokenizer → Entry point (text to numbers)

Transformer Stack → The “brain” (heavy processing)

LM Head → Exit point (numbers to prediction)

💡 Why This Architecture?

The three-component design elegantly separates concerns. The tokenizer handles text-to-numbers conversion, the Transformer stack does the computational heavy lifting, and the LM Head converts refined representations back into probabilities over words.

🔴 Critical Point: Everything is Numbers

By the time text reaches the Transformer blocks, it’s already converted into dense vectors of 768 numbers each. The model never sees actual words after tokenization – only numerical representations.

2. The Forward Pass

Let’s trace exactly what happens to “The cat sat on the” as it flows through the model.

GPT-2 Specifications

Vocabulary: 50,257 tokens
Embedding dimension: 768
Number of blocks: 12
Context window: 1,024 tokens
Total parameters: ~117 million

Step 1: Tokenization

Figure 3-5: Tokenizer Vocabulary & Token Embeddings

Our Example

“The cat sat on the”

Token Vocabulary

Token ID	Token
0	!
1	“
…	…
262	the
319	on
464	The
3332	sat
3857	cat
…	…
50,000	Zyzzyva

Total: 50,000 tokens

Token Embeddings

Each token ID maps to a vector
(e.g., 3,072 dimensions)

ID 0:

[0.23, -0.45, …, 0.89]
ID 1:

[-0.11, 0.78, …, 0.23]
…
ID 262:

[0.12, 0.56, -0.89, …, 0.34]

 ← “the”
ID 319:

[0.67, -0.34, 0.12, …, -0.56]

 ← “on”
ID 464:

[0.45, 0.23, -0.67, …, 0.12]

 ← “The”
ID 3332:

[-0.89, 0.45, 0.23, …, 0.78]

 ← “sat”
ID 3857:

[-0.23, 0.89, 0.34, …, -0.45]

 ← “cat”
…
ID 50,000:

[0.34, 0.56, …, 0.67]

📝 Transformation Flow

Step 1: Text Input

“The cat sat on the”

▼

Step 2: Tokenization

“The”

” cat”

” sat”

” on”

” the”

▼

Step 3: Token IDs

[464, 3857, 3332, 319, 262]

▼

Step 4: Each ID → Embedding Vector

→ [0.45, 0.23, -0.67, …, 0.12] ← 3,072 numbers
→ [-0.23, 0.89, 0.34, …, -0.45]
→ [-0.89, 0.45, 0.23, …, 0.78]
…

💡 Key Points

• Vocabulary size: Typically 32K – 100K+ tokens
• Unique IDs: Each token has a numeric identifier
• Dense vectors: Each ID maps to 1000s of dimensions (e.g., 3072)
• Learned representations: Trained to capture semantic meaning

🔴 Embeddings Are NOT Universal

The embedding for “cat” in GPT-2 is completely different from “cat” in GPT-3 or Claude. Each model learns its own embeddings during training. You cannot transfer embeddings between models.

Step 2: Through Transformer Stack

Figure 3-6: Complete Forward Pass & Probability Distribution

Input Text

“The cat sat on the”

▼

TOKENIZER

The

cat

sat

on

the

↓ Token IDs: [464, 3857, 3332, 319, 262]

▼

STACK OF TRANSFORMER BLOCKS

Each token flows through its own processing stream:

464

↓

3857

↓

3332

↓

319

↓

262

↓

LAST

Block 1 → Block 2 → … → Block 32

✓ All 5 tokens processed in parallel
✓ Only LAST position (“the”) used for prediction

▼

LM HEAD

Outputs probability for EVERY token in vocabulary (e.g., 50,000 tokens)

Top Candidates:

Token	Probability	Visual
!	0.01%
“	0.02%
…	…
mat	45% 👑	HIGHEST
floor	18%
couch	12%
rug	8%
chair	5%
…	12%	(49,995 other tokens)

All probabilities sum to 100%

▼

Selected Output Token

“mat”

Complete: “The cat sat on the mat“

💡 Key Insights

1. Parallel Processing: All 5 tokens flow through simultaneously
2. Last Position Matters: Only output of “the” position used for prediction
3. Full Distribution: Model scores ALL 50,000 tokens
4. Highest Score Wins: “mat” (45%) is most likely next token

💡 Parallel Processing

All 5 tokens process in parallel through each block. They don’t go one-by-one – imagine 5 separate streams flowing simultaneously. Each stream can “look at” the others (that’s attention), but all move forward together.

Step 3: Decoding Strategies

Figure 3-7: Choosing a Token – Decoding Strategies

Prompt

“The cat sat on the”

▼

Model Output: Token Probabilities

Token	Probability	Visual
mat	45%	🎯 HIGHEST
floor	18%
couch	12%
rug	8%
chair	5%
…	12%	(49,995 other tokens)

▼

How do we pick the output token?

🎯 Greedy Decoding

How it works:

Always pick the token with the highest probability

Decision:

Pick “mat” (45%)

Characteristics:

✅ Deterministic (same every time)

✅ Fast and simple

❌ Can be repetitive

❌ Less creative

Temperature = 0

🎲 Sampling

How it works:

Pick randomly based on probabilities

Possible outcomes:

“mat” (45% chance)

“floor” (18% chance)

“couch” (12% chance)

etc…

Characteristics:

✅ More creative/diverse

✅ Different each time

⚠️ Can be unpredictable

⚠️ May pick unlikely tokens

Temperature > 0

▼

📝 Example Outputs

🎯 Greedy Decoding (Always the same):

“The cat sat on the mat“

🎲 Sampling (Different each time):

Run 1:

“… on the mat“

Run 2:

“… on the floor“

Run 3:

“… on the couch“

Run 4:

“… on the rug“

Run 5:

“… on the chair“

🚀 Advanced Sampling Strategies

Top-k Sampling

Only sample from the top k tokens (e.g., k=3)

Sample from: “mat”, “floor”, “couch”

Top-p (Nucleus) Sampling

Sample from tokens whose cumulative probability is p (e.g., p=0.8)

Sample from tokens until reaching 80% cumulative probability

💡 Key Takeaways

1. Probability ≠ Certainty: Model gives scores, decoding strategy picks token
2. Greedy = Predictable: Always picks highest probability (deterministic)
3. Sampling = Creative: Picks based on probabilities (varied output)
4. Temperature: Controls randomness (0 = greedy, higher = more random)
5. Real LLMs: Often use advanced strategies (top-k, top-p) for best results

3. Positional Encodings

There’s a critical problem: the model has no idea about word order. “The cat chased the dog” would look identical to “The dog chased the cat” without position information.

Figure 3-8: Positional Encodings – Teaching the Model About Order

⚠️ The Problem

Without positional information, these sentences would look identical to the model:• “The cat chased the dog” ≠ “The dog chased the cat”
• “You love me” ≠ “You love me?”

Word order matters in language! The model must know which token comes first, second, third, etc.

✅ The Solution: Positional Encodings

Add position information to each token’s embedding:
Final Embedding = Token Embedding + Position EncodingThis way, the model can distinguish between “cat at position 2” vs “cat at position 5”

How Positional Encodings Work

Example: “The cat sat on the”

Token

“The”

Position

Final Embedding

“The” at position 1

Token

“cat”

Position

Final Embedding

“cat” at position 2

Token

“sat”

Position

Final Embedding

“sat” at position 3

⋮

Two Types of Positional Encodings

1. Learned Positional Embeddings

Used in GPT-2:
• Separate embedding table for positions
• Position 0 → vector [0.23, -0.45, …]
• Position 1 → vector [0.12, 0.67, …]
• …
• Position 1023 → vector [-0.34, 0.89, …]

Pros: Simple, learned during training
Cons: Fixed maximum length (1024 for GPT-2)

Parameters: 1,024 × 768 = 786,432

2. Sinusoidal (RoPE)

Used in Modern Models:
• Mathematical function, not learned
• sin/cos waves of different frequencies
• Applied during attention computation
• Can handle any length!

Pros: No parameters, any length
Cons: More complex implementation

Parameters: 0 (computed)

How Vectors Are Combined (GPT-2 Example)

Token “cat” (ID 3857) embedding:
[0.12, -0.45, 0.67, …, 0.23]
(768 dimensions)
+
Position 2 encoding:
[0.08, 0.23, -0.12, …, 0.45]
(768 dimensions)
=
Final input to Transformer:
[0.20, -0.22, 0.55, …, 0.68]
(768 dimensions)

Element-wise addition: Each dimension adds: 0.12 + 0.08 = 0.20, -0.45 + 0.23 = -0.22, etc.

Where Positional Information Is Added

Input Text

“The cat sat on the”

▼

Tokenization

→ Token IDs

▼

Token Embeddings

Lookup in embedding table

▼

⭐ ADD POSITIONAL ENCODINGS ⭐

Token embedding + Position encoding

▼

Into Transformer Blocks

Ready for attention & feedforward

Important: Positional info is added ONCE at the start, before any Transformer blocks. It’s not added again inside the blocks.

💡 Key Insights

1. Essential for word order: Without positional info, “dog bites man” = “man bites dog”
2. Added at input: Token embedding + Position encoding = Final embedding
3. Two approaches: Learned (GPT-2) vs Sinusoidal/RoPE (modern models)
4. Context limit: GPT-2 can handle max 1,024 tokens (positions 0-1023)
5. Once and done: Position info flows through the network but isn’t re-added

💡 When Position Info Is Added

Positional information is added ONCE at the very beginning, right after token embedding lookup. It’s not re-added at each block. The position info flows through the network encoded in the vectors themselves.

4. Inside Transformer Blocks

Now let’s see what happens inside those 12 Transformer blocks.

The Stack

Figure 3-11: Inside the Transformer – Stack of Processing Blocks

Token Embeddings In

5 vectors of 768 dimensions each

(From: “The cat sat on the”)

▼

STACK OF TRANSFORMER BLOCKS

Transformer Block 1

Self-Attention + Feedforward

▼

Transformer Block 2

Self-Attention + Feedforward

▼

⋮

▼

Transformer Block 12

Self-Attention + Feedforward

• Each block refines token representations

• Output of one block = Input to next block

• Example: GPT-2 has 12 blocks

▼

Processed Representations Out

5 refined vectors of 768 dimensions each

→ Ready for LM Head to predict next token

💡 Key Insight

The bulk of LLM processing happens in this stack. Each Transformer block progressively refines the representation of each token, incorporating more context and understanding at each layer. Models range from 6 blocks (original Transformer) to 100+ blocks (large modern LLMs).

Real Model Examples:

• GPT-2: 12 blocks
• GPT-3: 96 blocks
• Llama 2: 32-80 blocks (depending on size)
• Original Transformer: 6 blocks

Inside One Block

Figure 3-12: Inside a Transformer Block

Input Vectors

Token representations from previous block

▼

TRANSFORMER BLOCK

1️⃣ SELF-ATTENTION LAYER

Purpose:

Incorporates contextual information from other tokens in the sequence

How it works:

• Determines which previous tokens are relevant
• Combines information from relevant positions
• Example: Understanding “it” in “The cat sat on the mat because it…”
• Uses Query, Key, Value matrices

▼

2️⃣ FEEDFORWARD NEURAL NETWORK

Purpose:

Houses the majority of the model’s memorization and processing capacity

What it stores:

• Learned patterns from training data
• Factual knowledge (e.g., “Paris” after “France”)
• Language patterns and relationships
• Enables both memorization & generalization

▼

Output Vectors

Refined token representations

→ Passed to next Transformer block

📊 Division of Labor

Self-Attention

Handles context and relationships between tokens

Feedforward

Stores knowledge and makes predictions

💡 Key Insight

These two components work together: Self-attention figures out what information is relevant, then feedforward processes that information to make predictions. This combination is repeated in every Transformer block, progressively refining the understanding of the input.

📝 Simple Example:

Input: “The Shawshank”Self-attention: Looks at how “The” and “Shawshank” relate
Feedforward: Recalls from training data that “Redemption” often follows

Result: High probability for “Redemption”

5. The Feedforward Network

Let’s zoom into the feedforward component – it contains 48% of the model’s parameters!

Figure 3-13: Feedforward Neural Network – Where Knowledge Lives

💡 The Knowledge Store

The feedforward network (FFN) contains 48% of GPT-2’s parameters (~56 million out of 117 million total). This is where the model stores most of its learned knowledge, patterns, and linguistic understanding.

Feedforward Network Architecture (GPT-2)

INPUT

768 dimensions

Output from attention layer

▼

LINEAR LAYER 1: EXPANSION ⬆️

768 → 3,072 dimensions

4× expansion! (768 × 4 = 3,072)

Parameters: 768 × 3,072 = 2,359,296

▼

ACTIVATION FUNCTION (GeLU)

Non-linear transformation
Allows network to learn complex patterns
GeLU(x) = x · Φ(x)

▼

LINEAR LAYER 2: COMPRESSION ⬇️

3,072 → 768 dimensions

Compress back to original size

Parameters: 3,072 × 768 = 2,359,296

▼

OUTPUT

768 dimensions

Ready for next layer

Total FFN Parameters (per block):

2,359,296 + 2,359,296 = 4,718,592

(~4.7 million parameters per Transformer block)

Comparison: Feedforward vs Self-Attention

🟡 FEEDFORWARD

Parameters: ~4.7M

Purpose: Store knowledge

Operation: Transform each token independently

Expansion: 768 → 3,072 → 768

Share: 66% of block params

Largest component!

🔵 SELF-ATTENTION

Parameters: ~2.4M

Purpose: Understand context

Operation: Mix information across tokens

Matrices: Q, K, V, Output (4×)

Share: 34% of block params

Lighter weight

Parameter Distribution (per block)

Feedforward
66% (~4.7M)

Attention
34% (~2.4M)

Why Is The Feedforward Network So Large?

1. Knowledge Storage Capacity

The 3,072 hidden dimensions provide enormous space to encode patterns, facts, and linguistic structures learned from billions of training examples. More parameters = more knowledge storage.

2. The 4× Expansion Rule

Most Transformers expand by 4× (768 → 3,072). This empirically works well – provides enough capacity without being wasteful. GPT-3 uses 12,288 → 49,152 (also 4×).

3. Non-Linear Transformations

The expansion through activation (GeLU) allows the network to learn complex, non-linear patterns. Linear transformations alone can’t capture the complexity of language.

4. Per-Token Processing

Unlike attention (which mixes tokens), FFN processes each token independently. This allows specialized transformations for each position, requiring more parameters.

📊 GPT-2 Full Model Breakdown

Component	Parameters	Percentage
Token Embeddings	38.6M	33%
12 × Feedforward Layers	56.6M	48% 🏆
12 × Attention Layers	28.3M	24%
Other (LayerNorm, etc.)	~1M	~1%
TOTAL	~117M	100%

💡 Key Takeaway

The feedforward network is the largest component of each Transformer block. Its massive 4× expansion (768 → 3,072 → 768) provides the capacity needed to store knowledge, learn patterns, and make predictions. This is where most of the model’s “intelligence” is encoded through its 56.6 million learned parameters.

💡 Where Knowledge Lives

The feedforward network is the model’s primary knowledge store. That 4× expansion (768 → 3,072 → 768) provides massive capacity to encode patterns learned from billions of training tokens.

6. Self-Attention in Action

Now let’s see attention working with our example sentence.

Figure 3-14: Self-Attention Incorporates Relevant Context

🤔 Understanding Context

Input: “The cat sat on the”Question: When predicting the next word after “the”, which previous tokens are most relevant?
• Should we focus on “cat”? 🐱
• Or on “sat”? 💺
• Or on “on”? 📍

How Self-Attention Solves This

Step 1: Input Tokens

The

cat

sat

on

the

▼

Step 2: Self-Attention Scoring

Processing token: “the” (position 5)

The model calculates relevance scores for each previous token:

The

cat

25%

sat

HIGHEST

58% 🎯

on

12%

▼

Step 3: Enriched Representation

Result:

The representation of “the” now incorporates strong information from “sat” (58% attention weight)The model understands: We need a noun that something sits ON
→ Predicts: “mat” (common object to sit on)

Visual Flow

🐱

cat

25% attention

→

📍

the

current token

←

💺

sat

58% attention ★

💡 Key Insight

Self-attention is how the model understands context. When processing “the” (position 5), the model looks back at previous tokens and determines that “sat” is most relevant (58%). This tells the model we need a noun that something sits ON – leading to predictions like “mat”, “floor”, “couch”, etc.

🎓 How the Model Learned This:

During training on billions of sentences, the model saw many examples like:• “The cat sat on the mat”
• “The dog sat on the floor”
• “She sat on the chair”

The model learned that after “sat on the“, common completions are furniture/surfaces. The attention mechanism learned to focus on the verb “sat” to understand what type of word should come next.

🔴 Attention is Learned, Not Designed

The model doesn’t have hardcoded rules saying “attend to ‘sat’ when predicting furniture.” These patterns emerge through training on billions of tokens. The attention weights are learned parameters that get adjusted to minimize prediction error.

7. Mathematical Deep Dive

Let’s follow real numbers through the attention mechanism using a simplified 4D example.

Part 3: Attention Mechanism – Mathematical Deep Dive

Following the numbers through every step

📐 Our Simplified Example

To make the math clear, we’ll use:
• Sentence: “The cat sat on the”
• Vector dimension: 4 (instead of GPT-2’s 768)
• Focus: Processing the last token “the” to predict “mat”
• Real numbers: Actual matrix multiplications you can verify!

Step 0: Input Embeddings (Start Here)

Each token has a 4-dimensional embedding vector:

Token 1: “The”
x₁ = [1.0, 0.5, 0.2, 0.8]
Token 2: “cat”
x₂ = [0.6, 0.9, 0.3, 0.4]
Token 3: “sat”
x₃ = [0.8, 0.3, 0.7, 0.5]
Token 4: “on”
x₄ = [0.4, 0.6, 0.5, 0.9]
Token 5: “the” ⬅ Currently processing
x₅ = [0.7, 0.4, 0.6, 0.3]

Matrix form: All input vectors stacked as 5×4 matrix (5 tokens, 4 dimensions each)

Step 1: Projection Matrices (Learned Parameters)

The model has learned three 4×4 projection matrices during training:

W_Q (Query)

[0.5 0.2 0.3 0.1]

[0.1 0.4 0.2 0.5]

[0.3 0.1 0.5 0.2]

[0.2 0.3 0.1 0.4]

W_K (Key)

[0.4 0.3 0.2 0.3]

[0.2 0.5 0.1 0.2]

[0.3 0.1 0.4 0.5]

[0.1 0.2 0.3 0.4]

W_V (Value)

[0.6 0.1 0.2 0.4]

[0.3 0.5 0.3 0.1]

[0.2 0.3 0.4 0.5]

[0.4 0.2 0.1 0.3]

Important: These matrices are learned during training, not designed by hand. The model adjusts them to capture meaningful patterns in language.

Step 2: Compute Q, K, V Matrices (Matrix Multiplication)

Formula: Q = X · W_Q, K = X · W_K, V = X · W_V

Computing Query for token 5 (“the”):

q₅ = x₅ · WQ
q₅ = [0.7, 0.4, 0.6, 0.3] ·
[0.5 0.2 0.3 0.1]

[0.1 0.4 0.2 0.5]

[0.3 0.1 0.5 0.2]

[0.2 0.3 0.1 0.4]
q₅ = [0.59, 0.41, 0.56, 0.43]

Calculation: (0.7×0.5 + 0.4×0.1 + 0.6×0.3 + 0.3×0.2) = 0.59, …

Computing Keys for all previous tokens:

k₁ = x₁ · WK = [0.58, 0.55, 0.47, 0.61]
k₂ = x₂ · WK = [0.52, 0.63, 0.40, 0.57]
k₃ = x₃ · WK = [0.66, 0.56, 0.55, 0.68]
k₄ = x₄ · WK = [0.53, 0.58, 0.52, 0.65]

Computing Values for all tokens:

v₁ = x₁ · WV = [0.84, 0.45, 0.41, 0.73]
v₂ = x₂ · WV = [0.67, 0.58, 0.47, 0.54]
v₃ = x₃ · WV = [0.79, 0.48, 0.56, 0.75]
v₄ = x₄ · WV = [0.68, 0.55, 0.44, 0.61]

Step 3: Compute Attention Scores (Dot Products)

Formula: score_i = q₅ · k_i^T / √d_k (where d_k = 4)

Computing dot products:

score₁ = q₅ · k₁ =

[0.59,0.41,0.56,0.43] ·

[0.58,0.55,0.47,0.61]
= 0.59×0.58 + 0.41×0.55 + 0.56×0.47 + 0.43×0.61 = 1.13
score₂ = q₅ · k₂ = 1.09
score₃ = q₅ · k₃ = 1.28 ⬅ HIGHEST
score₄ = q₅ · k₄ = 1.15

Scaling: Divide by √4 = 2 to normalize:
[1.13/2, 1.09/2, 1.28/2, 1.15/2] = [0.565, 0.545, 0.640, 0.575]

Step 4: Apply Softmax (Normalize to Probabilities)

Formula: attention_i = exp(score_i) / Σ exp(score_j)

Scaled scores: [0.565, 0.545, 0.640, 0.575]
Apply exp(): [1.76, 1.72, 1.90, 1.78]
Sum = 1.76 + 1.72 + 1.90 + 1.78 = 7.16
Attention weights (softmax output):
α₁ = 1.76/7.16 = 0.246 (24.6%)
α₂ = 1.72/7.16 = 0.240 (24.0%)
α₃ = 1.90/7.16 = 0.265 (26.5%) ★ HIGHEST
α₄ = 1.78/7.16 = 0.249 (24.9%)

Check: 0.246 + 0.240 + 0.265 + 0.249 = 1.000 ✓ (All probabilities sum to 1)

Step 5: Weighted Sum of Values (Final Output)

Formula: output = Σ (α_i × v_i)

Recall our values:
v₁ = [0.84, 0.45, 0.41, 0.73]

v₂ = [0.67, 0.58, 0.47, 0.54]

v₃ = [0.79, 0.48, 0.56, 0.75]

v₄ = [0.68, 0.55, 0.44, 0.61]
And attention weights:
α₁ = 0.246, α₂ = 0.240, α₃ = 0.265, α₄ = 0.249
Computing weighted sum:
output = 0.246×[0.84, 0.45, 0.41, 0.73] +

0.240×[0.67, 0.58, 0.47, 0.54] +

0.265×[0.79, 0.48, 0.56, 0.75] +

0.249×[0.68, 0.55, 0.44, 0.61]
output = [0.745, 0.515, 0.470, 0.656]

Interpretation: This output vector is the new representation of token “the” after incorporating context from previous tokens, with emphasis on “sat” (26.5% weight).

🎨 Understanding “Coloring” – The Key Insight

What Just Happened?

We just “colored” or “tinted” the embedding of token 5 (“the”) by mixing in information from previous tokens!

BEFORE Attention

[0.7, 0.4, 0.6, 0.3]

Generic “the”
NO context

→

AFTER Attention

[0.745, 0.515, 0.470, 0.656]

Contextualized “the”
COLORED by “sat” (26.5%)

The “Mixing Recipe”:

output₅ = 24.6% of v₁ (“The”) +

0% of v₂ (“cat”) +

5% of v₃ (“sat”) ← MOST

9% of v₄ (“on”)

Result: Token 5 now “knows” it follows “sat on” and needs a noun that something sits ON → predicts: mat, floor, couch, chair, etc.

⚡ Why Parallel Processing is Essential

Notice the weighted sum requires v₁, v₂, v₃, v₄ from all previous tokens!To compute output₅, we needed:
• Keys (k₁, k₂, k₃, k₄) → to compute attention scores
• Values (v₁, v₂, v₃, v₄) → to combine in weighted sum
• Query (q₅) → to determine what to attend to

All tokens must be processed simultaneously to generate their K and V vectors. This is why Transformers process tokens in parallel – not for speed, but because the math requires it!

🎯 Complete Attention Flow Summary

Input: x₅ = [0.7, 0.4, 0.6, 0.3]

↓ (multiply by W_Q, W_K, W_V)

Projections: q₅, k₁…k₄, v₁…v₄

↓ (dot products & softmax)

Attention: [0.246, 0.240, 0.265, 0.249]

↓ (weighted sum of values)

Output: [0.745, 0.515, 0.470, 0.656]

💡 Key Insights

1. Self-attention is matrix multiplication: No magic, just learned weights!
2. Token “sat” got highest attention (26.5%): This is why the model predicts furniture/surfaces
3. All operations are differentiable: The model can learn optimal W_Q, W_K, W_V through backpropagation
4. In real GPT-2: Vectors are 768-dimensional, not 4, but the math is identical!
5. This happens in parallel: All 5 tokens are processed simultaneously

After self-attention, this output vector [0.745, 0.515, 0.470, 0.656] goes through:
→ Feedforward Neural Network
→ More Transformer blocks (repeat 11 more times in GPT-2)
→ LM Head to predict “mat”

💡 Why Parallel Processing is Required

The weighted sum formula needs value vectors from ALL previous tokens simultaneously. To compute output for position 5, we need v₁, v₂, v₃, v₄ all at once. This is why Transformers are inherently parallel – the math requires it!

8. Multi-Head Attention

GPT-2 actually runs attention 12 times in parallel using different learned matrices.

Figure 3-17: Complete Flow Through One Transformer Block

INPUT

768 dimensions

↓

SPLIT INTO 12 HEADS

768 → 12 × 64 dimensions

↓

EACH HEAD: Q, K, V MATRICES

Q = X·W_Q K = X·W_K V = X·W_V

↓

ATTENTION FORMULA

Attention(Q,K,V) = softmax(QK^T / √d_k) V

d_k=64, √d_k=8

↓

CONCATENATE 12 HEADS

12 × 64 → 768 dimensions

↓

W_O (OUTPUT MATRIX)

768 × 768 projection

↓

ADD & NORM

LayerNorm(input + output)

↓

FEEDFORWARD NETWORK

768 → 3,072 → 768

↓

ADD & NORM

LayerNorm(input + FFN)

↓

OUTPUT

768 dimensions

→ Next block or LM Head

🔗 Connection to LM Head

After all 12 Transformer blocks:Block 12 output (768 dims)
↓
LM Head matrix (768 × 50,257)
↓
50,257 logits
↓
Softmax → probabilities
↓
Select token (e.g., “mat”)

📐 The Complete Formula

Multi-Head Attention:

MultiHead(Q,K,V) = Concat(head₁,…,head₁₂)W_O
Where each head:

head_i = Attention(QW_Qⁱ, KW_Kⁱ, VW_Vⁱ)

💡 Information is Distributed

Each head sees dimensions 0-63, 64-127, etc. But information isn’t neatly separated – it’s distributed across all dimensions. Each head sees a partial, blurred view of everything. Through learned Q/K/V projections, different heads learn to emphasize different patterns.

🎓 Final Synthesis

The Complete Flow

Text → Tokenizer → Token IDs
Token IDs → Embedding Lookup → 768-dim vectors
Add positional encodings (once, at the start)
Flow through 12 Transformer blocks sequentially
Each block: Self-attention + Feedforward + Residual connections
Final output → LM Head → 50,257 logits
Softmax → Probabilities → Pick next token

Key Takeaways

💡 Essential Concepts

Everything is learned: Token embeddings, attention weights, feedforward parameters – all learned from data, not hand-designed.

Parallel processing: All tokens flow simultaneously. This isn’t just for speed – the math requires it.

Context is key: Same word has different representations depending on context, thanks to attention.

Scale matters: GPT-2’s 117M parameters enable good performance, but GPT-3 (175B) and GPT-4 (rumored 1.7T) achieve dramatically better results.

It’s predictive: Despite seeming intelligent, LLMs do next-token prediction based on learned patterns.

🎉 You’ve Completed the Guide!

You now understand how Large Language Models work from input to output. You’ve seen the architecture, followed numbers through calculations, and learned why design choices were made.

This knowledge applies to GPT-4, Claude, Llama, and virtually all modern LLMs.

Reference Model: GPT-2 (50,257 tokens • 768 dimensions • 12 blocks • 117M parameters)
Consistent Example: “The cat sat on the” → “mat”
Colors: Google Material Design Palette
All figures: Self-contained with inline CSS

📐 Appendix: Complete Mathematical Reference

This appendix provides all key mathematical formulas used in Transformer language models, with proper LaTeX notation.

1. Attention Mechanism

Single-Head Attention

Attention(Q, K, V) = softmax(QK^T / √d_k) V

Where:

Q = Query matrix (what we’re looking for)
K = Key matrix (what each token offers)
V = Value matrix (actual information to combine)
d_k = dimension of keys (64 per head in GPT-2)
√d_k = scaling factor to prevent large dot products

💡 GPT-2 Example

For each attention head: d_k = 64, so √d_k = √64 = 8

Multi-Head Attention

MultiHead(Q, K, V) = Concat(head₁, head₂, …, head_h) W^O

Where each head is computed as:

head_i = Attention(QW^Q_i, KW^K_i, VW^V_i)

Parameters:

W^Q_i ∈ ℝ^{d_model × d_k} = Query projection for head i
W^K_i ∈ ℝ^{d_model × d_k} = Key projection for head i
W^V_i ∈ ℝ^{d_model × d_v} = Value projection for head i
W^O ∈ ℝ^{hd_v × d_model} = Output projection matrix
h = number of heads (12 in GPT-2)
d_model = 768 (GPT-2)
d_k = d_v = d_model/h = 64 (GPT-2)

2. Softmax Function

softmax(x_i) = exp(x_i) / Σ_j=1ⁿ exp(x_j)

Properties:

Output range: (0, 1) for each element
Sum constraint: Σ_i softmax(x_i) = 1
Monotonic: preserves ordering of inputs
Differentiable: enables gradient-based learning

In attention context:

attention_weight_i = exp(score_i) / Σ_j exp(score_j)

Where score_i = (q · k_i) / √d_k

3. Layer Normalization

LayerNorm(x) = γ ⊙ x – μ / √(σ² + ε) + β

Where:

μ = (1/d) Σ_i=1^d x_i (mean)
σ² = (1/d) Σ_i=1^d (x_i – μ)² (variance)
γ, β = learnable parameters (scale and shift)
ε = 10^-5 (small constant for numerical stability)
⊙ = element-wise multiplication
d = feature dimension (768 for GPT-2)

4. Residual Connections

Post-normalization (original Transformer):

output = LayerNorm(x + Sublayer(x))

Pre-normalization (modern Transformers, including GPT-2):

x₁ = x + MultiHeadAttention(LayerNorm(x))

output = x₁ + FeedForward(LayerNorm(x₁))

5. Feedforward Neural Network

FFN(x) = GeLU(xW₁ + b₁)W₂ + b₂

Parameters:

W₁ ∈ ℝ^{d_model × d_ff} = First layer weights
b₁ ∈ ℝ^d_ff = First layer bias
W₂ ∈ ℝ^{d_ff × d_model} = Second layer weights
b₂ ∈ ℝ^d_model = Second layer bias
d_ff = feedforward hidden dimension (typically 4 × d_model)

GPT-2 Configuration:

d_model = 768
d_ff = 3,072 (exactly 4 × 768)

6. GeLU Activation Function

Gaussian Error Linear Unit:

GeLU(x) = x · Φ(x)

Where Φ(x) is the cumulative distribution function of standard normal distribution:

Φ(x) = ½[1 + erf(x/√2)]

Approximation (commonly used in practice):

GeLU(x) ≈ 0.5x(1 + tanh[√(2/π)(x + 0.044715x³)])

7. Positional Encoding

Learned Positional Embeddings (GPT-2)

E_total(t, p) = E_token(t) + E_pos(p)

Where:

E_token ∈ ℝ^{50,257 × 768} = token embedding matrix
E_pos ∈ ℝ^{1,024 × 768} = position embedding matrix
t = token ID
p = position (0 to 1,023)

Sinusoidal (Original Transformer)

PE_{(pos, 2i)} = sin(pos / 10000^2i/d_model)

PE_{(pos, 2i+1)} = cos(pos / 10000^2i/d_model)

Where:

pos = position in sequence
i = dimension index
d_model = model dimension (768)

8. Language Modeling Head

Project to vocabulary:

logits = x_final · W_LM

Apply softmax to get probabilities:

P(token_i) = exp(logit_i) / Σ_j=1^V exp(logit_j)

Where:

x_final ∈ ℝ⁷⁶⁸ = output from last Transformer block
W_LM ∈ ℝ^{768 × 50,257} = LM head weight matrix
V = 50,257 (vocabulary size)
logits ∈ ℝ^50,257 = scores for each token

9. Temperature Scaling

Temperature parameter controls randomness in sampling:

P_T(token_i) = exp(logit_i / T) / Σ_j=1^V exp(logit_j / T)

Temperature effects:

T = 0: Greedy decoding (always pick highest probability)
T = 1: Sample from raw distribution
T > 1: More random (flattens distribution)
T < 1: More confident (sharpens distribution)

10. Parameter Count Formulas

Per Transformer block:

Multi-head attention:

Paramsattention = 4 × dmodel² = 4 × 768² = 2,359,296

Feedforward network:

ParamsFFN = 2 × (dmodel × dff) = 2 × (768 × 3,072) = 4,718,592

Total per block:

Paramsblock = 2.36M + 4.72M ≈ 7.1M

Full GPT-2 model:

Params_total = V × d_model + L × 12d_model²

Where:

V = 50,257 (vocabulary size)
d_model = 768 (embedding dimension)
L = 12 (number of blocks)

GPT-2 breakdown:

Token embeddings: 50,257 × 768 = 38,597,376
Position embeddings: 1,024 × 768 = 786,432
12 Transformer blocks: 12 × 7.1M = 85.2M
Total: ~117M parameters

11. Training Loss Function

Cross-entropy loss for next-token prediction:

ℒ = -Σ_i=1ⁿ log P(x_i+1 | x₁, …, x_i)

Per-token loss:

ℒ_i = -log(exp(logit_target) / Σ_j=1^V exp(logit_j))

Where:

x_i = token at position i
P(x_i+1 | x₁, …, x_i) = predicted probability for next token
logit_target = score for the actual next token

📚 Formula Summary

Attention	softmax(QK^T/√d_k) V
Multi-Head	Concat(heads) W^O
Feedforward	GeLU(xW₁)W₂
Layer Norm	γ(x-μ)/√(σ²+ε) + β
Residual	x + Sublayer(LayerNorm(x))
Position	E_token + E_pos
LM Head	softmax(x·W_LM)

📖 References

“Attention Is All You Need” (Vaswani et al., 2017) – Original Transformer
“Language Models are Unsupervised Multitask Learners” (Radford et al., 2019) – GPT-2
“Layer Normalization” (Ba et al., 2016)
“Gaussian Error Linear Units (GELUs)” (Hendrycks & Gimpel, 2016)
“RoFormer” (Su et al., 2021) – Rotary Position Embeddings

LLM_log #003 Understanding Large Language Models – Transformers illustrative explanation

LLM_log #003 Understanding Large Language Models – Transformers illustrative explanation

🚀 Understanding Large Language Models: A Complete Visual Guide

🎯 What You’ll Learn

Tutorial Overview

1. The Three Main Components

Figure 3-4: The Three Main Components of a Transformer LLM

2. The Forward Pass

GPT-2 Specifications

Step 1: Tokenization

Figure 3-5: Tokenizer Vocabulary & Token Embeddings

Step 2: Through Transformer Stack

Figure 3-6: Complete Forward Pass & Probability Distribution

Step 3: Decoding Strategies

Figure 3-7: Choosing a Token – Decoding Strategies

3. Positional Encodings

Figure 3-8: Positional Encodings – Teaching the Model About Order

4. Inside Transformer Blocks

The Stack

Figure 3-11: Inside the Transformer – Stack of Processing Blocks

Inside One Block

Figure 3-12: Inside a Transformer Block

5. The Feedforward Network

Figure 3-13: Feedforward Neural Network – Where Knowledge Lives

6. Self-Attention in Action

Figure 3-14: Self-Attention Incorporates Relevant Context

7. Mathematical Deep Dive

Part 3: Attention Mechanism – Mathematical Deep Dive

8. Multi-Head Attention

Figure 3-17: Complete Flow Through One Transformer Block

🎓 Final Synthesis

The Complete Flow

Key Takeaways

🎉 You’ve Completed the Guide!

📐 Appendix: Complete Mathematical Reference

1. Attention Mechanism

Single-Head Attention

Multi-Head Attention

2. Softmax Function

3. Layer Normalization

4. Residual Connections

5. Feedforward Neural Network

6. GeLU Activation Function

7. Positional Encoding

Learned Positional Embeddings (GPT-2)

Sinusoidal (Original Transformer)

8. Language Modeling Head

9. Temperature Scaling

10. Parameter Count Formulas

11. Training Loss Function

📚 Formula Summary

📖 References

Recent Posts

Search