🤖 Understanding Transformers: A Progressive Q&A Journey

From basic embeddings to self-attention to generation – built step by step through questions

Prerequisites: Basic understanding of matrix multiplication
Reading time: 20-30 minutes
What you’ll learn: How transformers work from first principles

---

📚 What is a Transformer?

• Architecture: Neural network for processing sequences (text, images, etc.)
• Key Innovation: Self-attention mechanism (all words look at all other words)
• Parallel Processing: Unlike RNNs, processes entire sequence simultaneously
• Used in: GPT, BERT, Claude, ChatGPT, and most modern language models

---

1️⃣ Input & Embeddings

Q: What is the input to a transformer?

A: A sequence of tokens, where each token is converted to an embedding vector.

Example: "The cat sat"

Step 1: Tokenization
"The"  →  token_id: 245
"cat"  →  token_id: 1891  
"sat"  →  token_id: 3421

Step 2: Embedding (each token → vector)
245   →  [0.2, 0.5, 0.1, 0.8, ...]  (512 dimensions)
1891  →  [0.7, 0.1, 0.9, 0.2, ...]
3421  →  [0.3, 0.8, 0.4, 0.1, ...]

Step 3: Input to Transformer = [3 × 512] matrix

Q: Are embeddings like PCA? (100k vocabulary → 512 dimensions)

A: Conceptually similar, but technically different!

Aspect	PCA	Embeddings
Dimensionality	100,000 → 512 ✓	100,000 → 512 ✓
Method	Linear projection	Neural network (learned)
Objective	Maximize variance	Minimize task loss
Semantic meaning	❌ No	✅ Yes!

Better analogy: Embeddings are like an autoencoder bottleneck trained with semantic supervision. Similar words get similar vectors!

Q: What range are embedding values?

A: Any real numbers (NOT limited to 0-1). Typical range: -3 to +3

"cat"   →  [ 0.23, -0.81,  0.45,  1.23, -0.67, ... ]
"dog"   →  [ 0.19, -0.75,  0.52,  1.18, -0.71, ... ]  ← similar!
"car"   →  [ 2.13,  0.45, -1.89,  0.12,  3.45, ... ]  ← different

Q: How many tokens can be input at once?

A: Depends on the model’s context window:

Model	Max Tokens
GPT-2	1,024
GPT-3.5	4,096
GPT-4	8,192
Claude 3.5	200,000

---

2️⃣ Self-Attention Mechanism

Q: Where does the input go?

A: Into the Self-Attention block (first layer)

Input: [10 × 512]
    ↓
Self-Attention
    ↓
Output: [10 × 512]  (same shape!)

Q: What is self-attention?

A: Each word looks at ALL other words to understand context

Example: "The cat sat on the mat"

When processing "sat":
  - Look at "cat" → high attention (who sat?)
  - Look at "mat" → high attention (sat where?)
  - Look at "The", "on", "the" → low attention

Result: "sat" creates a new representation that includes 
        information from "cat" and "mat"

Key insight: EVERY word does this with EVERY other word simultaneously!

Q: How are vectors actually processed?

A: Through Query, Key, Value (Q, K, V) matrices

Step 1: Create Q, K, V

Input [10×512] × W_Q [512×64] = Q [10×64]
Input [10×512] × W_K [512×64] = K [10×64]
Input [10×512] × W_V [512×64] = V [10×64]

Compressed from 512 → 64 dimensions

Step 2: Calculate Attention Scores

Q [10×64] × K^T [64×10] = Scores [10×10]

Example for "cat sat mat":
        cat   sat   mat
cat  [  0.5   0.3   0.2 ]
sat  [  0.8   0.1   0.7 ]  ← "sat" attends to "cat" & "mat"
mat  [  0.2   0.6   0.4 ]

Step 3: Apply Attention to Values

Scores [10×10] × V [10×64] = Output [10×64]

For "sat": 0.8×V_cat + 0.1×V_sat + 0.7×V_mat
         = new "sat" vector (mix of all words)

Step 4: Project Back

Output [10×64] × W_O [64×512] = Final [10×512]

Back to 512 dimensions!

Q: Are Q, K, V different for each word?

A: The weight matrices (W_Q, W_K, W_V) are SHARED by all words, but each word gets different Q, K, V vectors when multiplied!

LEARNED PARAMETERS (same for ALL words):
W_Q [512×64]  ← trained once, used by everyone
W_K [512×64]  
W_V [512×64]  

EACH WORD GETS DIFFERENT VECTORS:
"cat" [512] × W_Q = Q_cat [64]  ← unique
"sat" [512] × W_Q = Q_sat [64]  ← unique
"mat" [512] × W_Q = Q_mat [64]  ← unique

Same matrix W_Q, different input → different output!

Analogy: W_Q is like a calculator function. Same function, different inputs → different outputs!

---

3️⃣ The Big Picture

Q: Can I think of this as words getting “colored” by context?

A: Perfect intuition! YES! 🎯

BEFORE Self-Attention (independent):
"cow"   = [0.2, 0.5, 0.1, ...]  ← generic cow
"milk"  = [0.7, 0.3, 0.8, ...]  ← generic milk
"white" = [0.9, 0.1, 0.4, ...]  ← generic white

AFTER Self-Attention (mixed/colored):
"cow"   = 0.4×cow + 0.3×milk + 0.3×white
          ↑ now contains info about milk & white!

All three are "whitened" by context!

Context matters:
“cow milk white” → cow becomes dairy-related
“cow grass field” → cow becomes farm-related
Same word, different context → different representation!

Q: Summary of one self-attention layer?

A: Complete flow:

Input: [10 × 512]
    ↓ multiply with W_Q, W_K, W_V
Q, K, V: [10 × 64 each]
    ↓ Q × K^T
Attention Scores: [10 × 10]
    ↓ Scores × V
Attended Output: [10 × 64]
    ↓ multiply with W_O
Final Output: [10 × 512]

Learned Parameters (4 matrices):

• W_Q [512×64]
• W_K [512×64]
• W_V [512×64]
• W_O [64×512]

Result: Words are now context-aware!

---

4️⃣ Why So Much Computing Power?

Q: It’s just matrix multiplication – why so much processing power?

A: Because the matrices are MASSIVE and repeated many times!

One self-attention layer for GPT-3:

Input: [2048 × 12288]  (2048 tokens, 12288 dims)

Q,K,V calculations: 3 × 300 billion operations
Attention scores: 4 billion operations
W_O projection: 25 billion operations

Total: ~330 billion operations PER LAYER

GPT-3 has 96 layers:
330 billion × 96 = 31 TRILLION operations

Plus: batch processing (32-64 sentences), training on billions of examples

Q: Wait – you said 512 dimensions earlier, now 12,288?

A: Different models use different embedding sizes!

Model	Embedding Dims
GPT-2 Small	768
GPT-3 Small	2,048
GPT-3 Large	12,288

I used 512 for simple explanation. Real GPT-3: each token = 12,288 numbers!

---

5️⃣ What Happens After Self-Attention?

Q: After self-attention outputs [10×512], then what?

A: Three more steps to complete one transformer layer:

Step 1: Add & Norm (Residual Connection)

Self-Attention output [10×512]
    +
Original input [10×512]
    ↓
Layer Normalization
    ↓
[10×512]

Step 2: Feed-Forward Network

[10×512] × W1 [512×2048] = [10×2048]  (expand)
    ↓
ReLU activation
    ↓
[10×2048] × W2 [2048×512] = [10×512]  (compress back)

Step 3: Add & Norm Again

Feed-Forward output [10×512]
    +
Input to feed-forward [10×512]
    ↓
Layer Normalization
    ↓
[10×512] ← SAME SHAPE!

This is ONE transformer layer. Then [10×512] goes to next layer (repeat 6-96 times)

Q: Does every layer have both QKV and feed-forward?

A: YES! Every layer has BOTH:

ONE Transformer Layer =
    ├─ Self-Attention (Q,K,V,O matrices)
    └─ Feed-Forward Network (W1, W2 matrices)

Each layer has its OWN parameters!
Layer 1's W_Q ≠ Layer 2's W_Q

Q: How many layers in general?

A: Depends on model size:

Model	Layers
BERT Base	12
GPT-2	12-48
GPT-3 Large	96
Claude	~80-100

More layers = deeper understanding but slower

Q: Any analogy with ResNet?

A: YES! Very similar hierarchy:

CNNs (ResNet)	Transformers
Layer 1-3: Edges, corners	Layer 1-3: Syntax, grammar
Layer 4-10: Shapes, parts	Layer 4-10: Word meaning, entities
Layer 11+: Objects, faces	Layer 11+: Abstract concepts, reasoning

Plus: Both use residual connections! This allows training 96+ layers without vanishing gradients.

Q: Why are residual connections added?

A: Two main reasons:

1. Gradient Flow (Training)

With residual:
Layer 96 →──→──→──→ Layer 1  (direct path!)
Gradient flows easily backward

2. Learning Refinements

Layer 1: "cat" → "cat" + [grammar info]
Layer 2: result → result + [word relations]
Layer 3: result → result + [context info]

Each layer adds refinements, not replacements

Analogy: Like editing a photo – each layer makes small adjustments, not starting from scratch.

---

6️⃣ Where is the “AI” Knowledge?

Q: Is all the knowledge in these QKV matrices and NN weights?

A: YES! Exactly right! 🎯

Per Layer:
- W_Q [512×64]     ← learned during training
- W_K [512×64]     
- W_V [512×64]     
- W_O [64×512]     
- W1 [512×2048]    
- W2 [2048×512]    

× 12 layers = ~50 million parameters

GPT-3: 175 BILLION parameters total

What Gets Learned:

"Paris is the capital of ___"
→ Encoded in specific weight patterns

"2 + 2 = ___"  
→ Encoded in different weight patterns

Grammar, facts, reasoning → ALL in matrices

Everything the model “knows” = patterns in weight matrices
No database, no lookup table – just matrix multiplication!

Q: Why fine-tune instead of training from scratch?

A: Because fine-tuning is cheaper and faster!

	Cost	Time
Train from scratch	$10 million	3 months
Fine-tune existing	$1,000	1 day

Fine-tuning adjusts ~1% of weights for specialized tasks (medical, legal, code, etc.)

---

7️⃣ Architecture Variants

Q: Encoder vs Decoder – what’s the difference?

A: Different use cases:

Type	How it works	Examples
Encoder	Processes input all at once	BERT (understanding)
Decoder	Generates one token at a time	GPT, Claude (generation)

Q: What is multi-head attention?

A: Running attention 8 times in parallel!

Head 1: [10×512] → Q,K,V → attention → [10×64]
Head 2: [10×512] → Q,K,V → attention → [10×64]
...
Head 8: [10×512] → Q,K,V → attention → [10×64]

Concatenate: [10×64×8] = [10×512]

Each head learns different patterns:

• Head 1: subject-verb relationships
• Head 2: adjective-noun relationships
• Head 3: long-range dependencies
• etc.

---

8️⃣ Real-World Models

Q: What is BERT used for?

A: BERT = Encoder-only = Understanding, NOT generation

What BERT does well:

• Classification: “This movie is great!” → Positive/Negative
• Named Entity Recognition: “Apple hired Tim Cook” → Apple=Company, Tim Cook=Person
• Question Answering: Extract answers from context
• Sentence Similarity: Compare text similarity

What BERT can’t do:

• ❌ Text generation
• ❌ Chatbot conversations
• ❌ Code completion

Why? BERT sees ENTIRE sentence at once (bidirectional). GPT sees only previous tokens (causal).

Q: Is BERT like a feature extractor?

A: YES! Perfect analogy!

CNNs:  Image → ResNet → Features [2048] → Classifier
BERT:  Text → BERT → Features [768] → Classifier

Exactly like ResNet extracts image features, BERT extracts text features!

Q: Where does CLIP fit in?

A: CLIP = Dual transformer (vision + text)

Image Encoder (ViT): Dog photo → Features [512]
Text Encoder (BERT): "a dog" → Features [512]

Training: Make these vectors similar!

Use cases:

• Search images with text: “sunset beach”
• Zero-shot classification: Image + [“dog”, “cat”, “bird”]

---

9️⃣ Decoder & Generation

Q: What is decoder input/output?

A: Decoder (GPT, Claude) works like this:

Input: Tokens generated SO FAR

Example: "The cat sat on the" [5×512] embeddings

Output: Probabilities for NEXT token

"mat"   → 0.35 (35%)
"floor" → 0.25 (25%)
"chair" → 0.15 (15%)

Pick highest → "mat"

Key: Decoder generates one token at a time (autoregressive)

Q: What happens after decoder layers?

A: One final projection to vocabulary:

Decoder layers: [5×512]
    ↓
W_output: [512×50000]
    ↓
Result: [5×50000] probabilities
    ↓
Take LAST position → [50000] probabilities
    ↓
Pick highest probability word

Q: Does input keep growing during generation?

A: YES! The input sequence grows:

Step 1: [10×512] → predict "mat"
Step 2: [11×512] → predict "and" (added "mat")
Step 3: [12×512] → predict "looked" (added "and")
...
Continue until  or max length

Important: Input is always embeddings [N×512], output is always probabilities [N×50000]

Q: Is this autoregressive?

A: YES! Literally an autoregressive (AR) process

Time Series AR	Language Model AR
x_t = f(x_{t-1}, x_{t-2}, …)	word_t = f(word_1, …, word_{t-1})
Current depends on previous	Current depends on all previous

GPT = Generative Pre-trained Transformer (Autoregressive)

Q: What about the original 2017 Transformer?

A: Original uses BOTH encoder and decoder (for translation)

ENCODER (source):
English "Hello world" [2×512]
    ↓
Output: [2×512] encoded representation

DECODER (target):
French: Generate word by word
Step 1: "Bonjour" → predict "le"
Step 2: "Bonjour le" → predict "monde"

Why M ≠ N? Different languages, different lengths!

Q: What’s inside the original decoder?

A: 3 parts (not 2 like encoder):

1. Masked Self-Attention – Can only see previous tokens
2. Cross-Attention (NEW!) – Decoder reads encoder output
3. Feed-Forward Network – Same as encoder

Masking: Prevents seeing future words (-∞ in attention matrix)

         Bonjour   le    monde
Bonjour [  ✓      -∞     -∞   ]
le      [  ✓       ✓     -∞   ]  
monde   [  ✓       ✓      ✓   ]

Q: Was it only for translation?

A: Original 2017 paper: YES. But architecture is general!

Year	Model	Architecture	Use
2017	Transformer	Encoder-Decoder	Translation
2018	BERT	Encoder-only	Understanding
2018	GPT	Decoder-only	Generation
2019	T5	Encoder-Decoder	Multi-task

---

🎯 Key Takeaways

1. Embeddings compress 100k vocabulary → 512 dims with semantic meaning
2. Self-Attention lets every word “see” every other word via Q, K, V matrices
3. Multi-head runs 8 parallel attention heads to learn different patterns
4. Layers stack (6-96) from syntax → semantics → reasoning
5. Residual connections enable deep networks by allowing gradient flow
6. All knowledge is stored in weight matrices (billions of parameters)
7. Encoder (BERT) for understanding, Decoder (GPT) for generation
8. Autoregressive generation: one token at a time, feeding output back as input

---

Understanding built progressively through questions – from embeddings to generation! 🎯