🎯 What’s Covered in Part 2

Building on the foundations from Part 1, we now dive into the practical systems and methods that make language models work at scale. You’ll learn how to implement parallelism strategies that enable training on multiple GPUs, understand the Chinchilla scaling laws that tell you the optimal model size for your compute budget, master the art of efficient benchmarking and profiling, explore the critical data pipeline decisions that make or break model quality, and implement tokenization from scratch using the BPE algorithm that powers models like GPT.

Making Parallelism Work

Data parallelism is very natural when training language models – you can distribute your batch across multiple GPUs and aggregate gradients. This is the foundation of scaling training.

Model parallelism techniques like FSDP (Fully Sharded Data Parallel) are more complex to implement from scratch, but understanding the principles is crucial. In practice, you’ll implement simplified versions while learning about the full techniques.

🔴 The Most Important Practice: Benchmarking

You can implement things, but unless you have feedback on how well your implementation is going and where the bottlenecks are, you’re just flying blind.

Always benchmark and profile. This is probably the most important habit to develop. Without measurement, you can’t optimize effectively.

The Fundamental Question

If I give you a FLOPs budget, what model size should you use? This isn’t a trivial question – if you use a larger model, you can train on less data. If you use a smaller model, you can train on more data.

What’s the right balance? This has been studied extensively and figured out by a series of papers from OpenAI and DeepMind.

💡 Chinchilla Optimal: The 20:1 Rule

For every compute budget (number of FLOPs), you can vary the number of parameters in your model and measure how good that model is. This reveals an optimal parameter count for each level of compute.

The simple rule of thumb: If you have a model of size N parameters, multiply by 20 – that’s the number of tokens you should train on.

Example: A 1.4 billion parameter model should be trained on 28 billion tokens (1.4B × 20 = 28B).

⚠️ Important Limitation

This doesn’t take into account inference cost. The Chinchilla scaling laws tell you how to train the best model regardless of how big that model is.

In production, you often want smaller models that are cheaper to run, even if they’re slightly less accurate. Scaling laws give you the ceiling – business requirements determine where you land.

The Process

The basic idea: for every compute budget, you train models with different parameter counts and plot the results. When you plot these optimal points, the relationship is remarkably linear.

You can fit a curve to this data and extrapolate. If you had, say, 1e22 FLOPs available, what should the parameter size be? The scaling laws give you the answer.

Real stakes: Once you run out of your FLOPs budget, that’s it. You have to be very careful in terms of how you prioritize what experiments to run – which is exactly what frontier labs face all the time.

🚀 Data is Everything

You’ve got scaling laws, you have systems, you have your architecture. Now comes the critical question: what data do you train on?

The Data Pipeline Components

The data pipeline involves several critical stages:

• Evaluation: How do you measure if your data is good?
• Curation: Selecting high-quality sources
• Transformation: Converting raw data into trainable format
• Filtering: Removing low-quality or harmful content
• Deduplication: Ensuring unique training examples
• Mixing: Combining different data sources in optimal ratios

Each of these stages has enormous impact on final model quality. Bad data in, bad model out – no matter how good your architecture is.

💡 What is Tokenization?

Tokenization is the process of taking raw text (unicode strings) and turning it into a sequence of integers, where each integer represents a token.

We need procedures that:

• Encode strings to tokens
• Decode tokens back into strings

The vocabulary size is the number of unique tokens – essentially, the range of possible integer values.

How Tokenizers Actually Work

The most common approach is BPE (Byte Pair Encoding), used by GPT models. Here are some key observations from looking at real tokenizers:

• Spaces matter: Unlike classical NLP where spaces disappear, everything is accounted for. These are meant to be reversible operations.
• Space position: By convention, the space usually precedes the token. So “hello” and ” hello” are completely different tokens.
• Number handling: Numbers get chopped into pieces left-to-right, which isn’t semantic but reflects the statistical patterns in training data.

⚠️ Tokenization Quirks

The fact that “hello” and ” hello” are different tokens might make you a little squeamish. It can cause problems, but that’s how modern tokenizers work.

Why it matters: The model learns completely different representations for a word depending on whether it appears at the start of a sentence or in the middle. This affects model behavior in subtle ways.

Compression Ratio

An important metric is the compression ratio: the number of bytes divided by the number of tokens.

How many bytes are represented by a single token? For GPT-2’s tokenizer, the answer is approximately 1.6.

What this means: Every token represents 1.6 bytes of text on average. Higher compression ratios mean more efficient encoding, which translates to faster training and inference.

💡 The Round-Trip Test

A critical sanity check: when you encode a string to tokens and then decode it back, you should get exactly the original string.

This reversibility is essential. If your tokenizer loses information, your model can never learn to generate that information correctly.

The BPE Algorithm

The BPE process has two main stages:

1. Pre-tokenization: Split text into initial units (usually by whitespace and punctuation)
2. Tokenization: Iteratively merge the most frequent pairs of tokens

The pre-tokenizer puts the space at the front of words – this is built into the algorithm, not an accident.

⚠️ Implementation Warning

Implementing the BPE tokenizer from scratch has been surprisingly challenging for many students. It seems simple conceptually, but the devil is in the details.

Budget time accordingly: This part of the assignment takes more work than you might expect. The edge cases and efficiency considerations add up quickly.

How Assignments Are Evaluated

Grading has multiple components:

• Correctness: Did you implement the algorithm correctly? Tests must pass.
• Performance: Did your model achieve a certain level of loss?
• Efficiency: Is your implementation efficient enough?

Each problem part in the assignment has points associated with it, giving you granular feedback on what’s working and what needs improvement.

💡 The Leaderboard Challenge

For assignments with leaderboards:

• Scaling Laws: Minimize loss given your FLOPs budget
• Training: Minimize perplexity on open web text with 90 minutes on an H100

These competitive elements simulate real-world constraints where compute is expensive and you need to make every FLOP count.

Key Takeaways from Part 2

• Always benchmark: Measurement drives optimization
• Scaling laws work: The 20:1 rule (tokens to parameters) is remarkably reliable
• Data quality matters: No amount of compute can fix bad training data
• Tokenization is subtle: Small choices in tokenization ripple through the entire model
• Implementation details count: The gap between understanding and correct implementation is significant

What’s Next?

• Alignment techniques: supervised fine-tuning and reinforcement learning
• Preference data and synthetic data generation
• Advanced architecture optimizations
• Production deployment considerations
• Verifiers and evaluation frameworks

We’re building toward a complete understanding of the entire language model stack – from first principles to production systems.