🎯 What You’ll Learn

In this comprehensive guide, we’ll explore the evolution of transformer architectures from the original “Attention is All You Need” paper to modern implementations. You’ll discover why today’s language models use specific design choices like RoPE position embeddings and SwiGLU activations, understand the trade-offs between serial and parallel layer arrangements, and learn how to make informed decisions about hyperparameters like head dimensions and aspect ratios. By examining the collective wisdom from hundreds of trained models, you’ll gain practical insights into what makes transformers work at scale.

Course Goals and Starting Point

Welcome to this deep dive into transformer architectures and training. This lecture has been titled “everything you didn’t want to know about LLM architecture and training” because we’re going to get into some of the nitty-gritty details that most other classes would spare you from—questions like what should my hyperparameters be? What architectural choices actually matter?

We’ll start with a quick recap of the transformer architecture, giving you two variants: the standard transformer you might see in CS224N, and then the modern consensus variant that you actually implement in practice. From there, we’ll take a much more data-driven perspective to understanding transformer architectures.

The central question: People have trained lots of LLMs at this point, and you can read all those papers to understand what has changed and what has remained common. Through this almost evolutionary analysis, we can try to understand what are the things that are really important to make transformers work. Today’s theme follows the class philosophy that the best way to learn is hands-on experience, but since we can’t train all these transformers ourselves, we’ll learn from the experience of others.

💡 Key Architectural Formulas

The original positional encoding uses sine and cosine functions at different frequencies to capture both higher and lower frequency positional information:

$$PE_{(pos,2i)} = \sin(pos/10000^{2i/d_{\text{model}}})$$
$$PE_{(pos,2i+1)} = \cos(pos/10000^{2i/d_{\text{model}}})$$

The feed-forward network in the original transformer applies a simple two-layer transformation with ReLU activation:

$$\text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2$$

🔴 Learning From Collective Experience

The philosophy driving this course is simple: The best way to learn is through hands-on experience. Since we can’t train all these transformers ourselves, we learn from the experience of others. By examining hundreds of published language models, we can perform an almost evolutionary analysis to understand which components are truly essential for transformer success.

The Challenge of Model Proliferation

The proliferation of language models has reached a point where keeping track of them all has become a significant challenge in itself. This explosion represents both an exciting opportunity and a practical challenge for researchers and practitioners. Each model brings different strengths, architectures, and use cases to the table.

However, from this diversity, we can extract common design principles that have proven successful across different implementations. This data-driven approach to understanding transformers forms the foundation of modern architecture design.

Dense Architecture Variations and Data

The landscape has become remarkably diverse. When you start looking deeper into what’s available, the sheer volume becomes overwhelming—you’ll find Gemma 3 and Qwen 2.5, and honestly, you can’t even cover the screen with all these different models. The proliferation of language models has reached a point where keeping track of them all has become a significant challenge in itself.

This explosion of models represents both an exciting opportunity and a practical challenge for researchers and practitioners. There’s simply a lot of models out there now, each with different strengths, architectures, and use cases. The field has moved from having a handful of notable language models to an ecosystem so rich and varied that comprehensive coverage requires significant effort and resources.

Convergent Evolution in Neural Architectures

To make sense of this architectural diversity, a comprehensive spreadsheet tracking what all these models are doing becomes essential—spanning from the original transformer in 2017 all the way to the newest models in 2025. What emerges is a fascinating pattern of convergent evolution in neural architectures.

Take position embeddings as an example: researchers used to experiment with all sorts of approaches including absolute, relative, RoPE, and even ALiBi for some teams. However, starting around 2023, virtually everyone has converged on RoPE as the standard approach. This convergence tells us something important about what actually works at scale.

💡 Architecture, Hyperparameters, and Training

First: Architecture Variations – We’ll explore activations, feed forwards, attention variants, and position embeddings. These are the fundamental building blocks that determine how information flows through your model and how effectively it can learn from data.

Second: Intelligent Hyperparameter Selection – Once we’ve nailed down the architecture, we need to intelligently select hyperparameters. Questions arise like: How big to make the hidden dimension? What size to use for the \(ff\_dim\) in the MLP projection layer? Whether \(multi\_head\) dimensions should always sum to \(model\_dim\)? How many vocabulary elements to include? These aren’t choices you want to pick out of a hat—they have significant implications for model performance and computational efficiency.

Third: Training Strategy – The approach to training, including optimization techniques, learning rate schedules, and other critical decisions that affect whether your model converges to a good solution.

🔴 The State of Architectural Consensus

Two important observations emerge from this analysis:

First, there’s still not much consensus on many architectural choices, despite some convergent evolution toward what we might call “LLaMA-like architectures” in recent years. Researchers continue to experiment with different approaches—swapping between LayerNorm and RMSNorm, choosing between serial versus parallel layer arrangements, and exploring various activation functions.

While there’s one choice that basically everyone has adopted since the very first GPT (which we’ll explore in detail), the landscape remains rich with variations that we can learn from. Each variation offers insights into the optimal design space for large language models, and by studying these patterns, we can make more informed decisions about our own architectures.

The remainder of this guide will systematically walk through these choices, examining the evidence for each decision and understanding not just what works, but why it works.

Pre-Norm vs Post-Norm: A Critical Architecture Choice

The original transformer paper used a post-norm approach where you had your residual stream, and layer norms were applied after every subcomponent. They would do multi-head attention, add back to the residual stream, then layer norm that result. The same pattern applied to the fully connected layer.

Very early on though, people realized that moving the layer norm to the front of the non-residual part performed much better in many different ways. Basically, almost all modern transformer models use this pre-norm approach (right image below).

💡 Gradient Behavior Insights

The arguments centered around gradient behavior across layers. With pre-norm, gradient sizes remained constant, whereas post-norm without warm-up would cause gradients to blow up.

Early work by Salazar and Yen identified that post-norm training showed many more loss spikes and generally unstable behavior. You’d see gradient norms spiking and remaining higher throughout training compared to the smoother pre-norm approach. This reinforced the view of pre-norm and other layer norm tricks as essential stability-inducing aids for training neural networks.

🔴 The Double Norm Innovation

Recently, there’s been a new innovation called “double norm”—a variant that didn’t exist when this material was first presented. While we know putting layer norms directly in the residual stream is bad, recent models like Grok and Gemma II have started adding layer norms both before and after the transformer blocks.

Some models like Llama II only add layer norm after the feed-forward and multi-head attention components. This approach has shown some promise for improved stability in larger models, though pre-norm had been dominant for quite a while.

What is RMSNorm?

RMSNorm simplifies the normalization process by dropping the mean adjustment. You don’t subtract the mean and you don’t add a bias term $\beta$. Notable models like the Llama family, PaLM, Chinchilla, and T5 have all moved to RMSNorm.

Why the switch? One reason is that it doesn’t really make a difference—if you train models with RMSNorm, it performs just as well as LayerNorm, so there’s a simplification argument.

The Real Argument: Speed and Memory

The real argument often given in papers is that RMSNorm is faster and just as good. If you don’t subtract the mean, that’s fewer operations. If you don’t have to add that bias term $\beta$ back, that’s fewer parameters to load from memory into your compute units.

Now, you might be thinking—didn’t you tell us that only matrix multiplies matter for runtime? And this isn’t a matrix multiply, so why should we care?

🔴 Memory Movement: The Hidden Bottleneck

Here’s the key insight for architecture design: You can’t just think about flops—you also have to think carefully about memory movement.

Even though tensor contractions are 99.8% of the flops, operations like softmax and layer norms represent only 0.17% of flops but actually take up 25% of the runtime! The big reason is that these normalization operations incur a lot of memory movement overhead.

So it does matter to optimize these lower-level operations, because it’s not just about flops—it’s about memory movement. When we talk about GPU architectures, thinking about memory becomes very, very important, not just flops.

💡 Concrete Performance Gains

This is one reason why RMSNorm has become much more popular. Narang et al. in 2020 had a nice ablation study showing concrete results:

• Vanilla transformer: 3.5 steps per second
• With RMSNorm: 3.68 steps per second

Not a huge gain, but it’s essentially free. And you get a final loss that’s actually lower than the standard transformer. So we’ve gotten runtime improvements and, in this case, even loss improvements—that’s a win-win situation for us.

The Original FFN with Bias Terms

The original transformer feed-forward network would have your inputs $X$ going through a linear layer with a bias term, then ReLU it, and then a second linear layer wrapping around it:

$$\text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2$$

But most implementations, if they’re not gated units, actually just drop the bias terms entirely. You can make this argument from basically the same kinds of underlying principles—they perform just as well, and matrix multiplies are apparently all that you need to get these models to work.

Generalizable Lessons in Architecture Design

This trend extends beyond just bias terms. Basically everyone does pre-norm or at least they do the layer norms outside of the residual stream, which gives you nicer gradient propagation and much more stable training.

Most people, or almost everybody, does RMS norm in practice because it works almost as well as layer norm but has fewer parameters to move around. This idea of dropping bias terms just broadly applies—a lot of these models just don’t have bias terms in most places.

The Activation Function Zoo

There’s a whole big zoo of activations: ReLU, GELU, Swish, GLU, and many others. And then there are different kinds of MLPs: GeGLU, ReGLU, SwiGLU, and more variants.

The bottom line? It really does matter—SwiGLU and other GLU variants consistently work well, and you should think about them carefully because they do work and you should internalize that pattern.

💡 Historical Adoption Patterns

The GPT family of models (GPT-1, GPT-2, GPT-3, and GPT-J) all use GELU. The original transformer and some of the older models used ReLU, but really almost all the modern models have switched to the gated linear units, like SwiGLU and GeGLU.

The Google folks really pushed for this approach with PaLM and T5, and since it’s been tried and true, basically almost all models post-2023 use a gated linear unit.

From Standard FF to Gated Variants

Originally, we had our fully connected layer with ReLU. GLUs modify the “first part” of a feedforward layer. Now, instead of doing just linear and ReLU, what we do is gate the output with an element-wise linear term:

$$y_i^{(x)} = \max(0, xW_1) \cdot xW_2$$

This introduces gating where you’re multiplying the ReLU activation by another linear transformation. The element-wise product ($\otimes$) creates a gating mechanism that allows the model to control information flow more flexibly.

💡 SwiGLU and GeGLU Variants

Different GLU variants use different activation functions in the gating mechanism:

$$\text{GeGLU}(x) = (\text{GELU}(xW) \otimes xV)W_2$$

$$\text{SwiGLU}(x) = (\text{Swish}_1(xW) \otimes xV)W_2$$

where Swish is defined as \(x \mapsto x \cdot \text{sigmoid}(\beta x)\).

🔴 Why Non-Monotonicity Isn’t a Problem

There’s a small negative region to the left of zero where the derivative flips direction, which intuitively might seem problematic and could potentially trap activations at zero.

However, in practice, neural network optimization dynamics involve high learning rates with momentum, so you’re not really going to converge to that zero point. The activations end up being distributed all over the place, so this tiny negative piece doesn’t have a huge practical effect on the model.

💡 The Two-Thirds Rule

The standard practice is to make the hidden size—basically the output dimensionality of \(W\)—slightly smaller by a factor of two-thirds compared to non-gated counterparts.

This ensures that the total number of parameters in the entire gated system remains the same as the non-gated versions. It’s a conventional approach that most practitioners follow: for gated linear units, you simply make everything a bit smaller to maintain parameter matching across different architectures.

Computational Overhead Analysis

What really matters here is the memory pressure, and it will be exactly the same when you’re reading the same level of elements. So the extra computational overhead is negligible anyway, and actually the memory calculus remains the same.

It’s entirely possible that SwiGLU or GELU might be implemented with lookup tables internally in CUDA. The bottom line is that these concerns don’t manifest as practical bottlenecks.

Empirical Evidence: GLU Variants Win

Gated linear units clearly work. Looking at Shazeer’s original paper evaluating all these GLU variants, the evidence is compelling. Looking at CoLA and SST-2 performance, the GLU variants consistently perform better, with scores of 84.2, 84.12, 84.36, and 84.67.

They even provide standard deviations so you can determine how significant those results are, and they are indeed statistically significant.

🔴 GLU Isn’t Strictly Necessary, But…

Of course, GLU isn’t strictly necessary for a good model—just because it’s probably slightly better and everyone uses it doesn’t mean it’s essential. You can find examples of very high-performance models that don’t use GLU:

• GPT-3: No GLU, still high performance
• Nemotron 340B: Uses squared ReLU
• Falcon 2 11B: Uses standard ReLU

These are all relatively high-performance models, showing that GLU isn’t absolutely required. However, the evidence does point toward somewhat consistent gains from SwiGLU and GeGLU variants, which is why modern implementations use exactly those variants.

Serial Layer Arrangements: The Traditional Approach

Traditionally, transformer blocks operate in a serial fashion. For each block, outputs come in from the bottom, you perform your attention computation, pass that result forward, then do your MLP, and pass that computation forward. This inherently serial approach means you do attention first, then MLP—one after the other in sequence.

However, this serial connection can create certain parallelism constraints when you want to distribute computation across gigantic sets of GPUs, potentially leading to lower GPU utilization and more difficult systems concerns. These limitations have motivated researchers to explore alternative arrangements.

Parallel Layers: A Systems-Driven Innovation

To address the limitations of serial processing, several models have adopted what we call parallel layers. Instead of having serial computation of attention and MLP, they perform both operations simultaneously. You take your input \(x\) from the previous layer, compute both the MLP and attention side by side, then add them together into the residual stream for your output.

This approach was pioneered by GPT-J, an open source replication effort, and the folks at Google were bold enough to implement this at really large scale with PaLM, with many others following since.

💡 Computational Advantages of Parallel Architectures

When implemented correctly, this parallel formulation offers significant advantages. You can share LayerNorm computations, fuse matrix multiplications together, and achieve systems efficiencies that result in roughly 15% faster training speed at large scales.

The MLP and attention input matrix multiplications can be fused, leading to better GPU utilization. Ablation experiments showed small quality degradation at 8B scale but no quality degradation at 62B scale, suggesting the effect should be quality neutral at even larger scales like 540B parameters.

🔴 Current Adoption Status

Despite these advantages, parallel layers haven’t been quite as popular in recent years. Most models we’ve seen in the last year have used serial layers rather than parallel ones. The main exceptions include Cohere’s Command-R and Command-R Plus, along with Falcon 2 11B.

This trend suggests that while parallel layers offer clear computational benefits, the transformer community has largely settled on the traditional serial approach for most applications. The reasons for this preference relate to both implementation complexity and the trade-offs between computational efficiency and model expressiveness.

💡 Serial vs Parallel Efficiency

When asked whether serial is more efficient than parallel, it’s actually the reverse—parallel is more efficient than serial, which is why people are willing to make this architectural choice despite potential expressiveness trade-offs.

In some sense, you might expect serial processing to be more expressive because you’re composing two computations rather than adding them together. However, the benefit of parallel architectures lies in computational efficiency through kernel fusion and shared computations.

Architecture Design Summary

When we examine the evolution of neural network architectures over the past several years, fascinating patterns emerge in how researchers have approached fundamental design decisions. The landscape has been dominated by transformer-based models, but the specific architectural choices have varied significantly across different implementations and research groups.

What’s particularly interesting is how certain design patterns have become more prevalent over time. We see a clear trend toward pre-normalization techniques, which help with training stability, and the adoption of more sophisticated position embedding methods. The activation functions have also evolved, with many modern architectures moving beyond simple ReLU to more advanced alternatives like GELU and SwiGLU.

Absolute Position Embeddings: The Original Approach

The earliest models like GPT-1, GPT-2, GPT-3, and OPT used absolute embeddings, which simply add a learned position vector to the token embedding:

$$\text{Embed}(x, i) = v_x + u_i$$

This straightforward approach directly encodes positional information but lacks the ability to generalize well to sequences longer than those seen during training. If your model trains on sequences of length 512, it has no learned representation for position 513 and beyond.

💡 Why Sine and Cosine?

The sinusoidal position encodings use different frequency ranges to capture both higher and lower frequency positional information. Lower frequency components (with larger wavelengths) can represent coarse positional information over long distances, while higher frequency components capture fine-grained local position distinctions.

This mathematical formulation allows the model to potentially extrapolate to sequence lengths longer than those seen during training, since the sine and cosine functions are defined for all positions.

The Core Intuition: Relative Positions Matter

The high-level thought process behind RoPE is that what matters is the relative positions of vectors. If I have an embedding \(f(x, i)\), where \(x\) is the word I’m trying to embed and \(i\) is my position, then I should be able to write things down in this way:

There should exist a function \(f\) such that if I take the inner product of \(f(x, i)\) and \(f(y, j)\), I can write this down as some different function \(g(x, y, i-j)\), which depends on the two words and the difference in their positions:

$$\langle f(x, i), f(y, j) \rangle = g(x, y, i-j)$$

This definition enforces absolute position invariance—you only pay attention to how far apart these two words are.

💡 A Concrete Example: “We Know”

Let me illustrate with an example. Say my embedding for the word “we” is this arrow, and my embedding for “know” is this other arrow.

For the sequence “we know that”:

• “we” is at position 0 → I don’t rotate it at all
• “know” is at position 1 → I rotate it by one unit of rotation

For the sequence “of course we know”:

• “we” and “know” have the same relative positioning but are shifted by two positions
• “we” → I rotate by 2 positions
• “know” → I rotate by 3 positions

If you look at these two arrow pairs, they have the same relative angle, so their inner products are preserved!

🔴 The High-Dimensional Challenge

While it’s easy to think about in 2D where there’s only one way to rotate a vector, in high-dimensional spaces where we operate, it’s not obvious how to do this rotation.

The RoPE approach came up with the simplest but effective way: take your high-dimensional vector of dimension \(d\) and cut it into blocks of 2 dimensions. Every 2-dimensional pair gets rotated by some \(\theta\), creating a rotation speed for each pair.

The Rotation Matrix Structure

Since the \(\theta\) values are fixed and the matrices \(M\) are predetermined, this essentially becomes a fixed matrix multiplication with your vector. The beauty of this approach lies in its simplicity—there are no additive or cross terms that complicate the computation.

When considering which trigonometric functions to use, the key insight is to pair up coordinates and rotate them in 2D space, drawing motivation from complex number representations.

💡 Implementation Simplicity

From an implementation perspective, this approach avoids computational complexity because you’re not learning the $\theta$ parameters themselves. If you were optimizing these angles, you’d face the challenge of differentiating through trigonometric functions, but RoPE sidesteps this issue entirely by keeping them fixed.

The $\theta$ values that determine the rotation angles are hyperparameters, not trained parameters. Much like in the sines and cosines, there’s a schedule to the rotation angles that are determined upfront.

RoPE Application Flow

Whenever these attention computations are going to be done, you’re going to intervene on that layer to give you your position information. You’ve got the initial normal attention stuff at the very top, like query, keys, and values—these are your normal linear projections:

query_states = self.q_proj(hidden_states)
key_states = self.k_proj(hidden_states)
value_states = self.v_proj(hidden_states)

RoPE Has Won the Position Embedding Battle

One of the things I want to highlight is that RoPE is actually one of the things that it seems like everyone has converged on. Examining all 19 major model papers, basically all of them now use RoPE.

For various different reasons, RoPE has many different algorithms for extrapolating context length, and that’s an important part of modern production language models. But also, it seems to be empirically quite effective, even at fairly small scales and small context lengths. So it’s kind of won out in this position embedding battle.

Understanding the Feedforward Dimensions

The feedforward dimension represents the output hidden dimension of your MLP, from which you project back onto \(d_{model}\). In general, these feedforward networks are designed as up-projections, meaning you’ll have more hidden units than there were inputs.

But the critical question remains: how much bigger should this dimension be?

$$\text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2$$

💡 Why 4x?

This 4x ratio has emerged as a strong consensus in the community. While there’s empirical evidence supporting this choice, there’s no fundamental law of nature dictating it. It’s simply a convention that has proven remarkably durable and effective across different architectures and applications.

The 8/3 Ratio for GLU

If you scale the GLU variants down by this factor of \(\frac{2}{3}\), you’ll find that the optimal way to achieve this is to set:

$$d_{ff} = \frac{8}{3} d_{model} \approx 2.67 d_{model}$$

You can convince yourself that this parameterization will give you the same number of parameters, and this ratio naturally emerges when you start with the traditional ratio of 4.

🔴 T5’s Extreme Choice

If you look at the 11 billion parameter T5 model, they have a pretty incredible setting:

• Hidden dimension: \(d_{model} = 1024\) (fairly standard)
• Feedforward dimension: \(d_{ff} = 65,536\) (absolutely massive!)

$$\frac{d_{ff}}{d_{model}} = \frac{65,536}{1024} = 64$$

This creates an astounding 64-times multiplier, which is far beyond what most other models attempt!

💡 The TPU Motivation

The T5 team specifically chose to scale up $d_{ff}$ because modern accelerators like the TPUs they trained on are most efficient for large dense matrix multiplications, exactly like those found in the Transformer’s feed-forward networks.

What’s particularly encouraging about T5’s approach is that it demonstrates these extreme ratios are not just theoretically possible but practically viable. The model turned out to be totally fine despite this unconventional choice!

A Wide Basin of Optimality

What emerges is a clear sweet spot pattern, with ratios ranging from 1, 2, 3, 4, and extending up to 10 or so, revealing a pretty wide basin where you can pick whatever feed-forward ratio you want and achieve roughly optimal performance.

The empirical evidence shows there’s a basin between 1-10 where this hyperparameter remains near-optimal, and the standard choice of 4 falls right within this sweet spot—it’s not too far off from the optimal choices.

Practical Guidelines

• If you’re not using GLU activation: multiply by 4 (\(d_{ff} = 4d_{model}\))
• If you’re using GLU: use roughly 2.66 (\(d_{ff} = 2.66d_{model}\))

These default choices have worked well for nearly all modern language models.

The Canonical Choice: Dimension Splitting

The canonical choice is to maintain a fixed total dimension \(D\) (the hidden dimension) while splitting it evenly across multiple heads. As you add more heads, you simply divide the available dimensions among them rather than keeping each head’s dimension constant.

While you could alternatively maintain the same number of dimensions per head as you scale up—letting the attention mechanism consume more parameters—most successful models follow the dimension-splitting approach.

💡 Computational Efficiency of Multi-Head Attention

From a computational perspective, computing \(h\) attention heads isn’t significantly more expensive than single-head attention. The process involves:

1. Computing \(XQ \in \mathbb{R}^{n \times d}\)
2. Reshaping to \(\mathbb{R}^{n \times h \times d/h}\) (same for \(XK\) and \(XV\))
3. Transposing to \(\mathbb{R}^{h \times n \times d/h}\)

After transposing, the head axis effectively becomes a batch axis, making the subsequent operations nearly identical with matrices of the same sizes.

The Practical Reality

If you have very few dimensions per head, that’s going to start affecting the expressiveness of the attention operation. However, in practice, it doesn’t really seem like we see too many significant low-rank bottlenecks, and most models with a ratio of one seem to do just fine.

This parameter has generally been held constant by most of the models we’ve seen, though there have been papers written against the 1-1 ratio (Bhojanapalli et al 2020).

Deep vs Wide: The Trade-Off

We can think about deep networks with more and more layers, or we can have wide networks. Generally, if you want one knob to control the width, that would be the hidden dimension of the residual stream, which controls essentially the width of almost all operations at once.

You might think that deeper networks are smarter and more expressive, while wider networks are more efficient.

🔴 Pipeline vs Tensor Parallelism

For pipeline parallelism, we typically take different layers and distribute them across different devices or blocks of devices, since we can also parallelize within each layer. This approach places certain constraints on our model design.

Alternatively, for really wide models, we can employ tensor parallelism, where we slice up the matrices and distribute them across GPUs.

These different parallelism paradigms come with distinct constraints: Tensor parallelism requires really fast networking, while pipeline parallelism can tolerate slower networking or higher latency connections. Your networking constraints might therefore drive some of these width versus depth considerations.

💡 The Sweet Spot Around 100

Remarkably, around the ratio of 100—which aligns with the consensus choice of hyperparameters—we see the minimum across different scales. This finding is backed by large-scale hyperparameter data published by Kaplan et al. and matches our intuition quite well.

A particularly encouraging result is that the optimal aspect ratio doesn’t shift significantly across several orders of magnitude in model size. If this pattern continues to hold, it’s very good news—you can keep training on one fixed aspect ratio as you scale up.

The Trend Toward Larger Vocabularies

There’s been a clear trend toward larger vocabularies in recent years. This shift is largely driven by the deployment of LLMs as real-world services, where they encounter diverse linguistic environments including multiple languages, emojis, and various other modalities that extend beyond traditional text.

Earlier models, particularly those focused on monolingual tasks, typically operated with vocabulary sizes in the 30,000 to 50,000 token range, as we saw with early GPT models and the original LLaMA.

💡 Vocabulary Scaling Benefits

Research suggests that as models scale up in size, they can effectively utilize increasingly large vocabularies, making good use of more vocabulary elements. This indicates we might see continued growth in token counts as models become larger and are trained on more extensive datasets.

Regarding multilingual vocabularies, they do contribute to improving performance, but the impact varies by language resource level. For high-resource languages like English, you can get away with smaller vocabularies. However, larger vocabularies become really helpful when dealing with minority or low-resource languages.

Why Regularize During Pre-training?

During pre-training, you typically do just one epoch because you have so much data that you can’t even go through all of it. You’re doing one epoch training, and you’re almost certainly not overfitting the data in that single pass.

So you might think: alright, we don’t need regularization for pre-training—let’s just set your optimizer loose and focus entirely on minimizing loss. These are really good arguments for why you shouldn’t need to regularize.

🔴 The Evolution of Regularization

These days, many researchers have stopped publishing the precise details of their training hyperparameters, but dropout has sort of gone out of fashion while weight decay has remained something that a lot of people continue to use.

And why is that? It’s really an odd thing to be doing. If you’re training a really large neural network for one pass on SGD with vast amounts of data, why would you use weight decay when you’re doing that? It’s very intuition-violating, at least for me.

Weight Decay Doesn’t Control Overfitting

If you examine different amounts of weight decay, they don’t really seem to change the ratio of training loss to validation loss. You can train with different amounts of weight decay—if you train for long enough and control your hyperparameters appropriately, you end up with the same train-to-validation loss gap.

So overfitting? Nothing’s happening here, even with zero weight decay.

💡 The Learning Rate Interaction

With weight decay, the model’s not training very well at this high learning rate, but when you decrease the learning rate, it’ll very rapidly drop off. When you examine cosine learning rate decay, models with high weight decay start out very slow, but then as they cool down—that is, their learning rate decreases—they very rapidly optimize.

🔴 The Real Purpose of Weight Decay

You don’t use weight decay because you want to regularize the model, which is what it was originally defined for. You’re using weight decay in order to get actually better training losses.

You end up doing that because of the various learning dynamics at the tail end of training as you decrease your learning rates to zero. It’s a very interesting and complex—and in some ways troubling—thing to be doing with language models.

Consensus Hyperparameters

When it comes to selecting hyperparameters for your model, there are certain choices that have become essentially no-brainers—they’ve been validated and basically everyone uses them:

• Hidden size of the MLP
• Head dimensions of multi-head attention (head_dim × num_heads = $D_{model}$)
• Aspect ratio
• Regularization through weight decay

All of these have fairly good consensus evidence for how to pick them.

Modern Stability Innovations

Looking at recent developments, particularly in multimodal architectures, most academic and open work uses what you might call shallow or late fusion of modalities—essentially bolting the vision component onto an existing language model.

However, multimodal models have pioneered some pretty interesting techniques for stabilizing language model training. When you bolt on a new vision piece and retrain, that’s a big shock to the model, requiring careful consideration of how to stabilize the training process.

These innovations have actually seeped back into pure text language model training, and what’s stood out in recent releases has been an emphasis on stability tricks—methods to train models in much more stable ways as they get bigger and train for longer periods.