Weight Decay and Regularization
AdamW is a variant of Adam that improves the weight decay approach.
Why Training Shapes Weights
Training a neural network means repeatedly adjusting its weights so that its predictions improve. The rule that decides how those adjustments are made is called the optimizer, and the choice of optimizer shapes how a model learns. This article follows one specific choice: a large language model training loop that uses AdamW instead of the more familiar Adam.
Keep one question in mind as you read: what does it mean for a training method to discourage large weights, and why would a model builder want that? The answer is the idea of weight decay, which sits at the center of the AdamW story.
Adam and the Role of AdamW
Adam optimizers are a popular choice for training deep neural networks. AdamW builds directly on Adam, so the easiest way to understand it is as a modified version of a method you may already know. The source describes AdamW as a variant of Adam, meaning it keeps the overall character of Adam but changes one part of how training is regularized.
AdamW is a variant of Adam that improves the way weight decay is applied. It keeps the Adam approach to training and changes only the weight decay handling.
| Aspect | Adam | AdamW |
|---|---|---|
| Overall approach to training | Popular method for deep neural networks | Keeps the Adam approach to training |
| Relationship | The base method | A variant of Adam |
| Weight decay handling | The familiar version | Improved so that weight decay is applied more effectively |
| Stated benefit | Not the focus of the source's discussion | More effective regularization and better generalization, which is why it is frequently used for training LLMs |
AdamW changes how weight decay is applied; the rest of the Adam approach is kept.
Why Large Weights Cause Trouble
Weight decay is the technique at the center of the AdamW story, so it is worth understanding on its own first. A model with very large weights can fit its training data in a fragile way. Its complexity grows, and it becomes more prone to overfitting, which means it memorizes details of the training examples instead of learning patterns that carry over to new data.
The diagram is a conceptual summary of the source's claims rather than a measured experiment. The key idea is directional: weight decay pushes against large weights, so the model is pulled toward a simpler fit.
Tracing a 10-Epoch Training Run
The source puts these ideas into practice by training a GPTModel instance for 10 epochs. An epoch is one complete pass through the training data. The run uses an AdamW optimizer together with the train_model_simple function, which was defined earlier in the material.
Training a GPTModel for 10 epochs with AdamW
Train a GPTModel instance for 10 epochs using an AdamW optimizer and the train_model_simple function.
- 1Create the model
Start with a GPTModel instance whose weights will be adjusted during training.
- 2Choose the optimizer
Use an AdamW optimizer, so the weight updates follow the Adam approach with the improved weight decay handling.
- 3Call the training function
Execute
train_model_simple, the function defined earlier in the material, to start the training process. - 4Run 10 epochs
Each epoch is one complete pass through the training data, so the training data is passed through 10 times.
- 5Watch the progress
Text printed during the run shows the progress of training.
- 6Finish
According to the source, the run takes about five minutes on a MacBook Air or a similar laptop.
The GPTModel has been trained for 10 epochs with AdamW and train_model_simple, with printed progress along the way and a total run time of about five minutes on a laptop of that class.
Mistakes Beginners Make with Weight Decay
Common mistake Treating AdamW as an unrelated optimizer
Why it failsThe source describes AdamW as a variant of Adam that keeps the overall character of Adam.
Do this insteadThink of AdamW as Adam with an improved application of weight decay.
Common mistake Believing weight decay alone guarantees generalization
Why it failsThe source says weight decay helps a model generalize by penalizing larger weights, not that it guarantees good performance on unseen data.
Do this insteadTreat weight decay as one form of regularization that supports generalization.
Common mistake Confusing an epoch with a single weight update
Why it failsAn epoch is one complete pass through the training data, which spans many updates.
Do this insteadCount epochs as full passes over the training data.
Common mistake Assuming the source explains the AdamW update arithmetic
Why it failsThe source identifies the relationship and the benefit but does not give the internal update equations.
Do this insteadLook to a dedicated optimizer reference for the exact update rule.
Check Your Understanding
A training run for a GPTModel uses AdamW and train_model_simple for 10 epochs. Explain in your own words what one epoch is, and describe how weight decay is meant to affect the size of the weights and the model's ability to generalize.
Hints
- Recall how the source defines an epoch.
- Think about what weight decay penalizes and what it favors.
- Consider how generalization is described in the source.
Key Takeaways
- AdamW is a variant of Adam that keeps the Adam approach to training and improves the way weight decay is applied.
- Weight decay aims to minimize model complexity and prevent overfitting by penalizing larger weights.
- Regularization is the general counterweight idea, and weight decay is one form of it that helps a model generalize to unseen data.
- AdamW is frequently used when training large language models because it supports more effective regularization and better generalization.
- In the source's example, training a GPTModel for 10 epochs with AdamW and
train_model_simpletakes about five minutes on a MacBook Air or similar laptop.