Training and Validation Losses
A weight parameter is a model parameter that is learned during training, not typed in by hand.
Where a Language Model's Behavior Comes From
When a large language model writes a sentence, it is not looking up a stored answer. It computes the next word from the input it receives and from a large collection of numbers built into the model. Those numbers are the subject of this article, and the script you will study measures how well a model is doing on two kinds of data: the training data it learns from and the validation data it is checked against.
What do you think happens?
If the model's behavior depends on a large set of numbers, who or what decides what those numbers should be?
Reveal answer
The numbers are learned during training. They are not typed in by a programmer.
The rest of this article follows that idea: first what a weight parameter is, then how it is used in text generation, and then the steps that come after it in building an LLM.
Weights Are Learned, Not Typed In
A weight parameter is a model parameter that is learned during training, not typed in by hand.
A model contains a very large number of weights, and each one is treated the same way. A weight begins with some starting value and ends training with a learned value. The starting value is not the final answer. Training moves the value, and the learned value is what carries the model's ability forward.
Trace the diagram in order. The weight is still the same weight, with the same role in the model. Only its value has moved. The source does not give specific numbers for any real model, so the important idea is the direction of change: from a value that was present before training to a value that training produced.
Why Weights Matter for Text Generation
The reason weights matter in this chapter is that they are used in text generation. Text generation is the step where the model produces output one piece at a time, and the weights are part of the calculation that decides what comes next. Learned weights therefore connect training to the output the model actually produces.
Learned weights are also valuable because training takes effort. The chapter describes saving and loading LLM weights. Saving captures the current state of the weights. Loading brings a saved state back into a model so it can be used again or trained further. A second route to learned weights is loading pretrained weights from OpenAI into the LLM model, which means the weights were learned elsewhere rather than in this chapter's own training run.
Choosing a Device at Run Time
A device is the hardware that performs the computation. The two options described are the CPU, the general processor, and the GPU, the graphics processor supported by CUDA. The first line of the script selects a device using a single expression: if a CUDA-supported GPU is available, the device is set to CUDA; otherwise it falls back to the CPU.
torch.cuda.is_available()checks whether a CUDA-supported GPU is present- The conditional expression picks cuda when the check is true and cpu otherwise
model.to(device)moves the model's parameters onto the selected hardware
The selection happens at run time. If the machine has a CUDA-supported GPU, the LLM trains on the GPU with no changes to the code. The same script therefore works on a laptop with no GPU and on a workstation with one, and you do not need separate versions of the program.
The source makes the matching requirement explicit: the device setting is what ensures the data is loaded onto the same device as the model. For this reason the device value is passed explicitly into calc_loss_loader as well. A model on the GPU with data still on the CPU would not be working on the same hardware.
Turning Off Gradient Tracking
Gradient tracking is the bookkeeping PyTorch normally does so that a model can be updated during training. The with torch.no_grad() block turns that bookkeeping off. The source explains that this is done for efficiency because the script is measuring loss rather than training the model.
- The with block wraps both loss calculations
- Every line inside the block runs with gradient tracking turned off
- Only the loader changes between the two calls
| Question | Training | Measuring loss in this script |
|---|---|---|
| Is the model being updated? | Yes | No, the model is not being trained yet |
| Is gradient bookkeeping needed? | Yes, so the model can be updated | No, that bookkeeping is not needed |
| Effect of no_grad() | Not used | Gradient tracking is off, which saves work |
Gradient tracking during training versus loss measurement
Tracing calc_loss_loader Through Both Loaders
The two loss lines are nearly identical. Each one calls calc_loss_loader with a data loader, the model, and the device. Only the loader changes: train_loader for training loss and val_loader for validation loss. Because the same function is used under the same conditions, the two numbers are directly comparable. The source does not describe the internals of calc_loss_loader, so this article traces its inputs and outputs only.
Tracing the two loss lines
Starting from the with block, follow exactly what the script does to produce train_loss and val_loss.
- 1Enter gradient-off mode
The with
torch.no_grad()block starts, so every line inside runs with gradient tracking turned off. - 2First call
calc_loss_loaderreceivestrain_loader, the model, and the device. Its result is stored astrain_loss. - 3Second call
calc_loss_loaderreceivesval_loader, the same model, and the same device. Its result is stored asval_loss. - 4Compare
Both numbers came from the same function under identical conditions, so a difference between them reflects the data, not a different calculation.
The script produces two loss values, train_loss and val_loss, that can be compared directly.
Where the Loss Script Can Go Wrong
Common mistake Moving the model to the device but not the data
What it looks likemodel.to(device) runs with device set to cuda, but the batches from the loader are still on the CPU when they reach calc_loss_loader.
Why it failsThe source states that the data must be loaded onto the same device as the model. A model on the GPU and data on the CPU are not on the same hardware.
Do this insteadPass the same device value into every loss call so the data is loaded onto that device.
Common mistake Assuming the code must change for a machine with a GPU
What it looks likeWriting one version of the script for a laptop and a second version for a workstation with a GPU.
Why it failsThe device line selects CUDA when it is available and falls back to the CPU otherwise, so one script covers both.
Do this insteadKeep the single device line and let the selection happen at run time.
Common mistake Treating a weight as a constant you can type in
What it looks likeExpecting a weight to keep the value you assigned to it before training.
Why it failsWeight parameters are learned during training, so their values change as training proceeds.
Do this insteadThink of a weight as a parameter whose value is the result of training, and treat saving and loading as the way that learned value is kept.
Common mistake Using different conditions for the training and validation loss
What it looks likeCalling one loss function with the device and the other without it, or with a different model state.
Why it failsThe two numbers are only directly comparable when both are computed under identical conditions.
Do this insteadUse the same calc_loss_loader function, the same model, and the same device for both calls, changing only the loader.
Checking the Device Path Yourself
A machine has no CUDA-supported GPU. Using the device line from this article, which device is selected, and does the loss script need any change to run on this machine? Then describe what must happen to the validation data so that it matches the model's device.
Hints
- Look at the conditional expression: it checks
torch.cuda.is_available()first - The fallback option is named in the same expression
- The data must be loaded onto the same device that the model was moved to
Key Takeaways
- A weight parameter is a model parameter learned during training, so its value moves from a starting value to a learned value rather than being typed in by hand.
- Learned weights are used in text generation, and they can be saved, loaded, or replaced by pretrained weights from OpenAI.
- The device line selects CUDA when a GPU is available and otherwise falls back to the CPU, and the model and data must be on that same device.
- The with
torch.no_grad()block turns off gradient tracking because the script is measuring loss, not training the model. calc_loss_loaderis called once with the training loader and once with the validation loader under identical conditions, producingtrain_lossandval_lossthat can be compared directly.