Evaluating Generative Text Models

Pretraining is stage 2 of coding an LLM and includes the training loop, model evaluation, and saving and loading model weights.

Programming

Why Look Inside the Weights

Building a large language model is a multi-stage process, and this part of the course focuses on the middle stage, pretraining. The values a model stores and adjusts are called weight parameters, and they sit at the center of the work done in this stage. Before you can judge whether a model is built correctly, measure its size, or later save and load its values, you need to be able to find those values and read them.

This article takes a hands-on approach. You will see where weight parameters fit in the pretraining stage, create a single PyTorch linear layer, read the shape of its weight tensor, and then count every value the layer holds. Each step is one that you will repeat when working with larger models.

Where Weights Sit in Pretraining

The source material describes three main stages for coding an LLM. Stage 1 covers building the model: preparing and sampling data, implementing the attention mechanism, and assembling the LLM architecture. Stage 2 is pretraining, which includes the training loop, model evaluation, and saving and loading model weights. Stage 3 covers fine-tuning on labeled or instruction data. The diagram below lays these stages out in order.

Which stages come before and after pretraining, and where do weights get read and written?

Notice where weights appear. Loading pretrained weights is a step inside the pretraining stage, and saving and loading weights is part of what the chapter covers. Weights are therefore not an afterthought. They are the product of training and the input to later stages, which is why reading them is a practical skill you will use repeatedly.

Within pretraining, the source names three activities: implementing the training code, evaluating the model's performance, and saving and loading model weights. Evaluation is one of these pretraining activities. It checks how the model is performing, and it sits alongside the training loop rather than replacing it.

Definition

A weight parameter is a learnable value that belongs to a model. During training, these values are adjusted so the model's behavior improves. Each layer in a PyTorch model can hold one or more parameters, and each parameter is stored as a tensor.

Reading a Linear Layer's Weight Shape

The simplest way to see weight parameters is to create a single linear layer and inspect what it holds. A linear layer that maps 3 input values to 2 output values stores a weight tensor and a bias tensor. In PyTorch, the weight tensor of a linear layer has shape (out_features, in_features). The first number in the shape is the output size, and the second is the input size.

Which dimension of the shape tuple is the output size and which is the input size, and how do the rows and columns of the weight matrix map onto them?

Read the diagram this way: the first shape dimension counts the rows of the weight matrix, one row for each output value, and the second dimension counts the columns, one column for each input value. A 3-to-2 layer therefore has a weight matrix with 2 rows and 3 columns. The bias tensor is a separate vector with one entry per output value, so for this layer its shape is (2,).

python

import torch
import torch.nn as nn

layer = nn.Linear(3, 2)

print(layer.weight.shape)
print(layer.bias.shape)

for name, param in layer.named_parameters():
    print(name, param.shape)

import torch
import torch.nn as nn

layer = nn.Linear(3, 2)

print(layer.weight.shape)
print(layer.bias.shape)

for name, param in layer.named_parameters():
    print(name, param.shape)

Output (expected)

torch.Size([2, 3])
torch.Size([2])
weight torch.Size([2, 3])
bias torch.Size([2])

Notes on the code

nn.Linear(3, 2) maps 3 input values to 2 output values
layer.weight.shape shows the (out_features, in_features) layout
layer.bias.shape shows one entry per output value
named_parameters() lists each parameter by name along with its tensor

The values inside the weight tensor are chosen when the layer is created and are then adjusted during training. The shape, however, is fixed by the layer's sizes. This is why you can predict the shape from the layer definition before you ever run training.

Worked example

Predicting a weight shape before printing it

A linear layer is defined as nn.Linear(5, 4). What shape will its weight tensor have?

1
Identify the input size
The first argument is the number of input values, so in_features = 5.
2
Identify the output size
The second argument is the number of output values, so out_features = 4.
3
Apply the weight layout
The weight shape is (out_features, in_features), so it is (4, 5).

Result

The weight tensor has shape (4, 5): 4 rows, one per output value, and 5 columns, one per input value.

Counting Every Value in a Model

A shape tells you how a tensor is arranged, but not how many values it holds. The number of values is found with numel(), which counts all the elements in a tensor. Summing numel() across all parameters gives the total number of values the model holds. The parameters() and named_parameters() methods are the way to reach those parameters in the first place.

How do the weight and bias tensors add up to the total parameter count, and what contains what?

Working through the hierarchy from the bottom up gives the total. The weight tensor holds $2 \times 3 = 6$ values, the bias tensor holds 2 values, and the layer holds $6 + 2 = 8$ values in all. Summing numel() across every parameter of the model gives the same kind of total for a model of any size.

python

import torch.nn as nn

layer = nn.Linear(3, 2)

total = sum(p.numel() for p in layer.parameters())
print(total)

import torch.nn as nn

layer = nn.Linear(3, 2)

total = sum(p.numel() for p in layer.parameters())
print(total)

Output (expected)

Notes on the code

parameters() yields each parameter tensor of the layer
numel() counts the elements in one tensor
sum() adds the element counts across all parameters: 6 + 2 = 8

What do you think happens?

Given nn.Linear(3, 2), the weight has 6 values. Does the bias add to the total returned by summing numel() across parameters?

No, the total is only 6 because the bias is not a weight
Yes, the total is 8 because the bias is also a parameter tensor

Reveal answer

Answer:

Yes, the total is 8 because the bias is also a parameter tensor

parameters() returns every parameter tensor in the layer, including the bias. Summing numel() across them gives 6 for the weight plus 2 for the bias, a total of 8.

Mistakes When Reading Weight Shapes

Learners who are new to reading parameters tend to make a few predictable errors. The table and the list below cover the most common ones, with the correct reading for each.

Quantity	What it is	For nn.Linear(3, 2)
Shape	The size along each dimension of a tensor	(2, 3)
Element count	The number of values in the tensor, found with numel()	$2 \times 3 = 6$
Bias shape	The size of the bias vector, one entry per output	(2,)
Total parameters	Element counts of all parameter tensors added together	$6 + 2 = 8$

A shape, an element count, and a total parameter count answer different questions

Common mistake
Confusing the shape with the number of values
What it looks like
Reading (2, 3) and concluding the layer holds 2 values or 3 values.
Why it fails
The shape is a tuple of sizes per dimension. The number of values is the product of those sizes.
Do this instead
Multiply the dimensions: 2 times 3 gives 6 weight values.
Common mistake
Reversing the dimensions of the weight tensor
What it looks like
Reading the (2, 3) weight of nn.Linear(3, 2) as having 3 output values and 2 input values.
Why it fails
In PyTorch the linear weight is laid out as (out_features, in_features), so the first dimension is the output size.
Do this instead
Map the first dimension to outputs and the second to inputs, then check against the constructor arguments.
Common mistake
Leaving out the bias when counting parameters
What it looks like
```
Reporting 6 total parameters for nn.Linear(3, 2).
```
Why it fails
The bias is a separate parameter tensor, and parameters() returns it along with the weight.
Do this instead
Sum numel() over all parameters to get 6 + 2 = 8.

Practice: Reading a Different Layer

EASY

Without running code, predict the weight shape, the bias shape, and the total number of parameters for nn.Linear(4, 5). Then check your answer by printing the shapes and summing numel() across layer.parameters().

Hints

The first constructor argument is in_features, and the second is out_features.
The weight shape is (out_features, in_features), which puts the output size first.
The bias has one entry per output value, and the total is the weight element count plus the bias element count.

Your attempt (not saved)

Key Takeaways

Pretraining is stage 2 of coding an LLM, and it includes the training loop, model evaluation, and saving and loading model weights.
A weight parameter is a learnable value stored as a tensor, and it is adjusted during training.
In PyTorch, a linear layer's weight tensor has shape (out_features, in_features), so the first dimension is the output size and the second is the input size.
Use parameters() or named_parameters() to reach a model's parameters, and sum numel() across them to count every value the model holds.

Related concepts

More concepts starting with E