Saving PyTorch Models

torch.load("model.pth") reads the saved file and reconstructs the Python dictionary containing the model's parameters.

Programming

Why Saved Parameters Must Be Restored

A trained model is only useful if its learned state can be brought back later. Once a model has been trained and its parameters saved to a file, a new program needs a way to restore those parameters into a model. PyTorch handles this with two pieces working together: a function that reads the file, and a method that applies what was read to a model.

What do you think happens?

Before reading on: the loading line calls load_state_dict on a model. Which object do you think must exist first, the model or the dictionary read from the file?

The model must exist first, because load_state_dict is a method called on it
The dictionary must exist first, because the model is built from it

Reveal answer

Answer:

The model must exist first. load_state_dict() is a method called on the model object, and the dictionary has to exist before it can be passed in.

The source pattern creates the model, reads the file, and then applies the parameters. Each stage depends on the one before it, so the order cannot be reversed.

Tracing the Three-Stage Load Pattern

The source presents the whole job as a short sequence: create a model, then load the saved parameters into it. The NeuralNetwork class is assumed to already exist, so this section focuses only on the loading step. Read the code once and notice which object is created first and which object is handed to load_state_dict().

python

model = NeuralNetwork(2, 2)
model.load_state_dict(torch.load("model.pth"))

model = NeuralNetwork(2, 2)
model.load_state_dict(torch.load("model.pth"))

Notes on the code

NeuralNetwork(2, 2) creates the model object first; its parameters start in their initial state
torch.load("model.pth") runs first inside the second line, reading the saved file
load_state_dict() is then called on the model object and receives the dictionary torch.load() returned

Reading the second line in the order Python evaluates it is the key to the whole pattern. The part inside the parentheses of load_state_dict() is computed first. Only after torch.load() returns its dictionary does load_state_dict() run, and it runs on the model that already exists on the left.

In what order do the model creation, the file read, and the parameter update happen, and what does each stage hand to the next?

What torch.load() Hands Back

The important word in the source's description is reconstructs. The saved file holds the parameters in stored form, and torch.load() rebuilds them as a Python dictionary your program can work with. This is a loading step only. Reading the file does not change the model by itself, because the dictionary still has to be applied to a model object.

Because the result is an ordinary dictionary, this stage has a clear output that you can check. If the call completes, you have a dictionary of parameters ready to hand to the model. Keeping this output in mind makes it easier to find errors later.

How do the dictionary keys, which are parameter names such as a layer's weights and biases, correspond to the parameters inside the NeuralNetwork instance?

Each key names one parameter, and each value holds the data for that parameter. The keys are what allow load_state_dict() to match stored values to the right places in the model. The dictionary is therefore a mapping from names to learned values, not the model itself.

What load_state_dict() Does to the Model

load_state_dict() is a method called on the model object. Its input is the dictionary that torch.load() produced. The method takes the parameters from that dictionary and applies them to the model, so the model returns to the learned state it had when it was saved.

The two steps are connected but separate. torch.load() handles the file and produces the dictionary. load_state_dict() handles the model and consumes the dictionary. Before the call, the model holds the parameters it was created with; after the call, the same model object holds the parameters that were read from the file. The object stays the same while its values change.

Finding Where a Load Goes Wrong

When a load fails to produce the expected model, the source advises checking two things at each stage: the output of torch.load(), and the identity of the model object. Work through the stages in order and ask at each one what it returned and whether it is the object you intended.

At which stage does a loading error occur, and what does each stage's output look like when it is correct versus wrong?

Worked example

Locating a failed load by checking each stage

A program runs model = NeuralNetwork(2, 2) and then model.load_state_dict(torch.load("model.pth")). The model's behavior does not match the saved model. Which stage should you check first, and what should you look for?

1
Check the torch.load() stage
Run torch.load("model.pth") on its own and inspect what it returns. A correct result is an ordinary Python dictionary of parameters. If the output is not that dictionary, the problem lies in the file read, before the model is involved.
2
Check the model object
Confirm that the object on which load_state_dict() is called is the model you meant to restore. If the call was made on a different instance than the one you later use, the parameters were applied to the wrong object.
3
Confirm the stage order
Make sure the model was created before the call and that the dictionary was produced before it was passed in. The stage order matters, so a mismatch in ordering points back to how the lines were written.

Result

The first stage to check is torch.load(), by inspecting its returned dictionary. The second is the identity of the model object that load_state_dict() is called on. Checking these two outputs in order tells you whether the fault is in the file read or in the model the dictionary was applied to.

Mistakes When Restoring Saved Parameters

Common mistake
Assuming torch.load() changes the model
Why it fails
Reading the file produces a dictionary. The dictionary still has to be applied to a model object, so the model is unchanged until load_state_dict() runs.
Do this instead
Treat torch.load() as a loading step only, and call load_state_dict() on the model to restore its parameters.
Common mistake
Calling load_state_dict() before the model exists
What it looks like
```
model.load_state_dict(torch.load("model.pth")) placed above model = NeuralNetwork(2, 2)
```
Why it fails
load_state_dict() is a method called on the model object, so the model must exist before the call can be made.
Do this instead
Create the model first, then read the file and apply the dictionary.
Common mistake
Applying the parameters to a different model object than the one you use
Why it fails
The restored values live in the object load_state_dict() was called on. A second instance created elsewhere does not receive them.
Do this instead
Check the identity of the model object at the point where the load happens, and use that same object afterwards.
Common mistake
Skipping the check of torch.load()'s output when a load fails

Practice: Checking Each Stage

MEDIUM

A colleague writes the following three lines and reports that the restored model behaves like a fresh, untrained one. Without running the code, trace the stages: model = NeuralNetwork(2, 2); weights = torch.load("model.pth"); new_model = NeuralNetwork(2, 2). Which stage produced the dictionary, which object was it applied to, and where did the restored state go?

Hints

Ask which line calls load_state_dict() at all. If none of the lines calls it, the dictionary is never applied to any model.
Compare the object you would check for the restored state with the object that was created in the first line.

Your attempt (not saved)

EASY

Rewrite the three lines so the dictionary from torch.load() is applied to the model that is actually used afterwards. Then state, for each stage, what it returns and which object it works on.