Model Training in PyTorch

After a model has been trained, its learned parameters are the result of all that work. If those parameters exist only in the running Python session, they are lost when the session ends. Saving them to a file lets you keep that result and reuse it later. This article examines one line of PyTorch code that does this job and works through exactly what happens at each step.

Keep your prediction in mind. The rest of this article checks it step by step, so you can see where each piece of the answer comes from.

The line contains two distinct pieces. The call torch.save is the function that writes data to a file. Its first argument, model.state_dict(), is the data to be written. Its second argument, the string "model.pth", is the name of the file. It helps to name these three roles separately: the data, the function that writes it, and the destination. Most of the mistakes covered later come from confusing one role with another.

Because the snapshot is a plain dictionary of the model's parameters, it stores the learned values rather than the model object that computes with them. Each entry pairs a name with a tensor. In a small model made of two layers, the names follow the position and role of each layer, so a key might read like a layer index followed by a part name, and its value is the tensor holding those numbers. The diagram below shows this mapping as a set of keys pointing at their tensors.

The data moves in a fixed order. The model holds its parameters. The call model.state_dict() reads those parameters and returns them as a dictionary. That dictionary is passed as the first argument to torch.save(), which writes it to the file named by the second argument. The model itself is never the thing being written. Only the dictionary produced by state_dict() is written.

The three objects in this process are easy to blur together. The model is the live object that computes. The state dictionary is a snapshot of the model's state, returned as a dictionary. The file is what torch.save leaves on disk. Keeping these three separate explains what is saved and what is left out.

Because the file holds the dictionary and not the model object, the saved content is the learned values organized by name. The model's computing behavior is not what gets written by this line.

When a save does not produce the result you expect, locate the failure by checking the three roles in order. First check the data: is the first argument a state dictionary produced by model.state_dict()? Second check the function: is the call to torch.save()? Third check the destination: is the second argument the file name you intended? Most failures sit in one of these three places. The first check is the one most likely to hide a divergence, because the code can look correct while passing the wrong value.