Loading Pretrained GPT-2 Weights into GPTModel

Training a language model takes a long time, and nobody wants to repeat hours of work just because a session ended. A saved model turns training into something you can pause and resume. The key idea in this article is that a saved checkpoint is not a finished product by default. It is a snapshot of a training run that can be restored later and pushed further along.

The source describes the resume workflow directly. After the weights are saved, you load the model and the optimizer again, possibly in a completely new Python session or in a Jupyter notebook file, and then continue pretraining for one more epoch using the train_model_simple function. Each step has a purpose, and the two loaded components play different roles.

Read the restoration from left to right. The saved weights have no effect until they are loaded back into a model, and training can only continue once the model and optimizer are both in place. The model supplies the learned values, and the optimizer supplies the machinery that keeps updating those values. If you restore only one of the two, you have not restored the training run; you have restored either a static set of values or a machine with nothing to work on.

A GPTModel is assembled from smaller PyTorch layers. Some of those layers are Linear layers and some are Embedding layers. Each of these layers stores its learned values in a .weight attribute. The weight values therefore do not sit loose on the model object; they live inside the layers that the model contains.

You reach these parameters through model.parameters(). The source notes that this is the same method used earlier in the chapter to inspect the model during training. Because the weights are spread across nested layers, you do not need to know the layer structure to read them all: model.parameters() walks through the model and hands back the weight parameters it contains. That is why it is the natural tool for checking what a restored model holds.

Once the model and optimizer are restored, the final stage is the train_model_simple call you used for the original training, now applied for one additional epoch. An epoch is one full pass over the training data. The source frames this as continuing a single run, so the restored model is not treated as a new model: it resumes from the weights it already holds.

Read the diagram as a single progression. The source describes the call only at the level of running one more epoch, so the diagram marks the stages at that same level. What matters for a learner is the starting state (restored model plus restored optimizer) and the ending state (the same model, with weights updated by the additional pass).

The earlier training in this chapter used a small GPT-2 style model on a limited dataset, a short story book. That kept the focus on fundamentals, but the model had seen very little text. The source points out that OpenAI openly shared the weights of its GPT-2 models. Using those shared weights means you do not have to train a model on a large corpus yourself, which would require a very large investment of money and computing resources.

The plan is to load these shared weights into the GPTModel class you built, and then use that model for text generation. Chapter 6 reuses the same pretrained weights to fine-tune the model for a text classification task and to follow instructions in a way similar to ChatGPT. Loading pretrained weights is therefore the shared starting point for both generation and later fine-tuning.