Saving and Loading Model Weights in PyTorch

Ask a trained language model the same question several times and you may get several different replies. That can look like a bug, but it is often a direct result of how the model chooses its words. This article covers two related ideas from the same chapter. The first is how generation settings, temperature and top-k, control how much variety appears in the text. The second is how the weights of a model are saved to storage and loaded back later so that training can continue instead of starting over. Both ideas come down to controlling what survives between runs and how the next step is chosen.

A trained language model does not simply hand back one fixed reply. At each step it weighs many possible next tokens, and how it picks among them determines what text comes out. If the choice is always the single most likely token, the output is predictable. If the choice is drawn at random from the candidates, the same prompt can produce a different result each time.

Two settings give you a practical handle on this behavior: temperature and top-k. Both are used to control the randomness of text generation. It helps to treat them as dials you can turn and observe, rather than as fixed rules. Both act on the choice of the next token, but they limit or reshape that choice in different ways.

The two dials are easiest to understand side by side. Each one changes the candidate pool or the weighting of candidates for the next token, and each one tends to change the variety of the output in its own way. The comparison below summarizes the difference, and it is worth keeping in mind as you read the rest of the article.

The diagram below places the steps of a single token choice in order. Read it left to right. Top-k first narrows the set of candidate tokens. Temperature then reshapes how the remaining candidates are weighted. Finally one token is chosen, either as the most likely candidate or by drawing at random. This is a conceptual picture of how the two dials relate to one choice, so focus on how each step passes its result forward.

Because the two dials act on the same choice in different ways, setting both low and setting both high can produce noticeably different styles of text. A low setting on both narrows the candidates and makes the output steadier. A high setting on both widens the candidates and makes the output more varied. Mixed settings produce styles between these extremes.

The right choice depends on what you want the text to do. A steady answer, where you want the same kind of reply each time, is usually better served by lower settings. A varied brainstorm, where you want many different ideas from the same prompt, is usually better served by higher settings. Notice that these are guidelines to test, not fixed rules.

Sometimes you want the opposite of variety. The same input should always produce the same output. This is called deterministic behavior: the output is always the same given the same input. Lower temperature and top-k settings make output more consistent, but consistency is not the same as determinism. Determinism is achieved by changing how the generate function selects tokens.

The key idea is disabling random sampling. When random sampling is turned off, the generate function always selects the same tokens for the same input and produces the same outputs. The source describes this as similar to the generate_simple function used earlier in the chapter, which also selects tokens without drawing them at random.

Saving model weights means storing the learned values of a model so that they can be used later. Training a large language model is a long process, and a training run does not always finish in one sitting. The weights hold the progress made so far, so writing them to storage lets that progress survive after the session ends.

The reason the chapter pauses to save weights comes down to cost. Pretraining an LLM is computationally expensive, and even with a relatively small model and dataset, the chapter's exercise made this clear. Saving the model means you do not have to rerun training every time you want to use it in a new session, and the same saved weights can be used to continue training later.

The chapter also looks ahead. It will later load a more capable pretrained GPT model from OpenAI into the GPTModel instance. The save and load functions built here are the bridge between the model trained in this chapter and that later model.

Follow one set of weights through a full cycle. Start with a model whose weights have been trained for some time. Saving writes the current weights to storage, so a copy of that learned state now exists in a file. The model in memory and the file are two separate places holding the same learned values at the moment of the save.

Later, in a new session, the saved weights are read back in. Loading is the reverse of saving: the stored weights are restored into the model, so it starts from that saved state rather than from scratch. Training can now carry on from where it left off. The diagram traces this movement, with the changed states marking what the save and load produce.

Saving and loading are used together to continue training an LLM rather than starting over. Because they are a pair, each one depends on the other to be useful. A save without a later load stores the progress but does not put it back to work. A load is only meaningful when there is a saved set of weights to restore.