Saving and Loading Model Weights

Training a large language model is a long process, and a training run does not always finish in one sitting. Instead of treating each session as a fresh start, the work already done can be stored and picked up again later. That is the job of saving and loading model weights. This article follows the weights through that process and asks a simple question: when you stop training and come back later, what has to survive the break, and what has to be restored before training can carry on from where it left off?

Answering that question depends on being able to read a training run in the first place. Once an LLM has been trained, the natural first question is whether the training worked. One way to answer it is to look at the loss values recorded during training. The sections that follow start with those raw numbers, predict where the curves go, and then explain why they behave the way they do.

The loss is a single number that summarizes how far the model's predictions are from the text it was supposed to produce. A lower number generally means the model is doing better on the data it was measured against. The trouble is that a model can look excellent on the data it practiced on while being poor at everything else. Reading two loss numbers side by side, instead of one, is how you tell these situations apart. The training loss tells you how well the model fits the material it sees. The validation loss tells you whether that fit carries over to material it has not practiced on.

Notice how the model behaved at the start of that run. The source describes it as only able to append commas to the starting context or repeat a word, which is consistent with a very high loss. By the end, the same model produces grammatically correct text. The falling training loss tracks that improvement in language skill.

Early in the run, both losses fall sharply together. That shared drop is a healthy sign: the model is picking up patterns that help on the training text and on the held-out text alike. The picture changes after the second epoch. The training loss keeps going down, while the validation loss stagnates. The divergence is the signal to read carefully. The validation loss stays much larger than the training loss, and that gap is evidence that the model is fitting something specific to the training data rather than a general pattern.

The source does not leave the diagnosis to the loss curves alone. It checks memorization directly by searching the generated output for snippets that appear verbatim in the training text file, such as a passage about the irony in a story. Finding those exact passages in the generated text confirms that the model is reproducing what it saw rather than producing new language.

The source explains why memorization occurs in this particular setup. The training dataset is very small and the model is trained for many epochs, which makes memorization expected. In practice, training is usually done on a much larger dataset for only one epoch, which leaves less room for verbatim recall. The source also notes that the overfitting does not occur when the model is trained on a much larger public domain corpus of 60,000 books, and it points readers to an appendix for details.

Saving exists because training is not always a single continuous session. The purpose the source gives for saving and loading is the same: they are used to continue training an LLM rather than starting over. The work done so far is stored so that it can be picked up later. Saving on its own only stores that progress; it does not put the progress back to work.

Loading is the reverse of saving. A load is only meaningful when there is a saved set of weights to restore. Saving and loading are therefore best understood as a pair: a save without a later load stores progress without putting it back to work, and a load without a prior save has nothing to restore. Together they let training continue from where it left off.