Saving and Loading LLM Weights
Because weights are learned, their values carry the model's ability forward. Each weight begins with some value and ends training with a learned value, and it is the learned values that the model uses.
Where a Language Model's Behavior Comes From
When a large language model writes a sentence, it is not looking up a stored answer. It computes a next word based on the input it receives and on a large collection of numbers built into the model. Those numbers are the subject of this article, and they are the reason saving and loading matter.
What do you think happens?
If the model's behavior depends on a large set of numbers, who or what decides what those numbers should be?
Reveal answer
The numbers are learned during training.
The source frames these numbers as learned during training rather than typed in or looked up. The rest of the article follows that idea: from its definition, to its role in text generation, to the steps that come after it in building an LLM.
Weights Versus Values You Type In
A weight parameter is a model parameter that is learned during training, not typed in by hand.
The difference lies in where the number comes from and whether it changes. A value a programmer types in is decided before the program runs and stays exactly as written unless someone edits it. A weight starts with some value and is then adjusted by training, so its final value is the result of that process rather than something a person chose directly.
| Feature | Value typed in by hand | Learned weight parameter |
|---|---|---|
| Where it comes from | A programmer writes it in | Learned during training |
| Does its value change? | Only when someone edits it | Yes, it is adjusted as training proceeds |
| Role in the model | Set by the author of the code | Used in the calculation that produces text |
Hand-typed values and learned weight parameters differ in origin and in whether they change.
Why Weights Matter for Text Generation
The source states that weight parameters are used in LLMs for text generation. Text generation is the step where the model produces output, one piece at a time, and the weights are part of the calculation that decides what comes next. This is why weights connect training to the output the model produces: what training changes in the weights shows up later in the words the model generates.
Because weights are learned, their values carry the model's ability forward. Each weight begins with some value and ends training with a learned value, and it is the learned values that the model uses.
Where Saving and Loading Sit in the Chapter
The chapter places weight parameters inside a larger workflow. It begins by recapping text generation, then sets up the LLM for text generation and discusses basic ways to evaluate the quality of the text it produces. Evaluation is where the learned weights are tested, because a model whose weights are good should produce text that is judged well. Evaluation and training both depend on a model whose weights are already defined, which is why weight parameters come first in the chapter's logic.
Read this sequence as a map rather than a checklist. The first stages prepare you to judge a model. The middle stages train and improve it. The later stages handle the weights as stored, reusable values. The source also describes loading pretrained weights from OpenAI as a different route to having learned weights: instead of learning them in this chapter's own training run, the model receives weights that were already learned elsewhere.
Saving Weights: Capturing the Current State
Saving model weights stores the learned values of a model so that they can be used later.
Saving is the act of writing the current weights to storage. The source describes implementing functions to save and load LLM weights, which allows the model to be used later or to have its training continued. Saving therefore captures the current state of the weights at the moment it is done, including whatever training has already adjusted them.
Tracing One Weight Across a Save and a Load
To make the idea concrete, follow a single weight from before training to after training. The numbers below are invented for illustration. The source does not give specific values for any real model, so what matters is the direction of change: the weight keeps its identity and role, and only its value moves.
Following one weight through a save and a load
A training run stops with a weight w at 0.50. The weights are saved, and a later session loads them and continues training. What value does w have at each point?
- 1Just before saving
The weight holds the value training has produced so far, 0.50 in this illustration.
- 2Saving
The current weights are written to storage. The saved copy records w at 0.50.
- 3Loading
The saved weights are read back into the model. The model now starts from that stored state, so w is 0.50 again rather than a fresh starting value.
- 4Continuing training
Training adjusts w further. In this illustration it moves to 0.37, and it is still the same weight with the same role.
The load restores the value 0.50 that was saved, and training then carries on from there to 0.37. The weight did not restart.
Loading Weights and Continuing Training
Loading model weights reads previously saved weights back in, so the model starts from that saved state.
Saving and loading are best understood as a pair, and the source gives them the same purpose: continuing training an LLM rather than starting over. A training run does not always finish in one sitting. The work done so far can be stored and picked up again later instead of treating each session as a fresh start.
Mistakes That Make a Resumed Run Start Over
A resumed training run is supposed to pick up the saved weights. If the save-and-load path is skipped or misordered, the model can begin from a starting state instead of the learned one, and training then behaves as if it started over.
Common mistake Saving the weights but never loading them
What it looks likeTraining ends with the weights written to storage, and the next session builds a new model and begins training immediately.
Why it failsA save stores progress but does not put it back to work. Without a load, the new model never receives the stored state.
Do this insteadAfter building the model in a new session, load the saved weights into it before training continues.
Common mistake Attempting a load when no save exists
What it looks likeA new session tries to restore weights from a location where nothing was ever saved.
Why it failsA load is only meaningful when there is a saved set of weights to restore.
Do this insteadConfirm that a save completed and that the load reads from the same stored set before resuming.
Common mistake Treating a fresh model as if it already holds learned values
What it looks likeA model is created and training is resumed without restoring anything, so its weights start from whatever the new model begins with.
Why it failsLearned weights exist only as the saved state. A new model does not automatically carry them.
Do this insteadRestore the saved state into the model before continuing, so the weights begin from the learned values.
Checking Your Understanding of a Resumed Run
A training session ends with a weight w at 0.50 and the weights are saved. A new session creates a model but skips the load, then trains for a while. Explain why the resumed run is not a continuation of the earlier training, and say what step would have fixed it.
Hints
- Ask what a load is responsible for restoring.
- Compare what the saved copy holds with what the new model holds before the load.
Key Takeaways
- A weight parameter is learned during training, not typed in by hand, and it is used in the calculation that produces text.
- Saving model weights stores their learned values so they can be used later or have training continued.
- Loading reads saved weights back into a model so it starts from that stored state rather than from scratch.
- Saving and loading work as a pair; a save without a later load does not put the progress back to work, and a load needs a saved set to restore.
- Saving and loading sit later in the chapter's sequence, after text generation, evaluation, and training, and alongside loading pretrained OpenAI weights.