Pretraining a GPT Model on Unlabeled Data
Loading a pretrained GPT model uses the same code as pretraining and fine-tuning, so the procedure you already know still applies.
Expecting a New Procedure
By this point in the course you have written code that pretrains a GPT-style model on text and code, and code that fine-tunes a model for a classification task. When it is time to work with a model that someone else has already trained, you might expect a whole new procedure to appear. It does not. Loading a pretrained GPT model reuses the same code you already wrote, and only one part of that code needs to change.
What do you think happens?
Before reading on: when you switch from loading one pretrained GPT model to a different one, how much of your existing code do you expect to rewrite?
Reveal answer
Only one part, the label that names the model
The source states that loading a pretrained GPT model reuses the same code you already wrote, and only one part of that code needs to change: the model specification. Everything around it, including the pretraining and fine-tuning code, stays put.
Tracing the Shared Code Path
Think of the work you have done as one pipeline with a fixed skeleton. Pretraining, loading a pretrained model, and fine-tuning all run through the same code. The part that differs from one case to the next is small and specific. The diagram below shows the sequence of steps and marks where the shared code sits.
Read the flow from top to bottom. The model label is read first. The code you already know then runs unchanged. Running it starts a download of the chosen pretrained model, and the result occupies storage on your machine. Only the first node is different from the procedure you already follow. The rest is the same code you have been writing.
The Model Label Selects the Weights
The model specification is a label that combines a name with a size. In the source, the two labels being contrasted are gpt2-small (124M) and gpt2-medium (355M). The number in parentheses, given as 124M or 355M, indicates the size of the model. Changing that label from the small variant to the medium variant is what tells the loading step which pretrained model to fetch.
Treat the model label as the one input that decides which pretrained network you get. Everything around it stays put. Keep in mind that the label carries both a name and a size, and the size token is the part that distinguishes the two variants the source compares.
Following the Download and Its Storage Cost
Once the size is specified, running the loading code starts a download of the chosen pretrained model. Notice that the download is the expensive part of this sequence, not the code itself. The code matches what you already know. The cost lands on your disk and on your network connection, which is why the storage figure matters.
Estimating the storage footprint of the small model
The source gives the medium-sized GPT model a storage requirement of approximately 1.42 gigabytes and says it is roughly three times larger than the small model. Estimate how much storage the small model needs.
- 1Identify the known value
The medium model requires approximately 1.42 GB of storage.
- 2Identify the ratio
The medium model is roughly three times larger than the small model, so the small model is roughly one third the size of the medium model.
- 3Divide the known value by the ratio
Computing $\frac{1.42}{3} \approx 0.47$ gives the estimate for the small model.
- 4Label the result correctly
The 0.47 GB figure is an approximate estimate derived from the stated ratio. The source does not state it directly.
The small model needs roughly 0.47 GB of storage, an estimate derived from the stated ratio.
The difference in size is tied to the size token in the label. The source ties the 355M variant to the medium model and the 124M variant to the small model, and it gives the ratio between their storage needs. The source does not explain beyond this ratio what drives the difference, so the size token is the only factor it identifies.
Mistakes Beginners Make with Model Labels
Common mistake Rewriting the pretraining or fine-tuning code when loading a pretrained model
Why it failsThe source says loading reuses the same code, and only the model specification needs to change.
Do this insteadKeep the existing code and change only the model label.
Common mistake Assuming the medium model is only slightly larger than the small one
Why it failsThe source states the medium model is roughly three times larger.
Do this insteadExpect the medium model to need about three times the storage of the small model.
Common mistake Treating the 0.47 GB figure as a number the source states
Why it failsThe source gives 1.42 GB for the medium model and a ratio. The small-model figure is derived from that ratio.
Do this insteadPresent 0.47 GB as an approximate estimate, not a sourced fact.
Common mistake Overlooking the download when the code looks short
Why it failsThe source notes the download, not the code, is the expensive part.
Do this insteadCheck your disk space and network connection before running the loading code.
Checking Your Understanding
You have working code that loads gpt2-small (124M). You want to load gpt2-medium (355M) instead. Which parts of the code should change, and roughly how much more storage should you plan for?
Hints
- Recall which single part of the code the source says must change.
- The source gives a ratio between the two models' storage needs.
Key Takeaways from Loading Pretrained Models
- 1Loading a pretrained GPT model uses the same code you already wrote for pretraining and fine-tuning.
- 2The only change required is the model specification, such as gpt2-medium (355M) in place of gpt2-small (124M).
- 3Running the loading code starts a download, and the medium model requires approximately 1.42 GB of storage.
- 4The medium model is roughly three times larger than the small model, so the small model needs about one third of that storage, an approximate estimate of about 0.47 GB.
At a glance
- Loading a pretrained GPT model reuses the code you already wrote for pretraining and fine-tuning.
- The model label, which combines a name with a size such as 124M or 355M, is the single input that selects the pretrained model.
- The download, not the code, is the expensive part, and the medium model needs approximately 1.42 GB of storage.
- The medium model is roughly three times larger than the small model, so the small model needs about one third of that storage, an estimate of about 0.47 GB.