Inspecting Model Architecture Settings
The settings output describes the architecture, which is the shape of the network.
Why Model Weights Must Exist on Disk
Before any GPT-2 model can do anything, its learned numbers have to exist on your machine. This article approaches that step the way a debugger would. You start with the code that performs it, predict what each line should produce, and then check that prediction against what the code actually does. The goal is not to memorize a recipe. The goal is to be able to point at the exact line where a download went wrong when something goes wrong.
The download logic is not written out line by line in the chapter. Instead, the chapter points you to a helper module hosted in an online code repository, and you download that module first. The first trace you will follow is therefore about fetching a single Python file. Only after that file is on disk does the more interesting work of loading GPT-2 begin.
Fetching One File with urlretrieve
Downloading a file from a URL in Python uses urllib.request.urlretrieve. The standard library module urllib.request does the network work, and urlretrieve takes two arguments: the address to fetch and the name to save the result under. Keep that two-argument contract in mind when you read the code. From your code's point of view, there are only two inputs going in and one file appearing on disk afterward. The network part of the job is hidden inside urlretrieve.
What do you think happens?
The helper module's URL ends in /gpt_download.py. Given that the local filename comes from the last segment of the URL, what local filename does urlretrieve save to?
Reveal answer
gpt_download.py
The filename here is derived from the last segment of the URL. The last segment of an address ending in gpt_download.py is gpt_download.py, so that is the name the file is saved under in the directory.
- The URL is the address to fetch, the first argument to urlretrieve.
- The last segment of the URL is found by splitting on slashes and taking the final piece.
- The local name is the second argument, the name to save the result under.
- This example is illustrative. Nothing here is executed or shown to produce output.
The diagram shows the mapping in the simplest form. The address is the key you fetch from, and the local filename is the name it is saved under. Because the source derives the filename from the last segment of the URL, you can predict the saved name just by reading the address, before you run anything.
Inspecting the Saved Module
A file appearing in the directory does not prove that it is the right file. The source recommends opening the downloaded module and reading it before continuing. You are checking two things: that the contents were saved correctly, and that the file really contains valid Python code. This inspection step is cheap insurance, because a file that is wrong will otherwise cause a failure later in a confusing way.
The Download-and-Load Flow
Once the module is in place, the process runs as a sequence of dependent steps. Each step relies on the one before it. Read the flowchart from top to bottom. The first step builds the URL and filename and saves the module. The next step checks that the saved file is valid Python. Only then can you import the function and call it, and that call is what fetches the seven GPT-2 files.
What download_and_load_gpt2 Returns
The call names a model size and a directory. In the source, the call uses model_size set to 124M and models_dir set to gpt2. The function returns two objects, which you assign to the names settings and params. Keep those two roles separate in your head, because later chapters depend on the difference.
| Output | What it holds | Role in the model |
|---|---|---|
| settings | GPT-2 architecture settings | Describe the shape of the network |
| params | Weight parameters | Learned values that fill that shape |
The two outputs returned by download_and_load_gpt2
The settings output describes the architecture, which is the shape of the network. The params output holds the weight parameters, which are the learned values that fill that shape. One tells you what the network looks like, and the other supplies the numbers that make it useful. Neither is a substitute for the other.
The Seven Files for the 124M Model
A single call to the loading function fetches several files, not one. For the 124M parameter model, the source lists seven files that the call downloads. The source names checkpoint, encoder.json, hprams.json, the model.ckpt files, and vocab.bpe. The full seven-file list is not reproduced in this source pack, so use the names above as the ones to recognize and do not treat this list as complete.
- checkpoint
encoder.jsonhprams.json- model.ckpt files
- vocab.bpe
The source does not spell out which individual file feeds which output, so the diagram shows the seven files as one group entering the loader. The reason they are fetched together is that the loader expects this set of artifacts for this model size. Because settings and params are both built from that set, a partial set is not enough to produce a complete result.
Locating a Failure in the Process
Finding the step where the process broke
You run the download code. The module saves and you open it, but the later import raises an error that suggests the file contents are not valid Python. Where did the process fail, and what should you check?
- 1Identify the first working step
The URL was built and urlretrieve saved a file, so the save step appears to have produced a file on disk.
- 2Check the inspection step
The source says to open the saved module and confirm it is valid Python. That check was skipped or did not catch the problem, so the bad contents got through.
- 3Match the symptom to the path
A server error page or a partial file would be saved under the expected name and would fail at import, which is consistent with the symptom.
- 4Choose the response
Rather than rewriting working code, reopen the saved file and compare it with what you expect, then retry the download if the contents are wrong.
The failure is at the stage between saving the module and importing it. The likely cause is a bad saved file, such as an error page or a partial download, rather than the loader code itself.
The source names three broad reasons the download might not work: an intermittent internet connection, problems on the server side, or changes in how OpenAI shares the weights of the open-source GPT-2 model. These are not all within your control. The right response is to step back and check the state of the process at each stage, using the flowchart, rather than rewriting code that may be correct.
Common Mistakes When Downloading
Common mistake Treating a file on disk as proof of a correct download
Why it failsA file can exist under the right name and still hold an error page or a partial download.
Do this insteadOpen the saved module and confirm it contains valid Python before importing from it.
Common mistake Misspelling the settings file when searching for it
What it looks likeLooking for hyperparams.json when the source spells it hprams.json
Why it failsThe file is present, but the search fails, which wrongly suggests the download failed.
Do this insteadUse the spelling exactly as shown in the source: hprams.json.
Common mistake Assuming the local filename is arbitrary
Why it failsThe filename here is derived from the last segment of the URL, so it is predictable.
Do this insteadRead the address and predict the saved name from its last segment before running the code.
Common mistake Rewriting working code when the download fails
Why it failsThe cause may be a connection problem or a change on the server side, which the code does not control.
Do this insteadCheck the state of each step in the process before changing the loader code.
Practice Tracing a Download
A helper module is hosted at an address whose final segment is tools_check.py. Predict the local filename urlretrieve will save it under, then name the step where a fetch failure would stop the process and the step where a bad file would be caught.
Hints
- The filename is derived from the last segment of the URL.
- Look at the flowchart: a fetch failure happens at the save step, while a bad file is caught when the saved module is inspected or imported.
Key Takeaways
urllib.request.urlretrievetakes the address to fetch and the local name to save it under, and in this process the local filename is derived from the last segment of the URL.- Downloading the helper module is a first step, and you should open the saved file to confirm it contains valid Python before importing from it.
download_and_load_gpt2returns two outputs: settings describes the GPT-2 architecture, and params holds the learned weight values that fill that architecture.- The 124M model fetches seven files together, including checkpoint,
encoder.json,hprams.json, the model.ckpt files, and vocab.bpe, because the loader expects this set for the model size. - A failure can be traced by following the steps in order, from saving the module to inspecting it, importing it, and fetching the seven files.