Loading Pre-trained Model Weights

Imagine you want a program that can write fluent sentences. One option is to write rules for grammar and word choice by hand. The source notes that language is too varied for that to work. A different approach is to let a model learn patterns from a large body of text, and then use what it learned to generate new text. The process of building that learned starting point is called pretraining. It is the stage where a model is first trained on text before anyone asks it to do a specific job.

This article is about one practical consequence of that idea. Once a pretrained model exists, you can either produce one yourself by running training, or you can start from weights that someone else already produced and load them into a model you build. The rest of the article explains what each route involves, and then traces a real loading listing line by line so you can see how the pieces connect.

In this course, pretraining is treated as the second major stage of work. The first stage prepares the ingredients. It covers two things: the data preprocessing steps and the attention mechanism that sits at the core of modern LLMs. The second stage puts those ingredients together, showing how to code and pretrain a GPT-like model that can generate new text, and then how to evaluate the result.

Data preprocessing comes first because raw text is rarely in a shape a model can learn from directly. Preprocessing is the part of the workflow that turns collected text into training material. Every later step depends on this preparation, so it is worth understanding before you move on to the model itself.

The attention mechanism is the second ingredient from stage 1. The source describes it as being at the heart of every LLM, and it is the component the course asks you to code yourself. Think of attention as the part of the model that lets each position in a sequence take other positions into account when it builds its representation. Stage 1 builds this mechanism in isolation, so that stage 2 can use it as a working part rather than a mystery.

A helpful way to hold the whole topic in your head is to follow one path. Start with raw data and ask what must happen before a model can learn from it. Then ask what the model needs in its design, and what training does with that design. Finally, ask how you judge whether the trained model is any good, and whether you should train it yourself at all or start from weights someone else already produced.

Read the path from left to right. The first two steps belong to stage 1 and are about preparing data and understanding attention. The pretraining step belongs to stage 2. The branch for loading open model weights sits beside pretraining because it is an alternative way to reach a trained model without running the full training yourself. Both routes end at evaluation, because a model needs to be measured whichever way it was obtained.

Pretraining an LLM from scratch is a significant undertaking. The source states that it demands thousands to millions of dollars in computing costs for GPT-like models. That range is wide, and the bottom and top ends differ enormously, so the practical lesson is that full-scale pretraining is a major commitment rather than a routine exercise.

This cost is the reason the course narrows its goal. Instead of pretraining at full scale, stage 2 focuses on implementing training for educational purposes with a small dataset. Loading openly available model weights is the other route: you skip the full training run and start from parameters that were already produced.

The third topic in stage 2 is evaluating LLMs, which the source describes as essential for developing capable NLP systems. A trained model is a set of learned values, and those values alone do not tell you whether the model produces good text. Evaluation is the step that measures what the model does, so that you can compare versions and decide what to change.

Evaluation applies to both routes. A model you pretrain on a small dataset needs to be evaluated, and so does a model whose weights you loaded from an open source. Because the weights in the second case came from elsewhere, you cannot assume they behave as you expect in your own setting. Measuring is how you find out.

A GPT model is built from a set of settings, and pretrained GPT-2 weights are stored values produced for a specific model shape. Loading those weights is a two-part job: first create a GPT model object with the right settings, then pass the pretrained parameters into that object. The configuration is the starting point, so trace it first.

The download step needs a short model size string, but CHOOSE_MODEL holds a longer name that includes the parameter count in parentheses. The source derives model_size from CHOOSE_MODEL with one chained expression, which splits the name on spaces and strips the parentheses. The worked example below traces that idea on the medium model name, one operation at a time.

The value 355M is then passed to download_and_load_gpt2 as model_size, together with models_dir set to gpt2. That call returns two values, settings and params. This article focuses on params, which is the object handed to the weight-loading function in the next step.

The loading itself takes two lines. The first creates a GPT model instance from the merged configuration. The second takes that instance and the pretrained parameters and loads the weights into the model. The model must exist before the second line runs, because load_weights_into_gpt receives the model object as its first argument.

The diagram shows the flow of data across these steps. The configuration and the downloaded parameters are two separate inputs. The model is created from the configuration alone, and the loading function is the point where the two inputs meet. Because the model is built before the weights are loaded, the configuration has to be correct first: the model object needs the right shape before the stored values can be placed into it.

The same model object passes through three states in the listing. It starts as a GPT model built from BASE_CONFIG. The loading call changes it so that it holds the pretrained weights. The final line, model.eval();, is the last statement in the listing. The source does not explain what it does, so this article does not expand on it.

When a loaded model does not match what you expected, the fastest route is to ask which line of the listing first produced a value you did not expect. Each line depends on the ones above it, so the earliest wrong value usually points to the real cause. The mistakes below follow the order of the listing.