Selecting Trainable Layers

Trainable layers in the GPT example are the output layer, the final LayerNorm, and the last transformer block. Everything else, including the embedding layers and the remaining 11 transformer blocks, is nontrainable.

Programming

Starting From the Encoded Text

This article works in the opposite order from most explanations. Instead of starting with fine-tuning in the abstract, we start with a short piece of code that prepares text for a pre-trained GPT model and checks what it produces. Once you can read that output, the idea of changing only some layers of the model becomes much easier to follow.

The starting point is a model that was already built and loaded in an earlier step. A new output layer has been added to it, and certain layers have been marked as trainable or nontrainable. Even after those changes, the model can still be used the same way as before: you feed it a text example and it processes that text. So the first question is simply what the text looks like after it has been prepared for the model.

What do you think happens?

The sentence is "Do you have time". How many tokens does it turn into, and what is the shape of the tensor after an extra dimension is added at position zero?

Three tokens, shape (3, 1)
Four tokens, shape (1, 4)
Four tokens, shape (4, 1)

Reveal answer

Answer:

Four tokens, shape (1, 4)

The sentence contributes four tokens. Adding a dimension at position zero with unsqueeze(0) creates a tensor whose first dimension is 1 (one sequence in the batch) and whose second dimension is 4 (four tokens in that sequence).

From Text to a Tensor in Two Steps

The conversion from text to model input happens in two distinct steps. Each step can be checked separately, which is what makes the snippet useful for debugging.

1
The tokenizer turns the text into a sequence of integer token IDs. Each token becomes one integer.
2
torch.tensor wraps that sequence of integers in a tensor, which is the data structure the model accepts as input.
3
unsqueeze(0) inserts a new dimension at the front of the tensor, so that the sequence is treated as one item in a batch.
4
The print statements in the snippet report both the values and the dimensions after each step, so you can confirm that each stage produced what you intended.

The step that adds a dimension is easy to overlook, but it changes how the model reads the data. Without it, the tensor would hold only the token sequence itself. With it, the tensor explicitly declares how many input sequences are processed together. That declaration is what the shape reports, and it is the first thing to read when you check the encoded input.

Reading the Shape of the Input Tensor

The shape of the encoded input is written as (batch_size, num_tokens). The two dimensions answer two different questions. The first dimension counts how many input sequences are processed together. The second dimension counts the tokens inside each sequence. Reading the shape in this way tells you immediately whether the text was encoded the way you intended.

What does each dimension of the input tensor index, and how does a position in the grid map to a specific token in a specific example? The tensor has shape (1, 4): one row for the single sequence, and four positions along that row, one for each token.

The diagram shows the grid for the example sentence. The first index selects the sequence, which here is always 0 because there is only one. The second index selects a token position within that sequence. So a position in the grid maps to one token of one example. If the batch held two sentences, the first dimension would grow to 2 and each sentence would keep its own row of token positions.

Worked example

Checking the shape of an encoded sentence

A snippet encodes the sentence "Do you have time" and reports the tensor shape. Is the encoding what you intended?

1
Count the tokens
The sentence contributes four tokens, so num_tokens should be 4.
2
Count the sequences
The example contains one sentence, so batch_size should be 1.
3
Compare with the reported shape
If the printed shape is (1, 4), the text was encoded as one sequence of four tokens, which matches the intent.
4
Interpret a mismatch
If the second dimension were not 4, the tokenization step is the first suspect. If the first dimension were not 1, the unsqueeze step or the batching is the first suspect.

Result

A printed shape of (1, 4) confirms the encoding is correct, so any later problem must come from somewhere other than the input text.

What Fine-Tuning Changes

Fine-tuning adjusts a pre-trained model's weights so that the model fits a specific task. A pre-trained model already has weights that were learned before this task, and fine-tuning continues training so that those weights move toward the new job. The change in weights is what alters the model's predictions, which is why fine-tuning can improve predictive performance on the task.

Fine-tuning does not have to touch every part of the model. There are two broad options. The first is to fine-tune the entire model, which can improve predictive performance, and then to assess how much that change helps. The second is to make only some layers trainable while keeping the rest nontrainable. The choice of which layers fall into each group is the core decision in this process.

Definition

A nontrainable layer keeps the weights it was loaded with. A trainable layer is allowed to have its weights adjusted during training.

Question	Update the whole model	Update only selected layers
Which parameters receive weight updates?	Every layer in the model	Only the layers marked trainable
What happens to the pre-trained structure?	It may shift throughout the model	Most of it is preserved because nontrainable layers keep their loaded weights
What is assessed afterward?	How much the full change helps predictive performance	Whether the selected layers are enough to adapt the model to the new task
Main trade-off	A broader change to what was learned earlier	A narrower change that keeps more of the original model intact

Practical difference between full fine-tuning and selecting trainable layers

Which GPT Layers Get Updated

In the GPT example, the model contains 12 repeated transformer blocks. The new output layer is trainable, and so are the final LayerNorm and the last transformer block. The remaining 11 transformer blocks and the embedding layers are kept nontrainable. The diagram below lays out this grouping from the whole model down to its parts.

Which parts of the stacked model (embeddings, each transformer block, the output head) are frozen and which get weight updates during fine-tuning? Nodes marked as changed receive weight updates; the others keep their loaded weights.

Read the diagram from the bottom of the stack upward to see the logic. The embedding layers and the earlier transformer blocks are left alone, so the representations the model learned before this task are preserved. Only the parts closest to the output are adapted to the new task: the last transformer block, the final LayerNorm, and the new output layer. This is why the grouping is described as preserving most of the pre-trained structure.

Key point

Tracing the Fine-Tuning Pipeline

When a fine-tuning run does not behave as expected, the cause usually sits at one of three points: the encoded input, the trainable-layer selection, or the way the model is then used. The pipeline below traces the order in which those points are reached. Checking them in that order narrows the failure quickly.

At which stage of the pipeline does the output first diverge from what was expected? Work through the stages in order and stop at the first one that does not match what you intended.

Follow the pipeline from left to right. The encoding step produces the tensor, and the shape check confirms what it holds. The trainable-layer selection decides which weights can change during training. The training loop then updates only the trainable weights. Finally, the model is used on a text example. If the output is wrong, check the shape first, then the trainable-layer selection, and finally the way the model is used. The earliest stage that does not match your expectation is where the outcome diverged.

MEDIUM

A fine-tuning run returns results that look wrong. The encoded input shape prints as (1, 4) for a four-token sentence, and the output layer, final LayerNorm, and last transformer block are marked trainable. Which stage should you examine next, and why?

Hints

The encoded input was already checked and it matches the expected shape.
The source identifies the trainable-layer selection as the second point to check, after the encoded input.

Your attempt (not saved)

Mistakes When Selecting Layers

Common mistake
Assuming that a trainable layer keeps its original weights
Why it fails
A trainable layer is allowed to have its weights adjusted during training. Only nontrainable layers keep the weights they were loaded with.
Do this instead
Treat trainable as the setting that permits change, and nontrainable as the setting that preserves the loaded weights.
Common mistake
Reading the first dimension of the input as the number of tokens
What it looks like
A tensor with shape (1, 4) is read as having one token instead of four.
Why it fails
The shape is (batch_size, num_tokens). The first dimension counts sequences in the batch, and the second counts tokens in each sequence.
Do this instead
Read the first index as how many sequences are processed together and the second as how many tokens each sequence contains.
Common mistake
Debugging the training loop before checking the input
Why it fails
When results look wrong, the source says to check the encoded input shape first, then the trainable-layer selection. Starting later in the pipeline can waste effort on a problem that began earlier.
Do this instead
Work through the checks in order: encoded input, trainable-layer selection, then the way the model is used.
Common mistake
Assuming that fine-tuning must update every layer
Why it fails
Fine-tuning can update the whole model, but it can also make only some layers trainable while keeping the rest nontrainable.
Do this instead
Decide deliberately which layers fall into each group, since that choice is the core decision in the process.

Practice: Locate the Divergence

EASY

Without looking back, write down the order of the three checks used when a fine-tuning run looks wrong. Then, for the sentence "Do you have time", state its tensor shape and what each dimension counts.