Concepts / Tokenization and Input Encoding

Tokenization and Input Encoding

The first dimension counts how many input sequences are processed together, and the second counts the tokens in each sequence.

  • Programming

Checking the Text Before the Model Sees It

This article works in the opposite order from most explanations. Instead of starting with fine-tuning in the abstract, we start with a short piece of code that prepares text for a pre-trained GPT model and checks what it produces. Once you can read that output, the idea of changing only some layers of the model becomes much easier to follow.

The starting point is a model that was already built and loaded in an earlier step. A new output layer has been added to it, and certain layers have been marked as trainable or nontrainable. Even after those changes, the model can still be used the same way as before: you feed it a text example and it processes that text. So the first question is simply what the text looks like after it has been prepared for the model.

What do you think happens?

The sentence is 'Do you have time'. How many tokens does it turn into?

  • 2

  • 3

  • 4

  • 5

Reveal answer
Answer:

4

The sentence contributes four tokens, so the number of tokens in the encoded sequence is 4.

From Text to Token IDs to a Tensor

The conversion happens in two steps. First, the tokenizer turns the text into a sequence of integer token IDs. Second, torch.tensor wraps that sequence in a tensor, and unsqueeze(0) inserts a new dimension at the front. The print statements in the source snippet report both the values and the dimensions, so you can check each step separately rather than trusting the final result alone.

python
Output (expected)
tensor([[ ... four integer token IDs ... ]])
torch.Size([1, 4])
Notes on the code
  • torch.tensor(ids) wraps the flat list of token IDs in a one-dimensional tensor
  • unsqueeze(0) inserts a new dimension at position zero, so the flat sequence becomes a batch of one
  • The printed shape confirms the two dimensions: one sequence and four tokens
Do you have timeTokenizerSequence of 4 tokenIDstorch.tensorunsqueeze(0)adds the batch dimensionInput tensor shape(1, 4)tokenizeinteger IDswrapadd dimensionmodel input
How does a text string move through tokenization, token IDs, and batching until it becomes the tensor the model receives?

What Fine-Tuning Changes in the Weights

Fine-tuning adjusts a pre-trained model's weights to fit a specific task. It does not have to touch every part of the model. One option is to fine-tune the entire model, which can improve predictive performance, and then assess how much that change helps. Another option is to make only some layers trainable while keeping the rest nontrainable. A nontrainable layer keeps the weights it was loaded with, while a trainable layer is allowed to have its weights adjusted during training. Choosing which layers fall into each group is the core decision in this process.

All layersWeights adjustedduring trainingSelected layersonlyRemaining layers keeploaded weights
How do full-model updates differ from selective-layer updates in terms of which parameters receive weight updates?

Which GPT Layers Are Trainable

In the GPT example, the model contains 12 repeated transformer blocks. The new output layer is trainable, and so are the final LayerNorm and the last transformer block. The remaining 11 transformer blocks and the embedding layers are kept nontrainable. This means most of the pre-trained structure is preserved, and only the parts closest to the output are adapted to the new task.

GPT modelEmbedding layersTransformer blocks 1to 11Transformer block 12Final LayerNormNew output layer
Where do the embedding layers, transformer blocks, final LayerNorm, and output layer sit, and which are trainable versus nontrainable?

Mistakes Beginners Make With Fine-Tuning Setup

  • Common mistake

    Assuming the tokenizer output is already a batch

    What it looks like

    Printing the tensor directly after torch.tensor(ids) and expecting two dimensions

    Why it fails

    A flat sequence of token IDs has only one dimension. The batch dimension appears only after unsqueeze(0) inserts it at position zero.

    Do this instead

    Print the shape after each step and confirm it reads (1, 4) for a single four-token sentence.

  • Common mistake

    Reading the first dimension as the number of tokens

    What it looks like

    Treating the 1 in (1, 4) as the sentence length

    Why it fails

    The first dimension is batch_size, which counts input sequences. The second dimension, num_tokens, counts tokens in each sequence.

    Do this instead

    Read the shape as (batch_size, num_tokens) every time.

  • Common mistake

    Assuming fine-tuning changes every layer

    What it looks like

    Believing all 12 transformer blocks update because the model is being fine-tuned

    Why it fails

    Fine-tuning can update only selected layers. In the GPT example, the 11 earlier transformer blocks and the embedding layers stay nontrainable.

    Do this instead

    Check which layers are marked trainable before interpreting training behavior.

  • Common mistake

    Treating nontrainable layers as removed from the forward pass

Finding Where an Unexpected Result Begins

When a fine-tuning run does not behave as expected, the cause usually sits at one of three points: the encoded input, the trainable-layer selection, or the way the model is then used. Checking those points in order narrows the failure quickly. The first check is always the encoded input shape, because a wrong tokenization or batching step affects everything downstream.

Tokenize datacheck encoded input shapefirstLoad pre-trainedmodelFreeze layerscheck trainable-layerselection secondTrainEvaluatecheck how the model is usedlast
Following the steps from tokenizing data through loading the model, freezing layers, training, and evaluating, which step is checked first when output looks wrong?

Tracing a Fine-Tuning Problem

Worked example

Diagnosing a run where the input looks wrong

A fine-tuning run returns unexpected predictions. The encoded input for the sentence 'Do you have time' is printed with shape (1, 3). Where should you look first, and what does the shape tell you?

  1. 1
    Read the second dimension

    The second dimension is num_tokens. It reads 3, but the sentence contributes four tokens, so the encoding does not match the intended text.

  2. 2
    Read the first dimension

    The first dimension is batch_size, which reads 1. One sequence is being processed, which is correct.

  3. 3
    Locate the failure point

    The problem sits at the first checkpoint, the encoded input, because the token count is wrong. Trainable-layer selection and model use come later in the check order.

Result

The encoded input is the first place to investigate. Re-run the tokenization step and confirm the shape reads (1, 4) before examining layer settings.

MEDIUM

A colleague says that after fine-tuning, the embedding layers must have changed because the model was retrained. Using the GPT example, explain why this may be wrong and which layers are the ones that receive weight updates.

Hints
  • Look at which layers are described as nontrainable in the GPT example.
  • The trainable group is the new output layer, the final LayerNorm, and the last transformer block.

Key Takeaways

  • Fine-tuning adjusts a pre-trained model's weights to fit a specific task, either across the whole model or only in selected trainable layers.
  • Tokenization turns text into integer token IDs, and unsqueeze(0) adds a batch dimension, giving a shape of (batch_size, num_tokens) such as (1, 4).
  • In the GPT example, the new output layer, the final LayerNorm, and the last transformer block are trainable, while the 11 earlier transformer blocks and the embedding layers are nontrainable.
  • When results look wrong, check the encoded input shape first, then the trainable-layer selection, then how the model is used.

Related Topics

Related concepts

More concepts starting with T