Text Generation with GPT Models

The model expects its input to carry a batch dimension, so this step is required even though the token values themselves do not change.

Programming

A Text Prompt Must Become Numbers

A GPT model does not read the characters of a sentence the way a person does. It works with numbers. Before any text generation can happen, the prompt has to be translated into a form the model can accept. This article follows one concrete goal from start to finish: take the short prompt "Hello, I am" and prepare it as input for the generate_text_simple function.

Preparing the prompt takes two conversions. The first turns the string into a list of integer token IDs. The second turns that Python list into a PyTorch tensor, and adds a batch dimension to it. Each conversion produces an intermediate result that you can inspect, and learning to inspect those results is the main skill of this article. When something goes wrong, the printed list and the printed tensor shape tell you which conversion caused the problem.

Tracing the Prompt Through Each Stage

Read the preparation code as a chain of stages. Each stage consumes the output of the previous one. The prompt string goes into the tokenizer, the tokenizer's list goes into torch.tensor, and the tensor is then reshaped with unsqueeze(0). Nothing in the chain can be skipped, and a problem introduced early is carried forward into every later stage.

What each stage of the preparation pipeline produces, and how the shape changes at each step. Only the final stage adds the batch dimension.

Notice that the text itself does not travel past the first stage. Only the list of IDs is passed to the tensor step. This is why, if the final tensor looks wrong, you should first check whether the list is wrong. The tensor step can only convert what it is given.

What tokenizer.encode() Returns

Definition

tokenizer.encode(input_text) takes a string as its input and returns a list of integers. Each integer is a token ID. These IDs are the representation the rest of the pipeline works with.

Think of the encoder as a lookup from pieces of text to numbers. The string goes in, and a list of numbers comes out, with one number standing for each token the tokenizer identified in the text. The exact numbers depend on the tokenizer's vocabulary, so the same text can produce different IDs under a different tokenizer. For that reason, you should always print the encoded list rather than assume what it contains.

Turning the List into a Tensor

The expression torch.tensor(encoded) converts the Python list of token IDs into a PyTorch tensor. After this step the data has a shape, and that shape is what the model checks. A flat list of n token IDs becomes a one-dimensional tensor whose shape has a single entry, the number of tokens.

python

prompt = "Hello, I am"
encoded = tokenizer.encode(prompt)
input_tensor = torch.tensor(encoded).unsqueeze(0)
print(encoded)
print(input_tensor.shape)

prompt = "Hello, I am"
encoded = tokenizer.encode(prompt)
input_tensor = torch.tensor(encoded).unsqueeze(0)
print(encoded)
print(input_tensor.shape)

Output (expected)

[101, 7, 42]
torch.Size([1, 3])

Notes on the code

prompt holds the string that will be encoded
tokenizer.encode(prompt) returns a Python list of integer token IDs
torch.tensor(encoded) converts that list into a tensor, then unsqueeze(0) adds the batch dimension
The two print calls report the intermediate list and the final shape

Adding the Batch Dimension with unsqueeze(0)

The method call .unsqueeze(0) adds a new dimension at index 0. This new leading dimension is the batch dimension. The model expects its input to carry a batch dimension, so this step is required even though the token values themselves do not change. The same numbers are still there; they are simply wrapped in one more level of structure.

Before

tensor([101, 7, 42])  # shape [3]

A one-dimensional tensor with 3 token IDs

After

tensor([[101, 7, 42]])  # shape [1, 3]

A two-dimensional tensor with one batch row containing the same 3 token IDs

The same three token IDs before and after unsqueeze(0). The values do not change; the new leading dimension changes the layout and the shape.

What do you think happens?

If the tensor before unsqueeze(0) has shape [3], what shape does it have after unsqueeze(0)?

[3]
[1, 3]
[3, 1]
[1, 1, 3]

Reveal answer

Answer:

[1, 3]

unsqueeze(0) inserts a new dimension of size 1 at index 0. The original dimension of size 3 moves to index 1, giving shape [1, 3]. The values are unchanged.

Worked example

Tracing the shape of a longer prompt

Suppose the encoded list for a prompt has 5 token IDs. What shape does the tensor have before and after unsqueeze(0)?

1
Start with the list
The encoded result is a flat Python list with 5 integers.
2
Apply torch.tensor
The list becomes a one-dimensional tensor whose only dimension has size 5, so its shape is [5].
3
Apply unsqueeze(0)
A new dimension of size 1 is inserted at index 0, and the existing dimension of size 5 shifts to index 1.
4
Read the result
The final shape is [1, 5]: one batch entry containing a sequence of 5 token IDs.

Result

Before unsqueeze(0) the shape is [5]; after unsqueeze(0) the shape is [1, 5].

Locating Where a Shape Mismatch Starts

When the model rejects its input, the shape is usually the first clue, but the shape alone does not say where the problem began. Because each stage consumes the previous stage's output, you trace backward through the chain. Start by printing the encoded list. If the list already has the wrong length or wrong content, the problem is in the encoding stage. If the list looks right but the tensor shape is unexpected, the problem is in the tensor-preparation stage, most often at torch.tensor or unsqueeze.

Mistakes When Preparing Model Input

Common mistake
Passing the original string to torch.tensor
What it looks like
```
torch.tensor("Hello, I am")
```
Why it fails
The tensor step is meant to receive the list of integer token IDs produced by the tokenizer. Only the list is passed to this step, and the text itself is not.
Do this instead
```
Pass the encoded list: torch.tensor(encoded).
```
Common mistake
Forgetting that unsqueeze(0) is required
What it looks like
```
input_tensor = torch.tensor(encoded)
```
Why it fails
The model expects its input to carry a batch dimension. Without unsqueeze(0), the tensor has shape [n] and lacks the leading batch dimension, even though the token values are correct.
Do this instead
Add the batch dimension with .unsqueeze(0) after creating the tensor.
Common mistake
Assuming the shape change alters the token values
Why it fails
unsqueeze(0) only adds a dimension. The same token IDs remain in the same order, so a mismatch is not caused by the batch dimension changing the content.
Do this instead
Compare the printed list with the tensor contents to confirm that the values are unchanged.
Common mistake
Treating the printed IDs as universal
Why it fails
Token IDs come from the tokenizer's vocabulary, so the numbers depend on which tokenizer is used.
Do this instead
Always print the encoded list from your own tokenizer rather than relying on numbers copied from elsewhere.

Practice Tracing the Pipeline

EASY

A colleague runs the preparation code on a prompt and reports that the model rejects the input. The printed encoded list has 4 integers, but the printed tensor shape is [4] instead of [1, 4]. Which stage is responsible, and what single change would fix it?

Hints

Ask which stage adds the batch dimension.
The encoded list is correct, since it has 4 integers.
Look at whether unsqueeze(0) was applied after torch.tensor.

Your attempt (not saved)

MEDIUM

Write the three lines that take a string called prompt, encode it, convert it to a tensor with a batch dimension, and print both the encoded list and the final shape. Then predict the printed shape for a prompt that encodes to 6 token IDs before you run it.