Decoding Token IDs Back into Text

The length of the decoded string tells you about the text, not about the list. Use the list length to check how many IDs you passed in, and use the decoded string to read what those IDs represent.

Programming

Why Models Need the Reverse Step

A language model does not work with text directly. It works with token IDs, which are integers. Before a prompt can be fed to a model, the text must be converted into those numbers. Sometimes you need to go the other way: you have a sequence of token IDs and want to know what text they stand for. That reverse step is called decoding. It is the piece that turns model-facing numbers back into human-readable text.

This article follows one concrete goal. Take the short prompt "Hello, I am" and prepare it as input for the generate_text_simple function. Along the way you will see the encoding pipeline, then reverse it with decoding. Before reading on, pause and predict: if you convert the prompt to a list of integers, and then ask for the shape of the tensor built from that list, what do you expect the shape to be? Hold on to your guess, because the trace below will check it.

Tracing the Encode Pipeline

The call takes the form tokenizer.encode(input_text). Its input is a string and its output is a list of integers. Those integers are the representation the rest of the pipeline works with. The text itself is not passed to the tensor step; only the list of token IDs is. Each stage consumes the output of the previous one, so a problem at any stage carries forward into every later stage.

python

prompt = "Hello, I am"
encoded = tokenizer.encode(prompt)
input_tensor = torch.tensor(encoded).unsqueeze(0)
print(encoded)
print(input_tensor.shape)

prompt = "Hello, I am"
encoded = tokenizer.encode(prompt)
input_tensor = torch.tensor(encoded).unsqueeze(0)
print(encoded)
print(input_tensor.shape)

Notes on the code

Line 1 assigns the prompt string
Line 2 converts the string into a Python list of integer token IDs
Line 3 turns that list into a PyTorch tensor and adds a batch dimension at index 0
The two print statements report the intermediate list and the final tensor shape

Reading the code line by line matches the pipeline. The expression torch.tensor(encoded) converts the Python list of token IDs into a PyTorch tensor. The method call .unsqueeze(0) then adds a new dimension at index 0. This is the batch dimension. The model expects its input to carry a batch dimension, so this step is required even though the token values themselves do not change.

The diagram shows that only the final stage adds the batch dimension, which is why its effect appears in the shape printed at the end of the trace. The encoding stage and the tensor stage do not add a batch dimension. If you want to locate a shape problem, check the printed list first, since that tells you whether encoding produced what you expected. Then check the tensor shape to see whether the tensor step and the unsqueeze step behaved correctly.

What unsqueeze(0) Changes

Suppose the encoded list has n token IDs. Converting it with torch.tensor(encoded) gives a one-dimensional tensor whose shape is [n]. Calling .unsqueeze(0) inserts a new dimension at index 0, so the shape becomes [1, n]. The number of token IDs, n, is unchanged. Only the number of dimensions has grown from one to two. The new leading dimension of size 1 is the batch dimension: it holds the one sequence being passed to the model.

Before

ids = torch.tensor(encoded)
print(ids.shape)

shape [n], one dimension

After

ids = torch.tensor(encoded).unsqueeze(0)
print(ids.shape)

shape [1, n], two dimensions

Decoding Token IDs into a String

Definition

Token decoding reassembles an original string from a list of token IDs using a tokenizer's decode method.

The decode method is called on the tokenizer object and receives a list of token IDs as its argument. It returns a string, and print can then display that string. Read the call from the inside out. The list of IDs is the input. The tokenizer uses its decode method to turn that input into text. The print function is the last step, which makes the returned text visible.

python

token_ids = [15496, 11, 314, 716]
decoded_text = tokenizer.decode(token_ids)
print(decoded_text)

token_ids = [15496, 11, 314, 716]
decoded_text = tokenizer.decode(token_ids)
print(decoded_text)

Notes on the code

token_ids is a Python list of integers, the input the decode method expects
tokenizer.decode(token_ids) returns a string
print displays the returned string

The diagram shows two paths leaving the list of token IDs. One path goes forward into the tensor the model consumes. The other path goes backward through decode to produce text again. Decoding does not need the tensor; it takes the list of IDs directly.

Why the Output Length Differs from the ID Count

A common expectation is that each token ID produces exactly one character or one word in the output. The source warns against this. The number of characters in the decoded output does not equal the number of token IDs, so you should not count characters to verify the input. The output has a different number of characters than the list has items, and it can contain characters, including a space, that do not map neatly onto one ID each.

Key point

The length of the decoded string tells you about the text, not about the list. Use the list length to check how many IDs you passed in, and use the decoded string to read what those IDs represent.

The relationship is not one-to-one because a single token may cover part of a word, a whole word, or punctuation. Some tokens therefore contribute several characters, and some contribute only a fragment that joins a neighbouring fragment in the final string. When you verify a decode, compare meaning and content rather than counting characters against the number of IDs.

Mistakes When Decoding Token IDs

Common mistake
Passing a single integer to decode instead of a list
What it looks like
```
tokenizer.decode(15496)
```
Why it fails
The decode method takes the list of token IDs as its input. The source states the method takes the list of token IDs, not a single integer.
Do this instead
```
Wrap the ID in a list: tokenizer.decode([15496])
```
Common mistake
Counting decoded characters to verify the number of token IDs
What it looks like
Expecting a decoded string of length 4 because the list has 4 IDs
Why it fails
The number of characters in the decoded output does not equal the number of token IDs, so the count does not check the input.
Do this instead
Compare the printed text with the original prompt, and use len() on the list to check the ID count.
Common mistake
Passing the raw text into the tensor step
What it looks like
```
torch.tensor(prompt).unsqueeze(0) where prompt is a string
```
Why it fails
Only the list of token IDs is passed to the tensor step. The text itself does not go into torch.tensor.
Do this instead
```
Encode first: torch.tensor(tokenizer.encode(prompt)).unsqueeze(0)
```
Common mistake
Assuming the shape is wrong at the last step without checking earlier ones
Why it fails
Each stage consumes the previous stage's output, so a problem can originate earlier and only appear later as a shape error.
Common mistake
Expecting unsqueeze(0) to change the token values
Why it fails
unsqueeze(0) adds a batch dimension at index 0; the token values are unchanged.

Practice: Predict Before You Run

EASY

A prompt is encoded into a list of 6 token IDs. You build a tensor from the list and apply unsqueeze(0). Before running anything, predict the shape printed by print(input_tensor.shape), and say which step in the pipeline determines the leading 1. Then predict whether the decoded string from tokenizer.decode(token_ids) will contain exactly 6 characters.

Hints

Start with the shape of torch.tensor(encoded) for a list of length 6.
unsqueeze(0) adds a dimension at index 0 without changing the number of IDs.
The decoded string length is not tied to the number of token IDs.

Your attempt (not saved)

MEDIUM

Write a short snippet that takes the prompt "Hello, I am", encodes it, converts the result into a batched tensor, prints the list of IDs, prints the tensor shape, and then decodes the list back into a string and prints it.