Encoding Input Text and Decoding Output: Tokenization and Token IDs
The tokenizer's .decode method converts a sequence of output token IDs back into text, and the decoded string is what gets printed.
Why the Printed Text Is Not Yet a Sentence
A language model does not produce words directly. It produces numbers, and those numbers are token IDs. Before you can read anything the model has generated, those IDs must be converted back into text. This article focuses on that final conversion step, which is performed with the tokenizer's .decode method.
The target for this pipeline is the sentence Hello, I am a model ready to help. When the trace finishes in the current chapter, however, the printed result is a string of plausible-looking words that do not form a coherent sentence. The source calls this gibberish and asks directly what happened. Before reading on, make a prediction: when the text comes out wrong, is the decode step the most likely culprit, or is something earlier in the pipeline responsible? Keep your answer in mind, because the rest of the article tests it.
Two Kinds of Data in the Pipeline
Notice the two kinds of data that appear in this pipeline. The input to the decode step is a sequence of token IDs, which are numbers. The output of the decode step is a string, which is what a person reads. The decode method is the bridge between those two representations.
Decoding is a translation step. It maps IDs to text and does not generate new content. This matters for debugging: if the decode step is translating faithfully, then whatever is wrong with the result must already be present in the IDs it received. Keep this idea in mind as you trace the code.
Tracing the Decoding Line
The decoding line from the chapter takes the output of the GPT model, turns it into a list of token IDs, passes that list to the tokenizer, and prints the resulting string. The source assigns the result of tokenizer.decode(...) to the name decoded_text. Inside the parentheses sits the expression out.squeeze(0).tolist(), which supplies the IDs.
Reading the decode expression from the inside out
What does the tokenizer.decode call receive, and what does it hand back, for the expression out.squeeze(0).tolist()?
- 1Start with out
The variable out holds the model's output. It is not yet a list of token IDs that the tokenizer can use directly.
- 2Apply squeeze(0)
The
squeeze(0)call reshapes the output so that it can be converted cleanly into a plain sequence. - 3Apply tolist()
The
tolist()call converts the prepared output into a plain Python list of token IDs. - 4Hand the list to decode
The tokenizer's .decode method receives that plain list of IDs. It looks up each ID and assembles the corresponding text into one string.
- 5Store and print
The returned string is assigned to
decoded_text, and then printed. The printed string is the end product of the whole pipeline.
decode receives a plain list of token IDs and returns a single string, which is stored in decoded_text and printed.
How Decode Turns IDs Into Text
The tokenizer keeps the correspondence between text pieces and token IDs. When it receives a sequence of IDs, it can look up each one and assemble the corresponding text. This is the reverse of the path that brought text into the model. Each integer is looked up on its own, and the pieces are then joined into one string.
The diagram below uses placeholder pieces, not real vocabulary entries, to show the lookup. Each token ID is a key. Its matching text piece is the value. Decode walks through the list in order and concatenates the values it finds.
Consider a list of three IDs returned by a model. Decode looks up the first ID, then the second, then the third, and returns one string made of the three pieces in that order. If the list had been in a different order, the output would be different, even though the same IDs were present. This is why the exact contents and order of the list matter.
Finding Where Gibberish Begins
The gibberish appears at the end of the pipeline, in the printed text. That position makes it tempting to suspect the decode step. The source's reasoning points elsewhere. Decode is a translation step that maps IDs to text. If the translation is faithful, the defect must be in the IDs it received, and those IDs come from the model's weights.
Why an Untrained Model Produces Gibberish
The chapter has implemented the GPT architecture and created a GPT model instance whose weights are initialized randomly. Those random weights are what generated the IDs that decode translated. The source's answer to the gibberish question is direct: the model cannot yet produce coherent text because it has not been trained.
This means the gibberish reflects the model, not the decode method. Decode did its job faithfully on the IDs it was given. Those IDs were produced by weights that have not yet learned anything about which words should follow which. Model training, which produces coherent output, is a large topic that the source defers to the next chapter.
Mistakes When Reading Decoded Output
Common mistake Assuming decode invents the words in the output
Why it failsDecode is a translation step. It maps IDs to text and does not generate new content.
Do this insteadTrace the printed words back to the IDs that decode received.
Common mistake Blaming the decode call as soon as the text looks wrong
Why it failsThe gibberish appears last in the pipeline, which makes decode look guilty even when it is translating faithfully.
Do this insteadCheck whether decode maps each ID to its piece, then check the IDs, then the model output.
Common mistake Expecting a coherent sentence from the untrained chapter model
Why it failsThe weights are initialized randomly, so the IDs they produce are not yet meaningful.
Do this insteadTreat the gibberish as expected behavior until model training is covered in the next chapter.
Common mistake Passing the raw model output straight to decode
Why it failsThe decode line first reshapes and converts the output with
squeeze(0)andtolist()so that decode receives a plain list.Do this insteadRead out.squeeze(0).tolist() from the inside outward before deciding what decode received.
Tracing a Bad Output
Suppose your script prints a string of plausible-looking words that do not form a sentence. Using only the pipeline you traced in this article, describe the order in which you would check the stages, and state what you would expect to find at each stage if the untrained chapter model is the cause.
Hints
- Start from the printed text and work backward through
decoded_textand the decode call. - Ask first whether decode mapped each ID faithfully before examining the IDs.
- Remember that the weights were randomly initialized in this chapter.
Summary of the Decoding Stage
- 1The tokenizer's .decode method converts a sequence of output token IDs back into text, and the decoded string is what gets printed.
- 2The decode line takes
out.squeeze(0).tolist(), which prepares the model output as a plain list of IDs before decode receives it. - 3Decode is a faithful translation step. If the printed text is wrong, first check the IDs it received, which come from the model.
- 4An untrained GPT model with randomly initialized weights cannot produce coherent text, so its gibberish reflects the model rather than decode.
At a glance
- The tokenizer's .decode method converts a sequence of output token IDs back into text, and the decoded string is what gets printed.
- The path runs from model output, through
squeeze(0)andtolist(), to a plain list of IDs, then to decode, which returns the string stored indecoded_text. - Decode is a translation step that does not generate content, so gibberish must be traced back to the IDs it received.
- An untrained GPT model with random initial weights cannot yet produce coherent text; model training is covered in the next chapter.