Encoding Text into Token IDs

Once text has been converted into encoded token IDs, the next step is to hand those IDs to the model and see what comes back. A reasonable first guess is that a model asked whether a message is spam would answer with a yes or a no. It does not. The model returns a tensor full of numbers. This article is about understanding how that tensor is organized and how to pick out the part that matters for a classification task, such as deciding whether a message is 'spam' or 'not spam'.

The program in this lesson runs the model on a small encoded input and prints both the output tensor and its shape. The shape is the key to everything else in this section. Read what the printed shape reports before reading any explanation of it, because the shape tells you how the numbers are arranged.

In the lesson's example the output tensor has three dimensions, and the reported shape is 1 x 4 x 2. Each dimension answers a different question. The first dimension has size 1 and holds the batch of inputs. In this example there is a single input, so the batch holds one item. The second dimension has size 4, and each entry along it corresponds to one input token. The third dimension has size 2, and it describes the embedding dimension, which is the number of values the model produces for each token.

The number of rows matches the number of input tokens, which is four in this case. Every row is one output vector, and that vector is what the model has produced to represent one token position. The rows do not represent words or labels yet. They are intermediate numeric descriptions of each position in the input. Each row contains exactly two numbers, which is why the third dimension has size 2.

Fine-tuning for classification does not require the model to make a decision for all four output rows. The section focuses on a single output token, specifically the last row, which corresponds to the last input token. The model is being adjusted so that its output becomes a class label, a decision between 'spam' and 'not spam'. Only one decision is needed per input, so the model does not have to give a separate verdict for every token. The final token's row is chosen as the single point where the classification will be read from.

A useful way to think about this is that the last row is the only place a single decision is read from, so the rest of the output is not wasted but simply is not the part the classifier reads. The source does not explain in detail how information from earlier tokens reaches the last row, so this article does not claim more than that. What it does establish is the goal: one row per input, taken from the final token position.

To pull out the last token's row, the section uses a slice that reads from the output tensor. The slice is outputs[:, -1, :]. Read it from left to right, one dimension at a time. The first position is a colon, which keeps the batch dimension whole. The second position is -1, which reduces the token dimension to its last position. The third position is a colon again, which keeps the embedding dimension whole.

The four rows are indexed by token position, and the last one is the row for the last input token. The source highlights this row as Row 3 in its numbering, which is the fourth row when counting from zero. Because -1 means the last position along an axis, the slice picks that final row without you needing to know how many tokens there are.

The starting point is a short text prompt, the string "Hello, I am". Before the model can use it, the text has to pass through two conversions, first into token IDs and then into a tensor. The first conversion is performed by tokenizer.encode(input_text). Its input is a string and its output is a list of integers. The integers are the representation the rest of the pipeline works with.

The text itself is not passed to the tensor step. Only the list of IDs is. Each piece of the input text corresponds to a specific integer in the encoded list, and the position of that integer in the list matches the position of the token in the input. The example in this lesson encodes its prompt into four token IDs, which is why the model output has four rows.

Each stage consumes the output of the previous one, so a problem at any stage propagates to every later stage. Checking the printed encoded list and the tensor shape separates encoding problems from tensor-preparation problems. If the list of IDs looks wrong, the trouble is in encoding. If the list looks right but the shape is wrong, the trouble is in tensor preparation.

The expression torch.tensor(encoded) converts the Python list of token IDs into a PyTorch tensor. The values inside the tensor do not change during this step. What changes is the container: a plain Python list becomes a tensor that the model can operate on. The code below uses a short list of illustrative IDs, not the real IDs for the prompt, to show the shape reported at each stage.

The method call .unsqueeze(0) adds a new dimension at index 0. This is the batch dimension. The model expects its input to carry a batch dimension, so this step is required even though the token values themselves do not change. Only the final stage of the pipeline adds the batch dimension, which is why its effect shows up in the shape printed at the end of the trace.

Most shape problems in this pipeline come from a small set of errors. The items below show each mistake with the correction the source's reasoning supports.