Tokenization and Token IDs in Language Models

Suppose you plan to turn a language model into a spam classifier. A natural first question is whether the model can already do the job before you invest any effort in adapting it. The simplest test is to give the model an instruction in plain language, ask it to answer with a fixed set of labels, and then read what it produces. This article uses that test as a way into how a model reads its input as token IDs, what it returns, and what that output tells you about the training the model has received so far.

The summary of the source puts the idea directly: running a pretrained model on a spam instruction and reading the generated text is a direct way to test whether it already follows instructions. Everything you need to answer the question is in that generated text.

The instruction in this practice asks whether a short message is spam and requests an answer of either yes or no. The message is a typical promotional line claiming the reader is a winner and has been specially selected for a cash prize and an award. The model does not read the instruction as letters on a screen. It receives the instruction as a sequence of token IDs, which are integers that stand in for pieces of text. The model then generates a continuation of a fixed length, and that continuation is converted back into readable text.

The conversion steps are the important part to track. Text goes in as token IDs and comes out as token IDs before it is shown to you as text. A token ID is the number the model actually operates on, so the model's behavior is always a behavior over sequences of these integers. The source does not describe how a particular tokenizer splits a sentence, so the next illustration is generic: it only shows that each piece of the prompt is mapped to an integer.

Before you look at what the model actually returns, make a prediction. The prompt is clear, the label set is restricted to yes or no, and the message is an obvious example of spam. If the model were behaving like a classifier that follows instructions, what would you expect the first words of its continuation to be?

The reveal is the core of this practice. Once you have run the test, the output is the evidence, and your job is to compare it against the expectation you formed a moment ago. The point is not that the model produced something unusual. The point is that its output differs from what an instruction-following classifier would produce.

When you read the generated text, look at the structure rather than at its surface meaning. The model does not stop after a single answer. It keeps producing words that mirror the form of the prompt, including another request to decide whether a text is spam, followed by the beginning of a new message. The output therefore looks like a continuation of a document, not a reply addressed to you.

The source gives a direct explanation for the result. The model has only undergone pretraining and has not undergone instruction fine-tuning. Pretraining teaches a model to predict and produce text in general. It does not, by itself, teach the model to treat a particular piece of text as a command and to respond to it in the expected format.

Instruction fine-tuning is the step meant to close this gap. It trains the model on examples of instructions so that its ability to follow those instructions improves. The source frames this as the reason the next step is to prepare the model for classification fine-tuning. The order matters: the model first needs to be able to follow instructions in general, and classification fine-tuning then specializes it for the spam task.

Read the pipeline as a dependency chain. Each stage adds a capability that the next stage assumes is already present. Classification fine-tuning assumes the model can already follow instructions in general, which is exactly what the spam test showed it could not yet do.