Classification Fine-Tuning for Spam Detection
Running a pretrained model on a spam instruction and reading the generated text is a direct way to test whether it already follows instructions.
Asking Before Adapting
Suppose you plan to turn a language model into a spam classifier. Before you invest any effort in adapting the model, a natural first question is whether it can already do the job. The question is worth asking because the answer determines how much work lies ahead. If a model already follows a plain-language instruction and returns the label you asked for, the adaptation effort may be small. If it does not, you need to understand what is missing before you can fix it.
The simplest test is to give the model an instruction in plain language, ask it to answer with a fixed set of labels, and then read what it produces. This section and the next follow that test step by step. You will trace what the model receives, what it generates, and what the generated text tells you about its behavior.
Setting Up the Spam Test
In this test, the instruction asks whether a short message is spam and requests an answer of either yes or no. The message is a typical promotional line claiming the reader is a winner and has been specially selected for a cash prize and an award. Everything the model is given is text: the instruction, the restricted label set, and the message itself.
The model does not read this text as English directly. The instruction is converted into a sequence of token IDs, and the model receives that sequence. It then generates a continuation of a fixed length, and that continuation is converted back into readable text. The generated text is the only evidence you need to judge the result.
What do you think happens?
The prompt is clear, the label set is restricted to yes or no, and the message is an obvious example of spam. If the model were behaving like a classifier that follows instructions, what would you expect the first words of its continuation to be?
Reveal answer
The model did not return a yes or no label. Its continuation followed the pattern of the prompt instead.
A model that follows instructions would answer with one of the requested labels. The observed continuation looked like a continuation of a document, not a reply addressed to the reader.
Reading the Generated Text
When you read the generated text, look at its structure rather than its surface meaning. The model does not stop after a single answer. It keeps producing words that mirror the form of the prompt. That includes another request to decide whether a text is spam, followed by the beginning of a new message. The output therefore looks like the continuation of a document, not a reply to the person who wrote the instruction.
Why Pretraining Alone Is Not Enough
The source gives a direct explanation for the observed behavior. The model has only undergone pretraining and has not undergone instruction fine-tuning. Pretraining teaches a model to predict and produce text in general. It does not, by itself, teach the model to treat a particular piece of text as a command and to respond to it in the expected format.
Two things can look similar but differ in behavior. A pretrained model can contain a great deal of knowledge, including knowledge of what spam looks like, and still continue a prompt as if it were the opening of a document. Knowing about spam and responding to a spam instruction with a label are different abilities. The output in this test shows the second ability is missing, even though the first one is not in question.
A pretrained model's output reflects what it learned to continue, not necessarily what it was asked to do. Following an instruction is a behavior that must be trained for.
How Instruction Fine-Tuning Closes the Gap
Instruction fine-tuning is the step meant to close the gap the output revealed. It trains the model on examples of instructions so that its ability to follow those instructions improves. The purpose is not to add spam knowledge. The purpose is to make the model respond to instructions in general.
The order matters. The model first needs to be able to follow instructions in general. Classification fine-tuning then specializes it for the spam task. If the model cannot yet treat an instruction as a request, specializing it for one particular classification problem is premature, because the more basic behavior is still missing.
Mistakes When Reading the Output
Common mistake Concluding that the model does not know what spam is
Why it failsThe source attributes the failure to missing instruction-following behavior, not to missing knowledge of spam.
Do this insteadDescribe the failure as a failure to follow the instruction and answer with the requested label.
Common mistake Judging the output by whether its words sound sensible
Why it failsThe generated text can be fluent and still fail the task. The key evidence is its structure and whether it answers the instruction.
Do this insteadCheck whether the first words are one of the requested labels and whether the text stops after one answer.
Common mistake Treating the test as proof that the model cannot be adapted
Why it failsThe test reveals a gap. The source describes instruction fine-tuning and then classification fine-tuning as the steps that address it.
Do this insteadUse the result to decide which training stage comes next.
Common mistake Skipping instruction fine-tuning and going straight to classification fine-tuning
Why it failsThe model first needs to follow instructions in general. Classification fine-tuning specializes that ability for spam.
Do this insteadKeep the stage order: pretraining, then instruction fine-tuning, then classification fine-tuning.
Practice: Diagnosing a Continuation
A model receives a prompt asking whether a message is spam and requesting a yes or no answer. Its generated continuation begins by repeating a similar question about a different message, and it then starts writing that message. Using only the evidence in the output, identify whether the model followed the instruction, explain which part of the continuation shows this, and state which training stage the source says should come next.
Hints
- Ask first whether the first words are one of the requested labels.
- Look for whether the continuation stops after one answer or keeps mirroring the prompt's form.
- Recall the order the source gives for the training stages.
Key Takeaways
- A pretrained model can be tested directly by giving it a plain-language classification instruction and reading the text it generates.
- The observed continuation repeated the prompt's pattern instead of returning a yes or no label, which shows the model is struggling to follow instructions.
- The source attributes this to pretraining alone, not to missing knowledge of spam. Pretraining does not by itself teach a model to treat text as a command.
- Instruction fine-tuning closes this gap, and it comes before classification fine-tuning, which then specializes the model for the spam task.