Pretrained Model Limitations

Prompting a pretrained model with an instruction is a quick test of whether it already performs the task, and no weights are changed to run it.

Programming

A Cheap Test Before Any Training

Before you spend effort fine-tuning a model, it is worth checking whether the model already does the job you want. A pretrained model has been trained on a large amount of text, and it may already be able to handle a task if you simply describe that task in plain language. The source frames this as a cheap experiment: you write a natural-language instruction, hand it to the model, and look at what comes back.

The most important property of this test is that no weights are changed. The model is used exactly as it was after pretraining. The only thing you vary is the input text. This makes the test quick, and it means that whatever you observe reflects the model's current state rather than the result of some new training run.

Running the Spam Prompt

The source uses a concrete question: does the model already classify spam messages if we prompt it with instructions? The test prompt asks the model to decide whether a given text is spam and to answer with a single word, yes or no. The text under test is a message announcing that the reader is a winner and has been specially selected for a cash prize and an award.

A model that had learned to follow instructions should answer that question. The run uses the same generation routine the model already has, applied to the token IDs of this prompt. So the only thing that differs from ordinary generation is the input text. Nothing about the code path is special for this test.

What do you think happens?

Before the trace runs, what would count as a successful answer to this prompt?

Reveal answer

Answer:

A short single-word response of yes or no that correctly addresses whether the message is spam.

The prompt asks for a single word, yes or no, so a successful answer is one that follows that format and responds to the question about spam.

Reading the Struggling Output

The important step comes after the trace. The output shows that the model is struggling to follow instructions. That is not a bug in the code, and it is not a sign that the spam question cannot be answered. The source explains the cause directly: the model has only undergone pretraining, and it has not yet had instruction fine-tuning.

The reason is about what pretraining taught. The model learned to continue text. When it receives the prompt, it treats the prompt as text to continue rather than as a request to satisfy. A sentence that asks a question is, to a text-continuing model, just more text, and the most natural continuation may be more prose rather than a one-word label.

The source is careful about what this evidence means. The test does not prove that the model can never classify spam. It shows that the model, in its current state, is not a classifier that can be used by asking it a question. Those are two very different claims, and a good reading of the output keeps them separate.

What the Output Does and Does Not Tell You

Worked example

Interpreting a prompt test result

You prompt a pretrained model with a yes-or-no spam question and it returns a long, rambling continuation instead of a single word. What conclusion is justified, and what conclusion is not?

1
Identify what was asked
The prompt requested a single-word yes or no answer about whether a message is spam.
2
Compare the output to the request
The output does not follow the requested format, so the model is not yet following this instruction reliably.
3
Locate the likely cause
The model has only been pretrained and has not had instruction fine-tuning, so this is the expected baseline behavior rather than a code fault.
4
State the justified conclusion
The model, in its current state, cannot be used as a classifier by prompting alone.
5
Avoid the overreach
The result does not show that spam classification is impossible for this model, and it does not show that a code defect caused the output.

Result

The output justifies the claim that the model is not yet a usable instruction-following classifier. It does not justify the claim that the task is impossible or that the code is broken.

Swapping the Output Layer

The source's answer to a struggling model is to prepare it for classification fine-tuning. The preparation is a structural change, not a prompt change. The original output layer maps the model's hidden representation to a vocabulary of 50,257 possible tokens. That layer lets the model choose its next word from the full vocabulary, and it is the reason the model keeps generating text.

The change replaces that layer with a smaller output layer. The new layer maps the same hidden representation to just two classes: 0 for not spam and 1 for spam. The source is explicit that everything else stays the same. The body of the network, which produced the hidden representation, is carried over, and only the final mapping changes.

What part of the network is swapped out, and how does a head over the whole vocabulary become a head with just two outputs? The model body is kept; only the output layer changes.

Read the diagram from left to right in your head: the body on the left is the same body in both versions, and only the output stage changes. The new output layer is smaller because it has only two outputs instead of one per vocabulary token. The number of outputs is what turns a text generator into a two-class classifier.

What do you think happens?

After the swap, how many outputs does the new output layer have?

Reveal answer

Answer:

Two outputs, one for each class: 0 for not spam and 1 for spam.

The original layer had one output per vocabulary token, 50,257 in total. The replacement maps the same hidden representation to just two classes.

Mistakes When Reading the Test

Common mistake
Treating a failed prompt as proof the task is impossible
What it looks like
The model rambles instead of answering yes or no, so the learner concludes spam detection cannot be done with this model.
Why it fails
The test only shows the model's current behavior. The source attributes the result to pretraining without instruction fine-tuning.
Do this instead
State the narrower conclusion: the model is not yet a classifier usable by prompting alone.
Common mistake
Assuming the output is a code bug
What it looks like
The learner rechecks the generation loop repeatedly, looking for an error in the code path.
Why it fails
The source states that the poor result is not a bug in the code, and the same generation routine is used for ordinary text.
Do this instead
Look at what the model was trained to do before suspecting the code.
Common mistake
Changing weights during the prompt test
What it looks like
The learner starts training the model to fix the answer before running the prompt.
Why it fails
The prompt test is defined as a test that changes no weights. Training is a separate step that comes after the baseline.
Do this instead
Run the prompt first with the model unchanged, then decide whether fine-tuning is needed.
Common mistake
Thinking the output layer change alters the whole model
What it looks like
The learner assumes the entire network must be rebuilt to get a two-class classifier.
Why it fails
The source is explicit that the rest of the model stays the same and only the output layer is replaced.
Do this instead
Keep the body of the network and replace only the final mapping to two classes.

Practice: Reading a Prompt Result

MEDIUM

A colleague prompts a pretrained model with an instruction to label a message as spam or not spam, and the model continues the message as if it were a story. Write two sentences: one stating what this result does justify, and one stating what it does not justify.

Hints

Ask whether the model has undergone instruction fine-tuning.
Separate the claim about the model's current state from the claim about what is possible for the task.

Your attempt (not saved)