Preparing a Fine-Tuned LLM for Spam Classification

classify_review takes a text together with a fine-tuned model, its tokenizer, a device, and a maximum length, and returns a predicted label.

Programming

A Classifier That Chooses a Label

This article starts from a working experiment rather than from theory. A fine-tuned LLM has already been prepared for spam detection, and a helper function called classify_review uses it to look at a piece of text and return a label. The model is not being asked to write anything. It is being asked to choose one label out of a small set of predefined categories, which here are spam and not spam.

The run you are about to follow sends a single message through classify_review and prints whatever label comes back. Two things are visible to you: the text you pass in and the label printed on the output line. Everything else happens inside the call, so your job is to reason about how the visible input relates to the visible output.

Predict Before the First Run

Before you read the explanation that follows, commit to a guess. The first test message is a short promotional notice that tries to convince the reader they have been chosen for a reward and should act quickly. Think about how you would label a message like that if someone handed it to you without any context. Writing your guess down first turns the run into a test of your own understanding.

What do you think happens?

What label do you think classify_review returns for the short promotional message about a reward and urgent action?

spam
not spam

Reveal answer

Answer:

spam

In this section's first test case, the promotional message is labeled spam. The message asks a stranger to act quickly for a reward, which is the kind of content the model is trained to flag.

The Five Arguments of classify_review

The function takes five arguments. The first is the text to be classified. The second and third are the fine-tuned model and the tokenizer that the model works with. The fourth is the device the computation runs on. The fifth controls the maximum length the input is allowed to have. In the source this length is set from the training dataset rather than typed as a fixed number.

Because the function is called with these arguments, the call site is where you decide which model, which device, and which length limit are used for a given prediction. The function itself does not choose them for you. The shape of such a call is shown below with placeholder names, so you can see which position each argument occupies.

python

label = classify_review(new_text, model, tokenizer, device, max_length)
print(label)

label = classify_review(new_text, model, tokenizer, device, max_length)
print(label)

Output (expected)

spam

Notes on the code

new_text: the message whose label you want
model: the fine-tuned model
tokenizer: the tokenizer that works with that model
device: where the computation runs
max_length: the limit on the input length, taken from the training dataset in the source

How Text Becomes a Label

Read the pipeline below from left to right. The text you supply enters the call together with the model, tokenizer, device, and length setting. Inside the call, the model works on the text and produces a decision about which of the two predefined categories fits. The label that comes back is the only value that leaves the function and reaches the output line.

What happens to the input text, in order, from the moment classify_review receives it until it returns a label, and where the model, tokenizer, and device each come in.

The diagram is a simplified picture. The source describes the model as choosing one label from a small set of predefined categories, and it describes the function as returning a predicted label. It does not walk through the internal computation step by step, so you should treat the middle of the pipeline as a black box whose input and output you can check.

Two Test Cases Side by Side

The section runs two test cases. The first is the promotional message. The second is a short personal note asking whether two people still plan to meet for dinner. Nothing in the second message offers a prize or asks for an immediate action from a stranger. The source reports that the model again makes a correct prediction here, returning the not spam label.

Worked example

Comparing two runs that differ only in the text

Run classify_review twice, once with the promotional message and once with the dinner-plans message. The model, tokenizer, device, and length setting stay the same. What differs, and what does that tell you?

1
Identify the fixed inputs
The model, tokenizer, device, and maximum length are held constant across both calls, so they cannot explain a difference in the printed label.
2
Identify the changing input
The text is the only argument that changes. It is the single variable between the two runs.
3
Compare the outputs
The promotional message produces spam, and the dinner-plans message produces not spam.
4
Draw the conclusion
Because the other inputs were held fixed, the difference in the label comes from the text. The label is an output of the call, and the text is the input you control.

Result

Two different texts passed through the same function produce two different labels, and the text is the lever you can change.

Finding Where a Prediction Can Diverge

When a prediction does not match what you expected, the fastest route to an explanation is to narrow down which link in the chain produced the unexpected label. The chain has only a few links: the text you supplied, the arguments that accompany it, the function call itself, and the label that comes back. A mismatch between your expectation and the printed label tells you that at least one of these links is not what you assumed.

Which steps between input and output could make the returned label differ from your prediction, and how you would tell which step caused it.

The flowchart turns the debugging advice into a procedure. If the label differs from your guess, do not change the model or the device first. Check the exact text that was passed in, confirm the other arguments are the ones you intended, and then rerun while changing only the text. The step whose behavior changes is the step that is responsible for the surprise.

Mistakes Beginners Make with Predictions

Common mistake
Treating the label as something the model writes out
What it looks like
Expecting the function to return a sentence explaining why a message looks like spam.
Why it fails
The model is asked to choose one label from a small set of predefined categories. The output is a label, not an explanation.
Do this instead
Expect a single label, spam or not spam, and reason about it as a classification.
Common mistake
Changing several arguments at once when debugging
What it looks like
Swapping the text and the length setting in the same rerun, then trying to explain the new label.
Why it fails
When more than one input changes, a different label cannot be attributed to any one of them.
Do this instead
Hold the model, tokenizer, device, and length setting constant, and vary only the text.
Common mistake
Assuming the length setting is a fixed number you typed
What it looks like
Believing max_length is always the same value no matter which dataset was used.
Why it fails
In the source, the length setting is taken from the training dataset rather than typed as a fixed number.
Do this instead
Check where the value comes from at the call site, because it is part of the input that shapes the prediction.
Common mistake
Skipping the prediction step and reading the output line first, which removes the chance to test your understanding

Predict and Check a New Message

EASY

Choose a short message of your own that reads like a reward offer with an urgent deadline, and a second message that reads like a plain personal note. Before running classify_review on either one, write down the label you expect for each. Then run both with the same model, tokenizer, device, and length setting, and compare your predictions with the printed labels. If a prediction is wrong, use the divergence flowchart to decide which link to check first.

Hints

Your two messages should differ only in their text, so the other four arguments stay fixed.
Ask first whether the message asks a stranger for something urgent, the way the promotional message in the section does.
If your guess and the printed label disagree, check the exact string you passed in before changing anything else.

Your attempt (not saved)

Key Takeaways

classify_review takes five arguments: the text, the fine-tuned model, its tokenizer, the device, and a maximum length taken from the training dataset, and it returns a predicted label.
The model chooses one label from a small set of predefined categories, spam or not spam, rather than generating text.
The promotional message is labeled spam and the dinner-plans message is labeled not spam, and the text is the only input that changes between these two runs.
Predicting the label before running the function turns each run into a test of your understanding.
When a prediction surprises you, check the text, the accompanying arguments, the call, and the returned label in that order, holding every argument except the text constant.

Related concepts

More concepts starting with P