Model Performance Evaluation

Once a spam classifier has been fine-tuned, the natural next question is whether it works. A single accuracy number is tempting to report, but a fine-tuned model is judged on several datasets, and each one answers a slightly different question. Reporting one number hides which question was answered, so the evaluation is split across datasets that play different roles.

Keep this idea in mind as you read: the same model can look different depending on which data you measure it on. The rest of this article shows why that is expected, how to read the differences, and how the finished model is then used to classify a new message.

The training set shows how well the model fits the examples it learned from. The validation set was used while development decisions were being made, so it is the data that choices were compared against. The test set is held back so that it can serve as a fresh check on the finished model. Each dataset therefore answers a different question about the same model.

The three sets are not interchangeable. A high score on the training set alone tells you the model has fitted its learning examples, not that it handles new messages. A score on the validation set is shaped by the decisions made while looking at it. Only the test set gives a check that was never part of those decisions, which is why it is held back.

Development is an iterative process. Settings are adjusted, and the model is compared against the validation set to see which choices work best. The timeline below shows when each dataset enters the work. The training set is the material the model learns from, the validation set is consulted repeatedly while choices are compared, and the test set is reserved for the end.

Because decisions are repeatedly shaped to the validation data, validation scores can drift upward in a way the untouched test set does not reflect. The test set was not used to make decisions, so its score is described as a more neutral estimate of how the model will behave on messages it has never seen.

In the reported results, training and test accuracy sit close together. The source describes them as almost identical, and the slight gap between them, 1.54 points, suggests minimal overfitting of the training data. The pattern to investigate is different: a large difference, such as a training score in the high nineties with a test score far below it.

The expected pattern comes from how development works. Choices are compared against the validation set, so they are gradually shaped to it, and the validation score benefits from that shaping. The test set receives no such benefit, so its score tends to sit a little lower.

The ordering is still only a tendency. The reported results follow the expectation, with validation above test. But whether validation sits above or below test on a given run depends on what happened during development. A validation score that comes out lower than the test score is not automatically a failure; it simply means the tuning effect was weaker in that run. Treat the ordering as evidence to interpret, not as a check that must pass.

When the gap between validation and test accuracy is larger than you want, the source notes it is common and could potentially be reduced by changing the model's settings. Two hyperparameters are named: the dropout rate, set through drop_rate, and the weight_decay parameter in the optimizer configuration. Raising the dropout rate or the weight decay is the suggested adjustment. Each change should be re-measured, because the source only says it can potentially reduce the gap.

After evaluation, the model is put to use. The classify_review function takes a piece of text and returns the name of its class. Its signature shows what it needs: the text, the fine-tuned model, the tokenizer, the device, an optional max_length, and a pad_token_id that defaults to 50256.

The function follows the same preprocessing ideas used for the SpamDataset earlier in the workflow. The text is turned into token IDs, the model then predicts an integer class label, and the function returns the matching class name. The diagram below follows one message through those stages in order, including the truncation check on sequences longer than the supported context length.

Notice that the classifier's output is a name, not a number. The integer label produced by the model is a stepping stone, and the function's last job is to map that label back to a readable class. Because the answer passes through several stages, a wrong result can come from more than one place: the text might be prepared differently from training, the truncation might cut off content that mattered, or the model might predict the wrong integer.

The evaluation in this article sits inside the larger workflow. Classification fine-tuning is organized in three stages: preparing the dataset, setting up the model, and fine-tuning the model and then using it. Final evaluation and classification of new messages both belong to that last stage, so the process does not end when training stops. Step 9 is the evaluation, and step 10 is the use of the finished model on new messages. If a final accuracy looks wrong, you can trace back through the steps that produced it.