Using a Fine-Tuned Model for Classification

Once a spam classifier has been fine-tuned, the natural next question is whether it works. A single accuracy number is tempting to report, but a fine-tuned model is judged on several datasets, and each one answers a slightly different question. The training set shows how well the model fits the examples it learned from. The validation set was used while development decisions were being made. The test set is held back so that it can serve as a fresh check on the finished model.

Each of the three datasets plays a different role, and understanding those roles is what makes the evaluation meaningful. The training data is the material the model learned from, so its accuracy describes how well the model fits examples it has already seen. The validation data is used during development: settings are adjusted and the model is compared against the validation set to see which choices work best. The test data is kept out of those decisions and is used at the end as a check on the finished model.

The evaluation is run in a deliberate order. Accuracy is computed on the training, validation, and test loaders, and each value is printed as a percentage with two decimal places. The source notes that this evaluation runs across the entire dataset, which is why the code does not define an eval_iter value. Limiting the number of evaluation batches is a different choice, and it is not used here.

Overfitting is indicated when training accuracy is significantly higher than test accuracy. The word significantly matters. A small difference between the two numbers is not evidence of overfitting. In the reported results, training and test accuracy sit close together, and the slight gap of 1.54 points between them suggests minimal overfitting of the training data.

Development is an iterative process. Over many rounds of adjusting settings and comparing against the validation set, those choices are shaped to the validation data, so validation scores can drift upward in a way that the untouched test set does not reflect. The test set was not used to make decisions, so its score is a more neutral estimate of how the model will behave on messages it has never seen. This is why the expected pattern is validation somewhat higher than test.

The expectation is a tendency, not a rule. If a validation score comes out lower than the test score, that is not automatically a failure. It simply means the tuning effect was weaker in that run. The ordering should be read as evidence to interpret.

If the gap between validation and test accuracy is larger than you want, the source names two hyperparameters to examine: the dropout rate, set through drop_rate, and the weight_decay parameter in the optimizer configuration. Raising the dropout rate or the weight decay is the suggested adjustment, and the source notes that it could potentially reduce that gap. Each change should be re-measured, because the effect is not guaranteed.

After evaluation, the model is put to use. The classify_review function takes a piece of text and returns the name of its class. Its parameters are the text, the fine-tuned model, the tokenizer, the device, an optional max_length, and a pad_token_id that defaults to 50256.

The function follows the same preprocessing ideas used for the SpamDataset earlier in the workflow. The text is turned into token IDs, the model then predicts an integer class label, and the function returns the matching class name. Each stage hands its result to the next, so a new message moves through the classifier in a fixed sequence.

Notice that the classifier's output is a name, not a number. The integer label produced by the model is a stepping stone, and the function's last job is to map that label back to a readable class. This also means a wrong answer can come from several places: the text might be prepared differently from training, the truncation might cut off content that mattered, or the model might predict the wrong integer. Knowing the sequence of stages makes it easier to locate where a mismatch began.