Classification Fine-Tuning Workflow

Once a spam classifier has been fine-tuned, the natural next question is whether it works. It is tempting to answer with a single accuracy number, but a fine-tuned model is judged on several datasets, and each one answers a slightly different question. Measuring accuracy on only one dataset hides information that the other datasets reveal.

The three datasets involved are the training set, the validation set, and the test set. The training set shows how well the model fits the examples it learned from. The validation set was used while development decisions were being made. The test set is held back so that it can serve as a fresh check on the finished model. Each role is different, and so each accuracy number must be read in light of the role of the data it came from.

The final evaluation uses one accuracy function applied to three data loaders: the training loader, the validation loader, and the validation-stage loader's counterpart, the test loader. Accuracy is computed for each loader and printed as a percentage with two decimal places. The source notes that this evaluation is run across the entire dataset, which is why no eval_iter limit is defined. Limiting the number of evaluation batches is a different choice, and it is not used in this workflow.

The training set measures fit to examples the model has already seen. The validation set measures performance on data that informed development decisions, so its score can be shaped by those decisions. The test set measures performance on data that played no part in making decisions. Because these three roles differ, the three numbers must be measured separately; blending them into one figure would erase the distinction the workflow is designed to preserve.

Development is an iterative process. Settings are adjusted and the model is compared against the validation set to see which choices work best. Over many rounds those choices are shaped to the validation data, so validation scores can drift upward in a way the untouched test set does not reflect. The test set was not used to make decisions, so its score is a more neutral estimate of how the model will behave on messages it has never seen.

The gap is a difference between two accuracy values, and it is interpreted through the word significantly. Overfitting is indicated when training accuracy is significantly higher than test accuracy. A slight gap suggests minimal overfitting. The reported results show a gap of 1.54 points, and the source describes the training and test accuracy as almost identical, which is why that gap is treated as small.

The pattern worth investigating is a large difference. The source gives a concrete picture of it: a training score in the high nineties paired with a test score far below it. That shape says the model fits its training examples much better than it handles new ones. The diagram below contrasts that pattern with the nearly identical scores of the reported run.

The expectation is real, and the reported results match it: validation accuracy is higher than test accuracy. The error comes from treating the expectation as a rule. Whether validation sits above or below test on a given run depends on what happened during development. A validation score that comes out lower than the test score is therefore not automatically a failure; it simply means the tuning effect was weaker in that run. Read the ordering as evidence to interpret, not as a check that must pass.

When the gap between validation and test accuracy is larger than you would like, the source names two hyperparameters that can potentially reduce it: the dropout rate, set through drop_rate, and the weight_decay parameter in the optimizer configuration. Raising the dropout rate or the weight decay is the suggested adjustment. Each change should be re-measured, because the effect is not guaranteed and the new run will produce its own three accuracy values.

After evaluation, the model is put to use. The classify_review function takes a piece of text and returns the name of its class. Its signature shows what it needs: the text, the fine-tuned model, the tokenizer, the device, an optional max_length, and a pad_token_id that defaults to 50256.

The function follows the same preprocessing ideas used for the SpamDataset earlier in the workflow. The text is turned into token IDs, the model then predicts an integer class label, and the function returns the matching class name. Pay attention to the order: raw text comes first, token IDs come second, and a label is produced only after the model has seen those IDs.

The classifier's output is a name, not a number. The integer label is a stepping stone, and the function's last job is to map that label back to a readable class. This layering matters for debugging. A wrong answer can come from several places: the text might be prepared differently from training, the truncation might cut off content that mattered, or the model might predict the wrong integer. Knowing the sequence of stages makes it easier to locate where a mismatch began.