Batch Processing in Neural Network Training

When you train a neural network, you make choices that the training process does not make for you. The learning rate, the number of training epochs, and the size of each batch are settings you pick before training starts. These settings are called hyperparameters, and finding good values for them is a central part of building a working model. This article answers a practical question: which data should you use to judge those choices, and why the answer is not always the dataset you expect.

Hyperparameters are settings chosen before training, and a validation dataset is the held-out data used to find the best of them. Keep that pairing in mind, because the rest of the article builds on it.

A very common first instinct goes like this. You already have a test set that the model never sees during training, so you should use it to decide which hyperparameter settings work best. You try a learning rate, check the score on the test set, try another learning rate, check again, and keep whichever one scores highest. This sounds like a reasonable process. Many learners reach this conclusion because they treat every held-out split as one generic kind of data that can be consulted as often as needed.

The instinct has a grain of truth. A validation dataset is in fact similar to a test set, so the two are easy to confuse. The mistake lies in assuming that because they look alike, they can be used in the same way. The difference is in how often each one is allowed to influence your decisions.

A validation dataset is meant to be used many times while tuning. You adjust a model setting, train with it, and check the results on the validation data after each change. Because the validation set absorbs this repeated tuning, it is the dataset that guides your decisions. Each round of the loop keeps the best setting found so far and then tries the next candidate.

The loop in the diagram is the reason the validation set exists. Each pass reads the validation score and uses it to choose the next move. The test set is kept out of that loop entirely, so that at the end it still represents data the tuning process has never been shaped by.

A test set should be used precisely once, at the end, so that it gives an unbiased view of the finished model. Repeatedly consulting it biases the evaluation, and the final reported performance stops being an honest estimate. A validation set, by contrast, is consulted over and over during tuning. Both are held-out data and both look alike, but their roles are different.

Evaluating a model on a validation or test set requires computing its accuracy, which means counting how many predictions match the correct labels. With large datasets this often cannot be done in one step, because memory limits usually mean only a small portion of the dataset can pass through the model at a time. The accuracy computation therefore works with chunks and combines the results.

The approach iterates over a data loader. In each iteration the model receives one chunk, its predictions for that chunk are compared with the true labels, and the number of correct predictions is added to a running total. Each chunk has the same size as the batches used during training, so the same procedure works for datasets of any size.

Here N is the size of the whole dataset. The running total is divided by N only once, after the loop finishes. The final accuracy depends only on that running total and the overall dataset size, which is why the whole dataset never needs to sit in memory at once.