Batching Text Data for Training

Text does not arrive in neat, equal-sized pieces. One message might contain four tokens, another might contain nineteen, and a third might be long enough to push against the limit that a model can handle. Training pipelines, however, usually want to hand the model a group of examples at once, and that group is much easier to organize when every example has the same length. This article is about the step that makes that possible: padding each input out to the maximum number of tokens the model supports.

Every model has a maximum number of tokens it supports. That limit works as a target length for every input. Padding extends each input that is shorter than the target until it reaches the maximum, so all inputs end up with a uniform width. The target is not chosen per message. It is set once by the model's limit, and every message is measured against it.

Consider a model whose limit is 8 tokens. An input of 3 tokens needs 5 extra positions. An input of 5 tokens needs 3 extra positions. An input of 8 tokens already fills the target and needs none. After padding, all three inputs occupy the same 8 slots, even though they started with different numbers of real tokens. The diagram below shows the three rows lined up against the same width.

The important point is that the target length belongs to the model, not to the message. A message does not get to decide how wide it is. The width is fixed by the maximum token count, and every input is brought to it.

Padding changes a short input position by position. The real tokens keep their original order at the start of the sequence, and the added padding positions fill the remaining slots up to the target length. The padding positions carry no message content. They exist only so that the input has the same width as every other input in the batch.

Once inputs have been brought to a uniform length, a data loader groups them into batches. The batch size controls how many training examples the loader hands over at each step. With a batch size of 8, each batch delivers eight examples of length 120, and each of those examples comes with its own class label. The loader therefore delivers two aligned pieces of information together: the padded text and the label it belongs to.

The mechanics of building the loader are much like those used in earlier text-data work. The difference is in the targets. In next-token prediction, the target for each position is the token that follows it. In text classification, the target is a single class label that describes the whole message. The model is not asked to guess the next word. It is asked to assign the message to a class.

The source describes three loaders, one each for the training, validation, and test sets. Each loader reads text messages and labels in batches of size 8. Every split gets its own loader, so the same batching rules apply to all three while the examples they draw from stay separate. Keeping the splits apart is what allows the validation and test sets to measure performance on data the model was not trained on.

Putting the pieces together gives a sequence of steps that a set of raw text examples passes through before the model sees any of them. Each step depends on the one before it: padding needs a measured length, and the loader needs uniform inputs to group.

The source is explicit that the effect of padding on predictive performance should be tested, not assumed. The sound approach is to keep everything else fixed, run the same training procedure once with padded inputs and once without, and compare the resulting predictive performance. The comparison tells you what padding did for this model and this dataset, which is more useful than a general claim in either direction.