Preparing Text Data for Training

Before a language model can learn from text, the text has to be converted into a form the training process can use. Earlier steps in this chapter covered how raw text is broken into tokens. This article picks up at that point. You already have a sequence of tokens, and the question is how to turn that sequence into the examples a model actually trains on.

The chain to keep in mind is this: tokenized data goes in, a sliding window slides across it, and input-target pairs come out. The pairs are what the model is trained on. Each pair ties a piece of input text to the target the model should learn to associate with it.

Tokenized data is the input to this process, and input-target pairs are its output. The source applies the idea to tokens specifically. The window does not slide over raw characters. It slides over tokens, because tokens are the units the model already treats as discrete pieces. That is why tokenization comes first.

The sliding window approach moves a window across the token sequence to produce these pairs. A single token sequence can yield several pairs, because consecutive window positions overlap. The next section traces exactly how that happens.

Consider a small illustrative token sequence of eight tokens: the, story, was, short, and, clear, to, read. Use a window that holds four tokens and moves forward by one position at each step. The input for each pair is the four tokens inside the window. The target is the four tokens one position later, so that each target position is the token that follows the matching input position.

Step 0 pairs the input the, story, was, short with the target story, was, short, and. Step 1 moves the window one token forward. The input becomes story, was, short, and, and the target becomes was, short, and, clear. Notice that three of the four input tokens in step 1 were also in step 0. That overlap is why one token sequence can produce several pairs.

The text used throughout this part is The Verdict, a short story by Edith Wharton. It was already used in chapter 2. Choosing it solves two problems at once. Because the story is in the public domain, there is no question about usage rights. Because it is small, the code runs on a standard laptop in a matter of minutes, even without a high-end GPU.

Size is the trade-off. A dataset this small cannot train a model that produces useful text on its own. The purpose here is education, so the code runs in minutes instead of weeks. Later in the workflow, pretrained weights from OpenAI are loaded into the model, which is how the small text is made useful in practice.

For scale, the source points to the 7 billion parameter Llama 2 model, a relatively popular openly available LLM. Training it required 184,320 GPU hours on A100 GPUs and processed 2 trillion tokens. A larger corpus is the route toward more extensive training, at a much higher computational cost.

The first code step reads the whole story into a single string. The file is opened in read mode with UTF-8 encoding, and file.read() returns its full contents. Once the text is in memory, it is measured in two different units. The number of characters comes from len(text_data). The number of tokens comes from passing the same text through the tokenizer and measuring the result.

The two numbers answer different questions. The character count describes the raw text. The token count describes how the model will actually read it. These are different counts, and the token count is the one that describes what the model reads. In the source's run, the text yields 5,145 tokens. That may look too small to train an LLM, and it is accepted here because the aim is quick educational runs.

Next the text is divided into a training part and a validation part. The validation part is used for model evaluation during training, so it must be kept separate from the training part. The split is set by a train_ratio of 0.90, which means 90 percent of the text goes to training and the remaining 10 percent goes to validation. The source describes this split as a split index computed from train_ratio, and the two parts are never mixed.

A split subset is still plain text, and the model needs fixed-size tensors. The source describes the preparation as a chain of operations. The subset is tokenized. The token sequence is divided into chunks of a user-specified length. The chunks are shuffled. The shuffled chunks are organized into batches, where the batch size sets how many chunks are processed together.

The source uses a batch size of 2 because the dataset is small. It notes that training LLMs with batch sizes of 1,024 or larger is not uncommon in practice. Batch size is therefore a resource choice, and the small value here reflects the small dataset rather than a recommended setting for large-scale training.

The loaders are built with the create_dataloader_v1 function from chapter 2. Both calls use the same batch size, max_length, and stride. The settings that differ are drop_last and shuffle, and those differences encode the training-versus-validation distinction. A random seed is also fixed so that the shuffling is repeatable.

Setting drop_last to True on the training loader discards an incomplete final batch, so every training batch has the full size. The validation loader sets it to False, so no validation chunk is silently discarded. Shuffling is switched on for training, where the order of examples should vary, and switched off for validation, where a fixed order keeps evaluation consistent.

The source suggests an optional check. Iterate through each loader and print the shape of every input and target tensor. Each loader yields a pair, an input x and a target y, so the loop unpacks two values per batch. The printed output confirms whether the loaders were built correctly.

In the source's printed output, the training loader yields nine batches, each with two samples of 256 tokens. The validation loader yields one batch of two input examples. This matches what the split and the batch settings predict, since only about a tenth of the text goes to validation.

Dataset choice is a matter of goal and constraints. A small public domain text answers the questions of licensing and speed, which matters when the aim is to learn the mechanics. A larger corpus is the route toward more extensive training, at a much higher computational cost.