Preparing Text for LLM Training

A large language model learns from large amounts of text, but text in its original written form is not yet ready for training. Preparing text for large language model training is a required step before the model can learn from it. Training does not simply consume text as it was originally written.

Text preparation means transforming text so that a language model can use it during training. This article focuses on one part of that preparation: tokenization, the process of splitting text into smaller units called tokens. Those tokens are the units the rest of the process works with, so the way text is split affects everything that follows.

As you read, keep one question in mind: when a sentence is split into tokens, what does the result look like, and how does the splitting decision change it? A tokenizer makes that decision for every piece of text it receives.

The source describes tokenization as splitting text into two kinds of token: word tokens, which are whole words, and subword tokens, which are pieces smaller than a word. A word-level split can keep a word as one token, while a subword split can produce several smaller tokens for the same word.

The text used throughout this part is 'The Verdict', a short story by Edith Wharton that was already used in chapter 2. It is a small text. Because the story is in the public domain, there is no question about usage rights. Because it is small, the code runs on a standard laptop in a matter of minutes, even without a high-end GPU.

Size is the trade-off. A dataset this small cannot train a model that produces useful text on its own. The purpose here is education, so the code runs in minutes instead of weeks. Later in the workflow, pretrained weights from OpenAI are loaded into the model, which is how the small text is made useful in practice.

Dataset choice is therefore a matter of goal and constraints. A small public domain text answers the questions of licensing and speed, which matters when the aim is to learn the mechanics. A larger corpus is the route toward more extensive training, at a much higher computational cost.

The first code step reads the whole story into a single string. The file is opened in read mode with UTF-8 encoding, and file.read() returns its full contents. Once the text is in memory, it can be measured in two different units.

The number of characters comes from len(text_data). The number of tokens comes from passing the same text through the tokenizer and measuring the result. The two counts answer different questions. The character count describes the raw text, while the token count describes how the model will actually read it.

The source notes that 5,145 tokens may look too small to train an LLM, and that is accepted because the aim is quick educational runs. The token count is the number to watch when judging what the model actually receives.

Next the text is divided into a training part and a validation part. The validation part is used for model evaluation during training, so it must be kept separate from the training part. The split is set by a train_ratio of 0.90, which means 90 percent of the characters go to training and the remaining 10 percent go to validation. The split index computed from train_ratio divides the text into these two parts, and the two parts are never mixed.

A split subset is still plain text, and the model needs fixed-size tensors. The source describes the preparation as a chain of operations. The subset is tokenized, the token sequence is divided into chunks of a user-specified length, the chunks are shuffled, and the shuffled chunks are organized into batches. The batch size sets how many chunks are processed together.

The loaders are built with the create_dataloader_v1 function from chapter 2. Both calls use the same batch size, max_length, and stride. The settings that differ are drop_last and shuffle, and those differences encode the training-versus-validation distinction. The code also fixes a random seed so the shuffling is repeatable.

The source suggests an optional check. Iterate through each loader and print the shape of every input and target tensor. Each loader yields a pair, an input x and a target y, so the loop unpacks two values per batch.

Based on the printed output, the training loader yields nine batches, each with two samples of 256 tokens. That means each training input batch has shape 2 by 256. The validation loader yields one batch of two input examples. This matches what the split and the batch settings predict, since only about a tenth of the text goes to validation.