Tokenizers and Encoding Text

Fine-tuning a model on instructions starts with a practical question: how do raw records become something a training loop can request one at a time? This article answers that question using one class, InstructionDataset. Instead of reading the class from top to bottom, you will first predict what it should do, then trace the code, then explain the mechanism behind the result.

Compare your guess with the outcome. If you guessed that the dataset answers requests from stored sequences, you already have the central idea of the class: it stores finished token sequences, and it answers length and index requests from those stored sequences. The sections that follow explain how the constructor creates them and why that work is done ahead of time.

InstructionDataset inherits from torch.utils.data.Dataset and provides three methods with a clear division of labor. The constructor, __init__, receives the raw data and a tokenizer and performs the preparation work. The __getitem__ method takes an index and returns one stored item. The __len__ method reports how many items the dataset holds.

Notice that the constructor does the heavy lifting and the other two methods are light. __len__ and __getitem__ do not rebuild anything. They read from data that already exists. This is the key to understanding why the class behaves as it does. Once you can name which list each method reads from, you can predict most of its output.

The constructor stores the raw data, creates an empty list called encoded_texts, and then loops over every entry. For each entry, it builds the instruction portion with a helper named format_input. The helper is called in the listing but defined separately from the class code shown there. The constructor then adds a response section that begins with a Response header followed by the entry's output text. The two parts are joined into one full text string.

Trace one entry through this loop. The raw record goes into format_input, which produces the instruction portion. A Response header and the output text are added next, and the two parts are combined into a single full text. That full text is still just a string at this point. The last step of the loop is the one that matters most for what follows.

The last step of each loop iteration is the important one. The full text is passed to tokenizer.encode, and the resulting token sequence is appended to self.encoded_texts. The text is therefore converted to tokens during construction, before any training begins. The source calls this conversion pretokenization: tokenizing texts using a tokenizer before training.

The source states that pretokenization accelerates training. The reasoning follows from where the tokenizing work happens. Because every entry is encoded once inside the constructor, the training loop later only has to look up a finished sequence when it asks for an item. The tokenizer does not need to run again on the same text each time that example is requested.

In this lifecycle, __init__ runs once and is the only method that calls the tokenizer. __len__ and __getitem__ run later, and each request reads from what was already stored. A call like dataset[i] therefore returns a stored token sequence rather than raw text. Each repeated request is cheap because no encoding happens during it.

Training usually processes several examples at once. The source explains that collecting multiple training examples in a batch accelerates training. Batching has a consequence, though: examples in the same batch need to line up, so all inputs are padded to a similar length. The source describes this padding as a necessity for batch collection rather than an optional tidy-up.

The example above uses made-up token labels to show the shape of the change. The principle comes from the source: padding brings inputs to a similar length so they can be grouped. The padding token the source specifies is <|endoftext|>, which is used in the same way as in classification fine-tuning.

Suppose a teammate reports that an InstructionDataset built from ten records returns items that look like raw instruction text when they are indexed, and the length is correct. Use what you have learned to decide where to look.