Padding and Batching Training Examples

Training usually processes several examples at once, and the source explains why this is worth doing: collecting multiple training examples in a batch accelerates training.

Programming

Interactive lab

Try it: Classes, Instances and Attributes

How a class creates instances with __init__ and self, why an attribute read checks the instance's __dict__ before the class, why assigning through an instance never changes the class, how a class-level list is shared by every instance, and why obj.method(x) is Class.method(obj, x).

How it works

Dog(...) makes a new, empty instance and calls Dog.__init__(new_object, ...); self.name = ... stores into that object's __dict__.
Reading obj.attr looks in obj.__dict__ first, then in the class.
Assigning obj.attr = value always writes obj.__dict__, so it shadows a class attribute for that one object only.
Mutating a class-level list (self.tricks.append) changes the one list every instance shares.
obj.method is found on the class and bound to obj, so obj.method(x) runs Class.method(obj, x).

Default run (30 steps): Ready: class Dog and the statements you picked. Watch where each attribute is found and where each assignment writes. … Finished. Printed: ['sit'] / Wolf / Canis familiaris / Fido is 5.

Simplified: The lab's own small Python interpreter runs a fixed, hand-written script (you choose the variant and the values, never the code); nothing is executed as real Python. Every script was run in CPython 3.12 with the same inputs and the lab reproduces its output and error messages. Object numbers are illustrative, not real id() values.

Educational simulation

Loading the simulation…

Predicting What the Dataset Class Does

Fine-tuning a model on instructions starts with a practical question: how do raw records become something a training loop can request one at a time? This article answers that question with a single class, InstructionDataset. Rather than reading the class from top to bottom, make a prediction first, then trace what the code does, then explain the mechanism that produced the result.

What do you think happens?

InstructionDataset stores finished token sequences built from the records. When you ask it for its length, what number should come back?

The number of tokens in the longest instruction
The number of records supplied when the dataset was created
The number of characters in the full text of the first record

Reveal answer

Answer:

The number of records supplied when the dataset was created.

In this class, __len__ returns the length of the original data list, so the count matches the number of records given at creation time. The length question is answered from the raw list, not from the encoded sequences.

If you chose the second option, you already have the central idea of the class: it stores finished token sequences, and it answers length and index requests from those stored sequences. The sections that follow explain how the constructor creates those sequences and why that work is done ahead of time.

The Three Methods and Their Roles

InstructionDataset inherits from torch.utils.data.Dataset and provides three methods with a clear division of labor. The constructor, __init__, receives the raw data and a tokenizer and does the preparation work. The __getitem__ method takes an index and returns one stored item. The __len__ method reports how many items the dataset holds.

What is stored once in the constructor, and what is computed or returned each time an index is requested?

Notice which list each method reads from. The length comes from the original data list, while items returned by __getitem__ come from encoded_texts, which the constructor filled earlier. Keeping that distinction in mind makes the rest of the class easier to reason about.

Building Each Entry in the Constructor

The constructor stores the raw data, creates an empty list called encoded_texts, and then loops over every entry. For each entry, it builds the instruction portion with a helper named format_input. That helper is called in the listing but defined separately. The constructor then adds a response section that begins with a Response header followed by the entry's output text. The two parts are joined into one full text string.

python

class InstructionDataset(Dataset):
    def __init__(self, data, tokenizer):
        self.data = data
        self.encoded_texts = []
        for entry in data:
            instruction_part = format_input(entry)
            response_part = f"\n\n### Response:\n{entry['output']}"
            full_text = instruction_part + response_part
            self.encoded_texts.append(tokenizer.encode(full_text))

    def __getitem__(self, index):
        return self.encoded_texts[index]

    def __len__(self):
        return len(self.data)

class InstructionDataset(Dataset):
    def __init__(self, data, tokenizer):
        self.data = data
        self.encoded_texts = []
        for entry in data:
            instruction_part = format_input(entry)
            response_part = f"\n\n### Response:\n{entry['output']}"
            full_text = instruction_part + response_part
            self.encoded_texts.append(tokenizer.encode(full_text))

    def __getitem__(self, index):
        return self.encoded_texts[index]

    def __len__(self):
        return len(self.data)

Notes on the code

The constructor keeps the raw data list so __len__ can report its size
encoded_texts starts empty and is filled during construction
format_input is called here but defined separately
The full text is encoded and the token sequence is appended
__getitem__ returns a stored sequence by index
__len__ returns the length of the original data list

Tracing encoded_texts Through One Record

The last step of each loop iteration is the important one. The full text is passed to tokenizer.encode, and the resulting token sequence is appended to self.encoded_texts. The text is therefore converted to tokens during construction, before any training begins. The source calls this conversion pretokenization: tokenizing texts with a tokenizer before training.

Worked example

Following one record from raw text to a stored sequence

Suppose the constructor receives a record whose full text, once joined, is encoded by the tokenizer into a short sequence. What does encoded_texts contain afterwards, and what does __getitem__ return for that index?

1
Start
encoded_texts is an empty list before the loop begins.
2
Build the text
format_input builds the instruction portion, then the Response header and the output text are added and joined into one full text string.
3
Encode once
tokenizer.encode converts the full text into a token sequence. Illustratively, suppose the result is the made-up IDs 12, 7, 33, 90.
4
Store
That sequence is appended to encoded_texts. The list now holds one finished entry at index 0.
5
Request an item
Calling __getitem__ with index 0 returns self.encoded_texts[0], which is the stored sequence. No tokenizer call happens at this point.

Result

encoded_texts[0] holds the made-up sequence 12, 7, 33, 90. Requesting index 0 later returns that stored sequence without running the tokenizer again.

Why Pretokenizing Speeds Up Training

The source states that pretokenization accelerates training, and the reasoning follows from where the tokenizing work happens. Because every entry is encoded once inside the constructor, the training loop only has to look up a finished sequence when it asks for an item. The tokenizer does not need to run again on the same text each time that example is requested.

What work is moved out of the training loop by pretokenizing, and how does the per-step work differ between the two designs?

The benefit is about where the cost is paid. The tokenizing cost is paid once per record, during construction. The per-step cost in the training loop shrinks to a lookup of a sequence that already exists.

Padding Examples to a Common Length

Training usually processes several examples at once, and the source explains why this is worth doing: collecting multiple training examples in a batch accelerates training. Batching has a consequence. Examples in the same batch need to line up, so all inputs are padded to a similar length. The source describes this padding as a necessity for batch collection rather than an optional tidy-up.

How do sequences of different lengths end up the same length, and where do the extra padding positions go in each row?

In the illustration, Example B is short, so its row is extended with padding positions until it matches the length of Example A. The token IDs 12, 7, 33, and 90 and the IDs 5 and 21 are made up to show the shape of the change. The principle comes from the source: padding brings inputs to a similar length so they can be grouped.

How do separate dataset items move through a batch and become one padded group?

Mistakes That Hide the Real Source

Common mistake
Assuming __len__ counts encoded tokens
Why it fails
In this class, __len__ returns the length of the original data list, so it counts records, not tokens.
Do this instead
Read __len__ as the number of records supplied at creation time.
Common mistake
Expecting the tokenizer to run inside __getitem__
Why it fails
Each entry is encoded once inside the constructor, so __getitem__ returns a stored sequence.
Do this instead
Look for tokenizer.encode in __init__ when you want to know where the encoding happens.
Common mistake
Thinking padding is optional
Why it fails
The source describes padding as a necessity for batch collection, because examples in a batch need to line up.
Do this instead
Pad inputs to a similar length before grouping them into a batch.
Common mistake
Treating the padding token as real content
Why it fails
The padding token fills positions added only to match lengths.
Do this instead
Recognize <|endoftext|> as the filler the source specifies for padding.

Locating an Unexpected Result

Suppose a learner reports that asking for the item at index 2 returns the text of the instruction rather than a sequence of numbers. Using only the reasoning from this article, decide where to look first.

MEDIUM

Name the list that __getitem__ reads from, then name the step in the constructor that fills that list. Which line would you check first, and why?

Hints

Ask which list __getitem__ returns from.
The stored items are appended during the loop in the constructor.
The last step of each loop iteration is the one that appends the encoded result.

Your attempt (not saved)

Key Takeaways

InstructionDataset inherits from torch.utils.data.Dataset and provides __init__, __getitem__, and __len__; __len__ returns the length of the original data list.
The constructor pretokenizes each entry by building a full text and calling tokenizer.encode, storing the results in encoded_texts, so the tokenizing work is done once rather than on every item request.
Batching requires padding inputs to a similar length, and the source uses <|endoftext|> as the padding token.
When an unexpected result appears, check which list the method reads from: counts come from the data list, and items come from the encoded list filled during construction.

Related concepts

More concepts starting with P