Tokenizing Text with the GPT-2 Tokenizer
SpamDataset loads a CSV file, tokenizes each text with the GPT-2 tokenizer, and returns padded token tensors with labels through __getitem__.
Why Every Text Must Become the Same Width
A model cannot read a spreadsheet row the way a person does. It needs every training example converted into a tensor of numbers, and every example in a batch has to have the same length. The SpamDataset class is the piece of code that takes a CSV file of labeled text messages and produces exactly that: for each message, a sequence of token IDs of a fixed width, together with a numeric label.
Keep one question in mind as you read: at which step does a text stop being a string and start being a list of equal-length numbers? Before reading the explanation, make your own prediction using the small case below.
What do you think happens?
A text has been tokenized into 4 token IDs, and the target max_length is 6. What happens to that list?
Reveal answer
It is padded with filler items until it has 6 items.
Any list shorter than max_length gets pad_token_id appended until it reaches the target width. The 4 real token IDs are kept, and 2 padding items are added at the end.
Tracing the Constructor Stage by Stage
The constructor is where all the preparation happens. It is easiest to read it as a sequence of five stages rather than as one dense block. Each stage hands its result to the next, so the order matters.
- 1Stage 1, load the file: the constructor reads the CSV into a data frame and keeps it on the object.
- 2Stage 2, tokenize: for every text in the Text column, tokenizer.encode converts the string into a list of integer token IDs. These lists are stored in
encoded_texts. This happens once, up front, for the whole file. - 3Stage 3, decide the target length: if
max_lengthwas passed as None, the constructor finds the longest encoded text and uses that length. If a number was passed, that number is used directly. - 4Stage 4, truncate: any list longer than
max_lengthis cut down to its firstmax_lengthitems. - 5Stage 5, pad: every list shorter than
max_lengthgetspad_token_idappended until it reaches the target width.
Truncation always runs before padding. That ordering means padding only ever has to add items, never remove them, so a list is never padded and then cut back.
Padding and Truncation Side by Side
Padding and truncation solve opposite problems. Padding handles a sequence that is too short for the target width by adding filler at the end. Truncation handles a sequence that is too long by discarding everything past the limit. Both produce the same outcome, a list of exactly max_length items, but they change the data in different ways.
The difference matters for meaning. Padding adds items that carry no meaning for the message, so the model learns to treat them as filler. Truncation removes real tokens from the end of the message, so any content past the limit is no longer visible to the model.
| Question | Padding | Truncation |
|---|---|---|
| When it applies | The list is shorter than max_length | The list is longer than max_length |
| What is added or removed | pad_token_id items are appended at the end | Items past the limit are discarded |
| Does real content change | No, real tokens are kept and filler follows them | Yes, the tail of the message is lost |
| Stage in the constructor | Stage 5, runs after truncation | Stage 4, runs before padding |
How padding and truncation differ inside SpamDataset
Following Two Texts Through max_length
The following examples use small, made-up token IDs to show the mechanism. The numbers are not real GPT-2 output. The steps are what matter.
Truncating a text that is too long
A message has been tokenized into the list [12, 88, 5, 31, 7, 64, 19]. The constructor is using max_length 5. What list does the constructor store?
- 1Measure the list
The list has 7 items, which is more than
max_length5, so truncation applies. - 2Keep the first 5 items
Truncation keeps the first
max_lengthitems and discards the rest. The first five are 12, 88, 5, 31, and 7. - 3Discard the tail
The items 64 and 19 are removed. Those tokens are lost, and the model never sees them for this message.
- 4Check padding
The list now has 5 items, equal to
max_length, so padding adds nothing.
The stored list is [12, 88, 5, 31, 7]. It has exactly 5 items, and the last two real tokens were lost.
Padding a text that is too short
A message has been tokenized into [40, 2, 90]. The constructor is using max_length 5. What list does the constructor store?
- 1Measure the list
The list has 3 items, which is shorter than
max_length5, so padding applies. - 2Keep the real tokens
The three real token IDs 40, 2, and 90 are kept in their original order.
- 3Append filler
Two copies of
pad_token_idare appended at the end until the list reaches 5 items.
The stored list is [40, 2, 90, pad_token_id, pad_token_id]. Every item from the original message is still present.
Compare the two results. The short text keeps all of its content and gains filler at the end. The long text keeps its opening and loses its tail. Both finish with exactly max_length items, which is the property the batching step depends on.
Choosing max_length and the Model Limit
When the training dataset is created with max_length set to None, the class stores the longest encoded length in its max_length attribute. That value becomes the width every training example is padded to, so no training text is ever truncated. The source example reports that its longest training sequence contains no more than 120 tokens, a length typical of text messages. You can inspect that value on any dataset object by printing its max_length attribute.
The 120 figure is a property of that particular file, not a fixed constant. A different CSV would produce a different longest length. The model can handle sequences of up to 1,024 tokens, so a dataset whose longest text is far below that limit works without changes. If some texts exceed the limit, passing max_length=1024 when creating the training dataset caps every sequence at the model's supported context length.
Reusing the Training Width for Other Splits
The validation and test datasets must match the training width. The source achieves this by passing the training dataset's max_length into both of them. Any validation or test text longer than that width is truncated by the same slicing step, and shorter texts are padded up to it. Because every split ends up with the same width, batches built from any of them have the same shape the model was trained on.
The mismatch problem sits at stage 3 and the stages after it. Stage 3 fixes the target width for each dataset. If a validation dataset chose its own longest length instead of receiving the training value, its truncation and padding stages would produce lists of a different width. Batches built from it would then not share the width the model was trained on. The source does not describe a specific error message, so the reliable check is to confirm that every split was created with the training max_length.
The source also notes that truncating these splits is optional. Setting max_length to None for the validation and test sets is acceptable as long as none of their sequences exceed 1,024 tokens. The reason to keep the choice consistent is that the model is evaluated on exactly the inputs it would see during training, so the widths should agree.
What __getitem__ Hands Back
The class inherits from torch.utils.data.Dataset, which means it must supply two things PyTorch relies on: the number of samples and the sample at a given index. The __len__ method returns the number of rows in the data frame, so PyTorch knows how many samples exist.
The __getitem__ method takes an index and returns a pair. The first element is the padded list of token IDs converted to a tensor with the long data type. The second element is the label for that row, read from the Label column with iloc and also converted to a long tensor. Both returned values are tensors rather than plain Python lists, which is what the model and the batching machinery expect.
The step where a text stops being a string is stage 2, when tokenizer.encode returns a list of integers. The step where it becomes a list of equal-length numbers is stage 5, when padding finishes. The __getitem__ method only converts that finished list into tensors.
Mistakes Beginners Make with max_length
Common mistake Assuming padding removes or changes the real tokens
What it looks likePadding the list [40, 2, 90] to width 5 and expecting [40, 2, 90] with its last items replaced
Why it failsPadding only appends filler at the end. The real tokens stay in their original positions.
Do this insteadExpect the original items followed by pad_token_id items.
Common mistake Measuring the longest length after padding
What it looks likePrinting the maximum list length after the constructor has already padded every list
Why it failsEvery list already has the same width, so the result says nothing about the original texts.
Do this insteadMeasure the raw encoded lengths before truncation and padding, as the constructor does when max_length is None.
Common mistake Giving the validation set its own longest length
What it looks likeCreating the test dataset with max_length None while the training dataset used 120
Why it failsThe test batches then have a different width from the one the model was trained on.
Do this insteadPass the training dataset's max_length attribute into the validation and test datasets.
Common mistake Believing a long text is fully kept when max_length is fixed
What it looks likeSetting max_length to 5 and assuming a 7-token message still reaches the model in full
Why it failsTruncation discards everything past the limit, so the last two tokens are removed.
Do this insteadCheck the max_length attribute and the longest encoded text before choosing a fixed value.
Trace a New Text Through the Constructor
A dataset is built with max_length 4. One message has been tokenized into [7, 15, 3, 22, 9, 30]. Write the list the constructor stores for that message, and say which stage produced each change. Then repeat the exercise for a second message tokenized into [11, 6], using the same max_length.
Hints
- Decide first whether the message is longer or shorter than
max_length4. - Truncation runs before padding, so the first message is only cut down, and the second is only padded.
- The first message keeps the first four IDs: 7, 15, 3, and 22.
Key Takeaways
- The SpamDataset constructor runs five stages in order: load the CSV, tokenize each text into integer IDs, set the target length, truncate lists that are too long, and pad lists that are too short.
- Truncation always runs before padding, so padding only adds
pad_token_iditems and never removes real tokens. Truncation removes the tail of a message, which can lose meaningful content. - When
max_lengthis None, the longest raw encoded length is measured before any changes and becomes the training width, so no training text is truncated. - Validation and test datasets receive the training
max_lengthso that their batches share the width the model was trained on. The mismatch problem lives in the target-length stage and the truncation and padding stages that follow it. __getitem__returns the padded token IDs and the label as long tensors, which is the form the model and batching machinery expect.