Tokenizing Text Messages with tiktoken

Imagine training a text classifier that decides whether a message is spam. The messages in your training data will not all be the same length. One message might be a single short line, and another might be a long paragraph. A spam detector can easily receive both in the same batch.

That mismatch causes a real problem. Models process batches as grids of numbers, so every message in a batch has to occupy the same number of positions. A short message and a long message cannot sit side by side in that grid until their lengths agree. Each message has already been turned into a sequence of token IDs, and those sequences are what must be made equal.

There are two basic ways to make variable-length messages agree in length: truncation and padding. Both produce a uniform grid, but they make different trades. Truncation shortens the longer messages. Padding lengthens the shorter ones. The rest of this article follows both routes so you can see what each one keeps and what each one gives up.

Read the comparison column by column. Truncation has one real advantage: it is computationally cheaper, because the model works with fewer positions overall. Padding costs more computation, since the model handles the extra padding positions, but in exchange it removes nothing from any message.

Truncation is a natural first instinct. Making every message as short as the shortest one sounds clean, and it is cheap to do. The belief to watch for is that this shortening is harmless because it only trims a little. That belief holds only when message lengths are similar. For a dataset where lengths vary a lot, truncation can remove a great deal.

When a message is longer than the target length, everything past the cut is lost, and the model never sees it. If most messages are close to the shortest length, the loss is small. If the shortest message is much smaller than the average or the longest message, the loss can be large, and model performance may suffer as a result.

Padding takes the opposite approach. Every message is extended to the length of the longest message in the dataset, using a padding token to fill the added positions. Nothing is cut away. The padding positions carry no message content; they simply fill space so that the batch forms a uniform grid.

The padding token used here is the string '<|endoftext|>'. The important detail is that you do not append that string to each text message. Instead, you add the token ID that corresponds to it to each encoded message. For the GPT-2 tokenizer from tiktoken, that ID is 50256. You can confirm it by encoding the string with the GPT-2 tokenizer, which keeps padding in the same numeric form as the rest of the encoded data.

Take the same three messages from the truncation example, with 3, 5, and 8 tokens. Padding changes the lengths so that all three become 8 positions. The longest message, with 8 tokens, is already at the target length, so it passes through unchanged. The other two receive padding token IDs appended after their real tokens.

In the diagram, the padding IDs always sit after the real tokens. The real tokens keep their original order at the start of each sequence, so the message content is intact and only the tail is extended. The longest message is the reference point, and every other message is measured against it.

Once every message has the same length, the batch can be read as a grid. Each row is one message, and each column is a position. Position 1 of every message lines up with position 1 of the others, and the same holds for every later position. That alignment is what lets the model process the whole batch at once.

In this illustration the labels t1 through t8 stand in for real token IDs, and PAD marks a padding position. Read straight down any column and you find the same position across all three messages. Column 8 holds a real token in Message C and a padding token in Message A, yet all three rows still have exactly eight positions, which is what the batch needs.

Padding is not applied by hand to each message. It is built into a PyTorch Dataset, which is a class that specifies how data is loaded and processed before any data loaders are created. The dataset class handles the padding step once, so it can be reused every time batches are created.

Seeing padding as one step inside a dataset class explains why it is written once and reused, rather than repeated for each batch by hand. Each time the data is accessed, the same rule produces a uniform grid, and the model receives batches of consistent shape.