Batching Text for Language Model Training

A tokenizer is only as trustworthy as the checks we run on it. One of the simplest checks is a round trip. You take some text, convert it into token IDs, convert those IDs back into text, and compare the result with what you started with. If the two match, the tokenizer handled that text faithfully. If they do not match, the difference tells you something about the vocabulary the tokenizer was built from.

The check used in this article is deliberately awkward. The sample joins two sentences that have nothing to do with each other: a question about tea and a line about a sunlit terrace. The two sentences are joined with the <|endoftext|> token placed between them. Because the sample is unrelated text pasted together, it puts pressure on the vocabulary in two places at once: at the words themselves and at the boundary where one text ends and the next begins.

Follow the sample through the round trip in order. The first step is the joining step, which decides what the tokenizer sees at the boundary between the two texts. The <|endoftext|> separator must be present in the joined text for the tokenizer to treat it as a boundary marker. Keep track of that separator as you move forward, because it is the one piece of the sample that is meant to be special.

Encoding walks through the text and assigns each piece a numeric ID from the vocabulary. Ordinary words the vocabulary knows receive their own IDs. The separator is handled as one special token rather than being broken into pieces, so it appears in the ID sequence as a single entry. Words the vocabulary does not contain are not dropped. They are mapped to an unknown-word token instead, so the ID sequence still has an entry at that position.

Decoding converts the IDs back into text. In this sample, the decoded text keeps the separator, the punctuation, and the ordinary words. Only the words the vocabulary lacked come back as unknown-word placeholders. Comparing the decoded text against the original is the sanity check. Any position where the two disagree tells you about the vocabulary, not about the decoding code, because decoding faithfully reproduces whatever the IDs say.

When you read the decoded output, do not scan for a general sense of whether it looks right. Read it position by position and ask one question: at which point does the recovered text stop matching the original? The first position of disagreement is the most useful one, because everything before it was handled correctly and the cause of the change lies at or just before that point.

Special tokens are markers that do not come from ordinary language. Their job is to tell the model something about structure: where a text begins, where it ends, or how to fill space so that texts of different lengths can be processed together. Some LLM pipelines define several such markers. The markers named in this course are [BOS], which marks a beginning, [EOS], which marks an ending, and [PAD], which fills unused space. The separator <|endoftext|> plays a similar structural role in the GPT family.

The GPT tokenizer does not need [BOS], [EOS], or [PAD]. For simplicity, it uses only the <|endoftext|> token. That single token plays the role of [EOS], and it is also reused for padding. This reuse is acceptable because of how training is set up. When training on batched inputs, a mask is applied so that the model does not attend to padded positions. Because of that mask, the specific token chosen for padding has no real effect on the result.

The sample above used an unknown-word token, and it produced placeholders for words the vocabulary lacked. GPT models take a different route. They do not use an unknown-word token at all. Instead, they rely on byte pair encoding, a tokenizer that breaks words down into smaller subword units. The detailed mechanism of byte pair encoding is taught in a later discussion, so for now the important point is the contrast between the two approaches.