Building a Simple Tokenizer with encode and decode

Before looking at how a BPE tokenizer works inside, it helps to run one and watch what happens. The code in this article takes a sentence that contains a special marker, turns it into a list of integers, and then turns that list back into text. Every step is visible in the code and its printed output, so you can judge the behavior yourself before reading the explanation.

The tokenizer used here is the BPE tokenizer from the previous step. Its usage is similar to the SimpleTokenizerV2 class you implemented earlier, and it exposes the same style of encode method. This article is about two calls. The encode call goes from text to token IDs, and the decode call goes from token IDs back to text.

The example runs the encode call on a sentence containing <|endoftext|>, passes the resulting list to decode, stores the returned text in a variable named strings, and prints it. The sequence of operations is shown in the code below. The variable names are the ones used in the example, and the sample text is a generated variant that includes the special marker so you can follow what happens to it.

Read the first printed line as a list of integers, not as words. Each integer is a token ID the tokenizer produced for a piece of the input. Two things stand out. The list is not the same length as the sentence in characters, because a token can cover more or fewer characters than one. Also, the value sitting where the special marker appeared is 50256 in the source output. That single integer stands for the whole marker, which is the key idea behind the special-token behavior explained later.

Encoding works on the input text and produces one integer per token. Ordinary text is converted into ordinary token IDs. A special marker such as <|endoftext|> is a different case, because it is not meant to be read as plain characters. Whether the tokenizer treats it as a special token depends on the allowed_special argument. The diagram below shows the decision that encode makes when the input contains the marker.

Following the diagram, the marker takes the left-hand branch only when the allowed_special argument names it. In that case the marker becomes one special ID, and the output list contains the value 50256 at that position. If the marker is not allowed, the text is handled as ordinary characters and split into ordinary text tokens. The source states this directly: the special marker is handled as a special token rather than being ignored as plain characters when the argument is passed.

The same contrast can be seen by comparing the two token sequences for the marker. The diagram below sets them side by side.

The decode method runs the process in reverse. It takes a list of token IDs and returns the text they represent. Encode takes text to integers, and decode takes integers to text. Together they form a round trip. The sample code passes the integers produced by encode directly to decode, and the returned text is stored in strings and printed.

The diagram shows the path that a check must follow. The returned text is compared with the original input. Any mismatch appears at the end of the path, but its cause could be at any stage: the input construction, the encode step, or the decode step. This is why the next section treats the comparison as a careful, character-level task rather than a glance at the printed output.

The most useful check is whether decode(encode(text)) reproduces the original text exactly. Return to the adjacent-literal question from the opening. The input contains terracesof with no space, because Python joins the two literals directly. The printed output in the example shows terraces of with a space. Two explanations are possible. Either a space was typeset in the sample output that the code does not contain, or the round trip changed the text. The example does not settle which, so do not assume either one.