Text Preprocessing for LLMs

A large language model does not read a book the way a person does. Before any learning can happen, the text has to be broken into smaller pieces that a model can work with. This article focuses on the first of those pieces of work: tokenization, the process of splitting input text into individual tokens.

Tokenization is a required preprocessing step for creating embeddings for an LLM. Keep that dependency in mind throughout: the embeddings come after the tokens, not before them.

This article follows one practical path. The text used is The Verdict, a short story by Edith Wharton that has been released into the public domain. You can copy the text into a plain text file; the file used here is named the-verdict.txt. A Python program reads that file into a variable using standard file reading utilities.

At this point the state of your program is raw text. The variable holds the characters of the story as they appeared in the file. Nothing has been split yet, so there are no tokens. If you print a slice of the variable, you are looking at the material that tokenization will later process, not at tokens themselves.

The definition has two kinds of token. A word is one kind. A special character, such as a comma, a full stop, a question mark, or an exclamation mark, is the other kind. Punctuation is therefore not thrown away during tokenization; it becomes tokens of its own.

Because tokens are not the same thing as words, the number of tokens in a sentence is not always the number of words. Punctuation adds tokens without adding words.

The source presents tokenization as one stage in a sequence of text processing steps that surround an LLM. Read the sequence below from left to right. It begins with the input text and ends with the output. Tokenization happens after the text is loaded and before token IDs and token embeddings are produced, and before the text reaches the transformer.

Notice what this sequence does and does not claim. It places tokenization directly after the input text and before the token IDs. It does not show how the tokenizer decides where one token ends and the next begins. So the rest of this article works only from the definition given above, and it does not claim anything beyond that about the internal rules of a real tokenizer.

Now apply the definition by hand. Take the sentence 'Hello, world!' and split it into tokens, using the rule that a token is either a word or a special character. Work through it one piece at a time, and decide what each piece is before you move on.

A common first guess is to split the sentence at its spaces. That gives two pieces: 'Hello,' and 'world!'. Compare that expectation with the rule from the definition. The comma and the exclamation mark are special characters, so each should be its own token. The space-based guess keeps them stuck to the words, and this is the exact point where the expected split stops matching the token boundaries.